texts:
I am new to this data science field.

I have data of points in 3D space and each point "helps" a metric. I have the sets of points and corresponding metrics.

Data might look like:

[Pt_set1_1], [Pt_set1_2], [Pt_set1_3], ..., [Pt_set1_20], Metric1, Metric2, Metric3
[Pt_set2_1], [Pt_set2_2], [Pt_set2_3], ..., [Pt_set2_20], Metric1, Metric2, Metric3

I want to identify which points help which metrics. Is this problem solvable with the data science algorithms. Which algorithm would help me? Please let me know. Any help would be appreciated.


texts: What is the exact definition of VC dimension?

texts:
I'm studying machine learning from Andrew Ng Stanford lectures and just came across the theory of VC dimensions. According to the lectures and what I understood, the definition of VC dimension can be given as,

  If you can find a set of $n$ points, so that it can be shattered by the classifier (i.e. classify all possible $2^n$ labeling correctly) and you cannot find any set of $n+1$ points that can be shattered (i.e. for any set of $n+1$ points there is at least one labeling order so that the classifier can not separate all points correctly), then the VC dimension is $n$.

Also Professor took an example and explained this nicely. Which is:

Let,

$H=\{{set\ of\ linear\ classifiers\ in\ 2\ Dimensions \}}$

Then any 3 points can be classified by $H$ correctly with separating hyper plane as shown in the following figure.

And that's why the VC dimension of $H$ is 3. Because for any 4 points in 2D plane, a linear classifier can not shatter all the combinations of the points. For example,

For this set of points, there is no separating hyper plane can be drawn to classify this set. So the VC dimension is 3.

I get the idea till here. But what if we've following type of pattern?

Or the pattern where a three points coincides on each other, Here also we can not draw separating hyper plane between 3 points. But still this pattern is not considered in the definition of the VC dimension. Why? The same point is also discussed the lectures I'm watching Here at 16:24 but professor does not mention the exact reason behind this.

Any intuitive example of explanation will be appreciated. Thanks


texts: Platt's Scaling for multi-label classification?

texts:
How does one do platt's scaling for multi-label classification? For example, if the final layer of my DNN is a soft-max activation with 10 classes, then how does platt's scaling work exactly?

Do I train multiple logistic regressions using one-vs-many classification? Or is there a better way?


texts: data pre-processing before image classification

texts:
I'm working on a machine learning project, Images classification (shape: 100 x 100)-> (vector of 10000), I did some pre-processing before applying decision trees algorithm , I got an accuracy of 55 % I tried to change parameters but the accuracy did not increase, Can someone suggest something to do in pre-processing exept what I did :

Denoising Images: Images are extremely noised, so I only kept relevant information in each image.(it works well)
Centring Images: I translated the relevant values for all images to the top left to keep relevent information in same part of my vectors
Resizing Images: I resized each image to (40,40) -> (1600 vector) to make the inputs smaller

Thank you


texts: Can we change the structure of the feature-extraction layers of deep networks architectures?

texts: I want to know if we can change the structure of the feature-extraction layers in the deep networks architectures, for example can we add more Rectified Linear Unit (ReLU) activation function or is it the same sequence that we should respect ?


texts: Can you l2 normalize word2vec vectors for density clustering?

texts:
I have a situation where I have to cluster word2vec vectors (200 length dimension vectors on a very large corpus). I decided to use Density based clustering (DBSCAN, HDBSCAN) because my dataset is very high in noise, and I do not want it to be part of my clusters. My knowledge on Cosine distance is limited but I find that Density based clustering algorithms do not have a direct implementation using cosine distance (pairwise_distance calculation is too memory intensive).

My question here is can I normalize the Word2vec vectors using L2 normalization using: norm_data = normalize(vector_array, norm='l2') from the sklearn library to normalize these vectors, and then use euclidean distance over the normalized vectors?

Can someone suggest any other better technique to cluster word vectors when there is noise in the dataset?


texts: Chinese word segmentation using neural networks

texts:
Chinese text uses a character set containing tens of thousands of characters. Words in Chinese are most commonly made up of 1, 2 or 3 characters. There are no spaces or other markers between words in Chinese text since native speakers can easily segment words from text. In order to assist language learning tools, it's helpful to have an automatic way to segment words within blocks of text.
Some approaches use dictionary based greedy algorithm methods but are prone to failing due to the common problems that most greedy algorithms present.
I want to try using neural networks, but my question is, How would one go about encoding the characters to the input neurons of the network?
I am not talking about OCR. The characters are already known and encoded in unicode, but how would I present the characters to the inputs of the network?
One method I can imagine is to have the network look at the text in portions of say 100 characters at a time, and have one neuron for each character. But how would I represent the character as a number to the network? Using the unicode integer number value for the character doesn't seem like a good idea.


texts: class activation mapping when accuracy is 100%

texts:
I am a beginner to image classification and apologies beforehand if the question I am asking is dumb. I am currently using the following model:

baseModel = VGG16(weights="imagenet", include_top=False,
    input_tensor=Input(shape=(224, 224, 3)))

# construct the head of the model that will be placed on top of the base model
headModel = baseModel.output
headModel = Flatten(name="flatten")(headModel)
headModel = Dense(64, activation=tf.nn.leaky_relu)(headModel)
headModel = Dropout(0.5)(headModel)
headModel = Dense(2, activation="softmax")(headModel)
model = Model(inputs=baseModel.input, outputs=headModel)

For this model, I am doing a binary image classification. The model is compiled as such:

opt = Adam(lr=INIT_LR, decay=INIT_LR / EPOCHS)
model.compile(loss="binary_crossentropy", optimizer=opt,
    metrics=["accuracy"])

This model will give a 100% for both the training and validation accuracy. I wanted to understand how the model is activated for certain images. So I tried to calculating the heatmap. However in one of the calculations we have:

    def compute_heatmap(self, image, eps=1e-8):

        gradModel = Model(
            inputs=[self.model.inputs],
            outputs=[self.model.get_layer(self.layerName).output,
                     self.model.output])

        with tf.GradientTape() as tape:
            inputs = tf.cast(image, tf.float32)
            (convOutputs, predictions) = gradModel(inputs)
            loss = predictions[:, self.classIdx]

        grads = tape.gradient(loss, convOutputs)

The gradient is calculated from the loss which is roughly zero since the accuracy is 100%. This will result in a bank/flat heatmap.

When this happens, how should I deal with class activation mapping?

Sorry for the long post and any help will be appreciated!


texts: Better to crop or compress training data?

texts:
I've been trying to make an object recogniser in Tensorflow and have used labelImg to classify large electrical transmission towers at varying distances. In order to make 10-16MP (~2-7MB) images train with my computer's 16GB of RAM, I need to crop them down until they're about 200x500 and <100kB.

The way I see it, this preserves the detail of a far away tower, and means the image_resizer function, set at 1024x600, won't be smashing a 200x500 tower in a 5000x3000 image into total unrecognisability. The trouble is that the algorithm seems only to recognise towers up close and large in frame, and I suspect that it might be a lack of outside perspective in the cropped version that has done this. (Either that or not enough shots of far away towers) But if I compress the images down to even 1920x1080, the towers still become blurred and artefacted to hell.

Just wondering if anyone has any experience with new training datasets and could tell me what's better to feed in.
Have attached a 2MP resized version and a cropped version as an example. Cropped version is of the tower on the right. Tower in 2MP is much more artefacted.


texts: How do we ensure that a model generalize well in the real world

texts:
When we have trained a model on a training data set as well as the test set, how can we ensure that a model will generalize in the real world.I mean both the test set and the training set are a very small subset compared to what exist in the real world and there might be inherent bias in the datasets as well.

As a beginner in machine learning , it will help me and others a lot if you share the techniques to avoid the common pitfalls


texts: Geometric Interpretation of Whether SVMs are performing well or not

texts:
I came across this research paper which contained this figure

which talks about the center of mass (presumably, of the training dataset's datapoints?) and represents the solution of an SVM as polygon (or is it a point?).
I'm having trouble understanding this figure, and since it seems to provide a geometric interpretation of when an SVM performs poorly or not, I'm interested in it. Any thoughts?


texts: How do you calculate the information capacity of a neural network?

texts:
Let's say I wanted to train a neural network to teach it the rules in a decision tree, so I generated a dataset by feeding arrays of random numbers into the pre-trained decision tree, and then used the random input arrays as the features in the dataset and the output of the decision tree as the output.

I feed my dataset to the neural network, and start it training. If there are 6 features, logically, a two-layer network can't learn a million-bit decision tree, because there simply isn't enough "information capacity", essentially entropy, in the neural network structure to store such a complex ruleset. With decision trees, calculating entropy is fairly simple. But how would you do it for a neural network?

How would you calculate the minimum size neural network needed to learn a given ruleset?


texts: Transformation from Datawarehouse into Big Data structure

texts:
What are the required stages for transferring data in datawarehouse into big data structure. Are there any tools and methods that support it?
How to use the schema for such transformation, how to deal with different data types like facts and dimensions for instance. What is the criteria for data separation into machines, indexes or unique keys?


texts: Can a recommendation system be built without any user ratings?

texts: I was planning to make an artwork recommendation system as a project by using the WikiArt open source dataset available on kaggle, I'm still looking for datasets which might already have user ratings but is it possible to make a recommender without any user ratings? The dataset currently has these features. Any help or insight is appreciated. Thanks.


texts: Why is the cost increasing in the linear regression method?

texts:
I am implementing the linear regression model from scratch.

#!/usr/bin/env python2
# -*- coding: utf-8 -*-
"""
Created on Thu Nov 16 14:40:53 2017

@author: user
"""

import os
import random
os.chdir('/home/user/Desktop/andrewng/machine-learning-ex1/ex1')

import pandas as pd
data = pd.read_csv('/home/user/Desktop/andrewng/machine-learning-ex1/ex1/ex1data1.txt',header=None)

theta_0 = random.random()
theta_1 = random.random()
alpha = 0.001

print(len(data))

hist = -90
cost = 0
print('theta_0    + theta_1    ')
while(cost-hist>0.001):
    hist = cost
    cost = 0
    a = 0
    b = 0
    for i in range(len(data)):
        k = data.iloc[i]
        a = a +  theta_0 + theta_1*k[0] - k[1]
        b = b + (theta_0 + theta_1*k[0] - k[1])*k[0]
    theta_0 = theta_0 - alpha*a/len(data)
    theta_1 = theta_1 - alpha*b/len(data)
    #print(str(theta_0)+'    '+str(theta_1))
    for j in range(len(data)):
        k = data.iloc[i]
        cost  = cost + (theta_0 + theta_1*k[0] - k[1])**2
    cost = cost/(2*len(data))
    print(cost)
    #if(cost>hist):
    #    print(str(theta_0)+'    '+str(theta_1))
    #    break
print(str(theta_0)+'    '+str(theta_1))

According to the theory the cost should decrease at each iteration but for me the cost keeps increasing.

(DATA)


texts:
I aint no data scientist/machine learner.

What Im Lookin for

text = "Donald Trump became the president of America"
#some data science works
print(topics)
#prints ["politics"]

text = "Rihanna is starring in the new movie The Inception 2"
#some data science works
print(topics)
#prints ["movie","music"]

What I can do

I can extract words like Donald Trump,America,Rihanna using POS

I can get huge paragraphs/lists of articles/words on politics,movies etc and save them in a text file.

What i can't do

Make meaningful topics out of  those words like sports,politics,movies

What i want you do to

Point me in the right specific direction towards solving this problem aka enlighten me


texts: Suppose I have data in the form of  Query/Document Pairs, along with corresponding relevance scores (or class labels). Is there a way to use topic modeling to devise a model so that later given a query and a document, we can predict its relevance score?


texts: What are the best practices for data formatting?

texts:
I used to do some analysis on Excel but my company want to use Python to increase the analysis.
I am novice on python and overwhelmed by all of this knowledge I have to learn :'(

So for my first project, I have imported my csv file and I had this error message "C:\Users\xxx\AppData\Local\Continuum\anaconda3\lib\site-packages\IPython\core\interactiveshell.py:2785: DtypeWarning: Columns (8,9,31,32,35,36,37,38,39,40) have mixed types. Specify dtype option on import or set low_memory=False.interactivity=interactivity, compiler=compiler, result=result) "

Sadly I have no clue about all of this so I looked on the web and guess what?it's still not clear for me. So my questions are

What should I do to avoid this error message ?
What are the best practices/ formatting when you work with dataset ?

Thanks


texts: Any pretrained models for Style Transfer trained with large images (>512px)?

texts:
I have found some pretrained models that are prepared for small image sizes.

https://tfhub.dev/google/magenta/arbitrary-image-stylization-v1-256/2' (256x256px)
https://keras.io/api/applications/vgg/#vgg19-function (224x224px)

Are there any opensource models pretrained for larger images (>512px)?
If not, is it for a computing power reason?


texts: How to train neural network a math multiplication table?

texts:
I am trying to train neural network (brain.js) a multiplication table. It is not going too well: requires lots of hidden layers, iterations and very small error threshold, and the results are still incorrect. What am I doing wrong?
console.clear();
let net = new brain.NeuralNetwork({hiddenLayers: [128]});
let training_data = [
    {input:[2,2], output:[4/81]},
    {input:[2,3], output:[6/81]},
    {input:[2,4], output:[8/81]},
    {input:[2,5], output:[10/81]},
];
net.train(training_data, {
    errorThresh: 0.0000001,  // error threshold to reach before completion
    iterations: 500000,   // maximum training iterations
    log: true,           // console.log() progress periodically
    logPeriod: 10000,       // number of iterations between logging
    learningRate: 0.01    // learning rate
});
for(let i = 2; i < 3; i++){
    for(let j = 2; j < 10; j++){
        console.log(i + " * " + j + " = " + net.run([i,j])[0]*81);
    }
}

The output is:
2 * 2 = 4.265779517591
2 * 3 = 5.825156696140766
2 * 4 = 7.789406754076481
2 * 5 = 10.152319103479385
2 * 6 = 12.860703930258751
2 * 7 = 15.819618478417397
2 * 8 = 18.90912254154682
2 * 9 = 22.006048411130905
After receiving the answer from SRJ I have changed the neural net so, that it now has 4 hidden layers and receives 10 000 inputs:
console.clear();
let net = new brain.NeuralNetwork({hiddenLayers: [4]});
let training_data = [
];

for(let i = 0; i < 100; i++){
    for(let j = 0; j < 100; j++){
        training_data.push({input:[i,j], output: [i*j/10000]});
    }
}
net.train(training_data, {
    errorThresh: 0.001,  // error threshold to reach before completion
    iterations: 20000,   // maximum training iterations
    log: true,           // console.log() progress periodically
    logPeriod: 100,       // number of iterations between logging
    learningRate: 0.0001    // learning rate
});
let random_numbers = [Math.floor(Math.random()*50), Math.floor(Math.random()*50)]
console.log(random_numbers[0] + " * " + random_numbers[1] + " = " + net.run([random_numbers[0], random_numbers[0]])[0] * 10000);

Still does not work. The output: "29 * 42 = 728.9011776447296". (while in reality should be 1218).


texts: How to automatically test for the best parameters for transformed independent variable in linear model

texts:
Let's assume that I have a linear model with $k$ variables:

$y = \beta_0 + \beta_1\cdot x_1 + \dots + \beta_k \cdot x_k$.

Now, I want to add variable $x_{k+1}$, but, according to domain knowledge, the dependency of $y$ onto $x_{k+1}$ is not linear, but rather "S-shaped". To capture this dependency, a well-established method is to use parametrised arcus tangens function: $D \cdot \arctan{\frac{x_{k+1} - A}{B}} + C$. To include it in the linear model we can safely ignore $D$ and $C$ parameters (as they will contribute to $\beta_{k+1}$ and $\beta_0$ respectively), so eventually I'd end up with the model:

$y = \beta_0 + \beta_1\cdot x_1 + \dots + \beta_k \cdot x_k + \beta_{k+1}\cdot \arctan{\frac{x_{k+1} - A}{B}}$.

What I would like to do is to find a way to test different transformations (with different $A$ and $B$ parameters automatically. Firstly, I went for searching the grid of different $A$ and $B$ values, i.e. fitting the model to different transformations of $x_{k+1}$ and returning list of models, sorted by $R^2$, $RMSE$ or $AIC$. However, this usually favours very sharp curves:

This doesn't make much sense in the eye test though. I know that eye-test may be biased but to me, more propable dependency would be captured by a shape as below:

One could already noticed that there are plenty of observations where $x_{k+1}$ is or is near to $0$, which is actually a characteristic of those variables that will be approximated by $arctan$. That may probably affect the fitting, but still something clearly doesn't work here.

Another approach that I considered is to fit the model using gradient descent search rather than using standard software (like lm from R). The problem is that with this $arctan$ transformation, we don't have a convex objective function anymore.

How can I then automatically test for the best parameters of the $arctan$ transformation? Am I on the right track with either idea or can I approach this problem differently?r


texts: What the differences between self-supervised/semi-supervised in NLP?

texts: GPT-1 mentions both Semi-supervised learning and Unsupervised pre-training but it seems like the same to me. Moreoever, "Semi-supervised Sequence Learning" of Dai and Le also more like self-supervised learning. So what the key differences between them?


texts: 'PCA' object has no attribute 'explained_variance_'

texts:
Elbow Method - Finding the number of components required to preserve maximum variance.

My code:

pca = decomposition.PCA()

vectorizer = TfidfVectorizer(min_df=10)
preprocessed_essay_tfidf = vectorizer.fit_transform(preprocessed_essay)
pca_data = preprocessed_essay_tfidf
pca.n_components = 3000

percentage_var_explained = pca.explained_variance_ / np.sum(pca.explained_variance_);
cum_var_explained = np.cumsum(percentage_var_explained)

I get the attribute error:

AttributeError: 'PCA' object has no attribute 'explained_variance_

Why is this?

I can find explained_variance_ present here.


texts: If my data looks like this, is linear regression just never gonna be useful?

texts:
Clearly you can see that there is no linear relationship. If there was one it would be a vertical line which is not that useful. I need some type of regression rather than classification as the output is continuous. The only options I can think of are neural networks and SVR (support vector regressors). Is linear regression a waste of time here? What would you use?

The use case is - I am trying to predict the rgb color of a polygon based on its area, length, index, complexity, #of lines, # of curves, and intensity (black and white).

In this first step I am merely trying to predict the amount of red from 0 to 1. (scaled down from 255).

Edit:

here are the labels - rgb values: http://www.heypasteit.com/clip/2HZX

here is the data with the 7 features: http://www.heypasteit.com/clip/2HZY


texts: Identifying Customers who are more likely to purchase a given product category- Which model to use?

texts:
I'm pretty new to data science. I am working on a model to identify customers who are more likely to purchase a given product category.

I did try market basket using arules package from R.
I also used MatchIt, Matching algorithm and did try Propensity score matching to find similar customers who tend purchase in the given product category. But I'm not quite sure the model is good enough for this kinda problem.

I'm just wondering what kind of model will be useful to find customers who are more likely to purchase a given product group.
Any help is greatly appreciated.

Thanks in advance.


texts: How to do batch inner product in Tensorflow?

texts:
I have two tensor a:[batch_size, dim]  b:[batch_size, dim].
I want to do inner product for every pair in the batch, generating c:[batch_size, 1], where c[i,0]=a[i,:].T*b[i,:]. How?


texts:
I have a data set of recruiting pipeline information that has 12 columns of categorical variables (they range from binary variables like gender to non-binary variables like the name of the school the applicant attended). The last column (13) contains a "Yes" or "No" value that tells if the person is still in the recruiting process.

The idea here is to build a model that can predict (reasonably well) the likelihood of the person withdrawing from the recruiting process at any given stage (there are many stages and this field is one of the 12 independent variables captured).

I was thinking of using logistic regression to create the model but all the predictor variables are categorical, which I hear, doesn't bode well for logistic regression. Another factor is that some fields have missing values (like gender) that I cannot reasonably input/capture accurate data for atm.

Given my situation, what approach do you think would be a good starting point?

Was thinking of random forest but not sure if there's a better way to go about tackling this problem.


texts: Changing reference class in imbalanced data drastically affects the error rate

texts:
Working on a binary classification problem that tries to predict customer churn, the data set is imbalanced with 2000 observations of non-churn cases vs 600 observations of churn cases.

On using GLM I see that when the majority class[Non-churn] is the reference level I get ~40 % error rate[confusion matrix] on both the levels [churn non-churn] but when the minority class is set as the reference level I get 100% error rate in predicting the minority class or in a way everything gets predicted as non-churn case.

After balancing the data using SMOTE the same trend continues, how should I interpret this behaviour. ?

Is it in a way saying that the non-churn population has users who have similar behaviour as the churners and hence the high error rate, but at the same time the non-churn users have a subset which are quite different than the churners in their behaviour and hence lower error rate when the reference class is the majority or the non-churn class.

Outcome on test data when majority class is set as the reference class:
Confusion Matrix (vertical: actual; across: predicted) for F1-optimal threshold:
             0   1    Error      Rate
    0      268 419 0.609898  =419/687
    1       46 168 0.214953   =46/214
    Totals 314 587 0.516093  =465/901

Outcome on test data minority class is set as the reference class:
Confusion Matrix (vertical: actual; across: predicted) for F1-optimal threshold:
                   1   0    Error      Rate
            1      3 211 0.985981  =211/214
            0      1 686 0.001456    =1/687
            Totals 4 897 0.235294  =212/901


texts: Isn't (steps_per_epoch = total training data/batch size)?

texts:
Suppose i have 1000 dog images and my batch size is 10.
It will take 1000/10=100 steps to complete 1 epoch.
So doesn't it mean steps_per_epoch=100 ?
Then why do we have to specify it separately in keras while applying .fit().


texts: Representing 2 dimensional data for facial recognition

texts:
My goal is to train a neural network to recognize faces based on a list of landmark points generated by Google Firebase ML-Kit. Since I just started familiarizing myself with ML, I only want my model to recognize if a face belongs to a person in a smaller set of people and if so, to which one.
Hence, each face is represented by a list of (x, y) coordinates denoting each landmark point recognized by the Google library. Variations in the absolute locations of the points and their relative position to each other are cues telling people apart.

Therefore, from a list of points such as this one:

I get a list of coordinates with the person one-hot encoded at the end:

1975.72,120.265564,2108.6755,141.79669,2344.71,214.35474,[...], 000010000

and I want to create a .csv file containing multiple rows for each person.

My problem is, that I do not know how to represent that the points are actually grouped by two as (x, y) coordinate of the very same point in a way that my model can understand it.

Since I'm new to this, I'm not even sure if my model actually needs to understand this in order to recognize someone out of a list of points so sorry, if this is a stupid question. Also, I do not need the problem to be solved, I only request help with the notation system in my training/test/validation data.

Clarification

I already know how to extract the list of points from an image using Google ML-Kit. What I do not know is how to represent points in a way that the net will pick up that x and y belong together constructing a 2d object.

For example, I could have a raw csv file like in the example and all I would need to do is splice the content alongside the commas, but then the net would never know about the relationship. Instead of a list of 1d values, however, I could also expose the net to a list of 2d points such as [[x, y], [x, y], [.....]], but I don't know if data with n-dimensional elements needs any special attention or the net can understand it as if it were a regular list with consecutive elements.


texts: How to combine two logistic regression models trained on different set of data?

texts:
My data has a hierarchy structure - meaning that there is an N class at level 1 and an M class at level M. After training both models separately with a different set of data (both are Logistic regression but can be changed) I want to predict to N + M classes.
Currently, I am solving it by collecting all data before training and adding examples from level 2 to level 1 results in (N + 1) classes (+ 1 meaning all examples from level 2) and if the model at level 1 predicts that "+1" class then a send the query to the second model. But collecting the examples before training is costly and also the training set is unbalanced after collecting. Pseudocode:
clf1 = LogisticRegression()
clf1.fit(X1 + X2, Y1 + ["ID of +1 class"] * len(Y2))
clf2 = LogisticRegression()
clf2.fit(X2, Y2)
if cl1.predict(X_test_example) == ["ID of +1 class"]:
    clf2.precict(X_test_example)

But I need to preserve this hierarchical structure so the merging the two models directly is not possible. Is there a way to combine two outputs of classification models (in a form of the probability distribution) to get meaningful output are avoid problems with the overconfidence of models? Pseudocode:
clf1 = LogisticRegression()
clf1.fit(X1, Y1)
clf2 = LogisticRegression()
clf2.fit(X2, Y2)
combined_clf = clf1 + clf2


texts: Training on a subset of data for few epochs and then proceeding to the next subset for few epochs and so on?

texts:
My training data is very huge and it's impossible to load all of it at once even into main memory. So I'm loading a few blocks (subset) of data and training till convergence, then proceeding to next subset and training till convergence and so on. Is it the right approach ?

The model performance kind of remains the same even when training on a new subset of data.

Is this method fundamentally wrong, why ?


texts: What tools can I use to make a visualization similar to this one? I want to have the mean be bolded and the standard deviation be shaded.


texts: How to estimate the mutual information numerically?

texts:
Suppose I have a sample {$z_i$}$_{i\in[0,N]}$ = {($x_i,y_i$)}$_{i\in[0,N]}$ which commes from a probability distribution $p_z(z)$. How can I use it to estimate the mutual information between X and Y ?

$MI(X,Y) = \int_Y \int_X
                 p_z(x,y) \log{ \left(\frac{p_z(x,y)}{p_x(x)\,p_y(y)}
                              \right) }$

where $p_x$ and $p_y$ are the marginal distributions of X and Y:

$p_x(x) = \int_Yp_z(x,y)$

$p_y(y) = \int_Xp_z(x,y)$.


texts: A deep learning approach that can calculate distance to obstacles

texts:
After reading about the most famous object detection CNN based methods: YOLO, YOLO 9000, r-cnn, faster r-cnn, etc., I was wondering if there is an architecture that can calculate the distance to the obstacle after detection?
Thank you.


texts: Is it possible to change misclassification cost in caret?

texts:
0
down vote
favorite

I have an imbalanced dataset to work with (with about 20-fold positive examples than negative). I know several solutions to deal with this type of data (under/oversampling, optimizing for AUC, etc.) but I would like to try changing the misclassification costs for the two groups. However, I cannot figure out the proper way to do this in caret. Do you have any idea how to do this?

Thanks for any help!


texts:
Using the unit step activation function for the artificial neuron, determine a set of weights (and threshold value) that will produce the following classification:

x1  x2  output
-0.2 0.5    0
0.2 -0.5    0
0.8 -0.8    1
0.8 0.8     1

I understand the theory of unit step activation function (maybe beginner) but i am stumped as to how to calculate the weights and threshold values. I thought that it is specific to the problem and is provided as input. Please help


texts: I'am reading an article called "FaceNet: A Unified Embedding for Face Recognition and Clustering". And there, they use something called "inception". I guess it's something about layers, but I can't find any information about it. Just found some code, which doesn't explain much. Can somebody give a clue, how can I search for this correctly, or maybe, kindly and briefly tell the concept idea?


texts: Why not always use the ADAM optimization technique?

texts:
It seems the Adaptive Moment Estimation (Adam) optimizer nearly always works better (faster and more reliably reaching a global minimum) when minimising the cost function in training neural nets.

Why not always use Adam? Why even bother using RMSProp or momentum optimizers?


texts: Initializing a recurrent neural network (RNN) in Keras

texts:
When we create an RNN in Keras, does it learn an initial hidden layer (ie. $h_0$, like a bias term)? I'm having trouble figuring this out, but I have looked here.

Can anyone highlight a part in the code that shows an initial hidden layer is being learned? Or is it just randomly initializing the hidden layer?

EDIT: The code is here , I'm just having trouble identifying where the hidden layer is initialized and what it is initialized to.


texts:
I got the following problem: When I trained my model I created my dummy variables(before train-test split) in the following way:

dummy <- dummyVars(formula = CLASS_INV ~ ., data = campaign_spending_final_imputed, fullRank = TRUE)
dummy %>% saveRDS('model/dummy.rds') #I save it to use it later
campaign_spending_final_dummy <- predict(dummy, newdata = campaign_spending_final_imputed) %>% as.data.frame() %>%
  mutate(CLASS_INV = campaign_spending_final$CLASS_INV)

The model was trained and tested successfully. Now I want to test it on 'real world' data and I want to create dummy variables from a single new record. What I tried to do was :

dummy_inv <- readRDS('model/dummy_inv.rds') #The file I saved above
predict(dummy_inv,single_record)

The single record has the same features of the training and test set, it is just a single row.
However when I execute the predict function the following error comes out:

Error in `contrasts<-`(`*tmp*`, value = contr.funs[1 + isOF[nn]]) :
  contrasts can be applied only to factors with 2 or more levels

Am I proceeding in the correct way? Do I need to create a new dummyVars  object? Shouldn't I use the one 'adapted' to my trainin data?
Thank you


texts: Accuracy after selftraining didn't change

texts: I used Decisiton Tree Classifier which I trained with 50 000 samples. I have also set with unlabeled samples, so I decided to use self training algorithm. Unlabeled set has 10 000 samples. I would like to ask if it is normal, that after retrainig model with these 10 000 unlabeled samples, accuracy didn't chaned as well as confusion matrix has same values? I expected some changes (better or worse prediction). Thank you in advance.


texts: How many instances should be synthesized for each class when using over-sampling techniques?

texts:
As for an imbalanced multi-class dataset, how many instances should be synthesized for each class if we use over-sampling techniques such as SMOTE?
For example, there is 4 class including 'A', 'B', 'C' and 'D'. The number of samples for each class is as follows:

A: 5
B: 50
C: 100
D: 500

Are the following methods meaningful?

Make all class contain 500 instances.
Set a threshold and over-sample the class number of whose instances is smaller than it. For example, set threshold = 100, and make A and B to 100.


texts:
I'm relatively new to data science, having done some machine learning projects at work. My background is in R, although I have some experience in Python. I'm considering choosing one language and getting really good at using it for data science / machine learning.

Now, I've read various articles touting Julia's strengths - namely ease of use and massive speed benefits. It sounds almost too good to be true!

So my question is, is there any particular reason(s) why I should choose Julia over R/Python?

Or alternatively, is there any reason(s) why I shouldn't choose Julia over R/Python?


texts: Mathematic explanation needed for a univariate classification method based on solving a quadratic equation

texts:
I read a piece of codes on classifying image hue values into three classes with derived thresholds. The thresholds are calculated by simply using a quadratic formula. The related documentations for the codes are as follows:

/*
*       (v_B - v_A)x^2 + 2(m_B*v_A - m_A*v_B)x + (m_A^2*v_B - m_B^2*v_A) = 0
*
*       v_A : variance of group A
*       m_A :   mean of group A
*       v_B : variance of group B
*       m_B :   mean of group B
*
*       Quadratic formula
*       ax^2 + bx + c = 0
*       a = v_B - v_A
*       b = 2(m_B*v_A - m_A*v_B)
*       c = (m_A^2*v_B - m_B^2*v_A)
*
*       x1 = (-b+sqrt(b^2-4ac))/2a
*       x2 = (-b-sqrt(b^2-4ac))/2a
*
*/

Here above is to find a threshold to delimit the class A and B, based on solving a quadratic equation involving the mean and variance of the two groups, and the threshold is one of the solutions (i.e., x1 and x2) depending on their signs.

I haven't seen such a quadratic equation based classification method before. Anyone know what formal name for the method? It, on the first sight, seems to me a simplified case of quadratic discriminant analysis, but not sure about that. I need a little bit mathematic background for the method. Educate me if you know that. Thanks!


texts: How is the input gate in the LSTM learn?

texts:
How is the input gate neural network trained what to remember by propagating the error rate from predicting the next word in the language model? How does it help it to learn if it remembered the right data or not?
I imagine maybe doing it heuristically. For example, when the full sentence is -

"It takes all the running you can do, to keep in the same place. If
you want to get somewhere else, you must run at least twice as fast as
that!"

And the input to the language model is -

"It takes all the running you can do, to keep in the same place. If
you want to get somewhere else, you must "

But it outputs "go" for the next word instead of "run" I can see that it forgot and didn't save the "running" word earlier. So in this scenario, I can take the stem of the next word which is "run" and search it backward in the sentence to see where it forgot (or didn't save) the "running" word, then I can penalize the NNs and train them like that.
What do you think?
What am I missing here?


texts:
I am trying to build a news tagging system. Given a piece of news article, find 5-6 key terms from the news article that best describe the article. Refer to the image below from google news.

What are some approaches I can look at to get human readable tags ?


texts: How to choose the best hyper-parameter when it is directly influenced by the random_state?

texts: While trying to evaluate my Ridge Regression model and using GridSearchCV to find the best parameter. I noticed that the best estimator changes every time I change the random_state in my KFold object (cv parameter). With this in mind how do I choose the most optimal hyper parameter to implement my model.


texts:
Currently working on resume parser and struggled with embedding words with '-' symbols in them. Such as 'IT-manager'.

Vector representations of these words are incorrectly classified by doc2vec.

  ['it-manager']
  [('salary', 0.23328335583209991), ('responsibilites', 0.22327110171318054), ('schedule', 0.14869527518749237), ('position', 0.12755176424980164)]

But when I remove '-' symbol, it is tokenized and classified right.

  ['it', 'manager']
  [('position', 0.9306046962738037), ('schedule', 0.6630333662033081), ('responsibilites', 0.6081600189208984), ('salary', 0.5934453010559082)]

How do you work with such data properly? For this kind of task, I guess, it is better to exclude the symbol. But there may be a way to tell Doc2vec to treat these words like two different ones. Or perhaps tell the word_tokenizer to tokenize them in this fashion?


texts: Find points on a map close to given points

texts:
I have a locus L of points (lat, long). And I would like to find N=10 points (let's call them warehouses) such that:

$$loss = \sum_{l \in L} maximum_{w \in W}(distance(l, w))^2 $$

is minimized.

Is there a documented algorithm or approach that solves this problem? Right now I am thinking Excel may be able to handle this task. However I have too much data for Excel and will need to implement this in Python / Pandas.


texts:
From the paper, lightGBM does a subsampling according to sorted $|g_i|$, where $g_i$ is the gradient (for the loss function) at a data instance.

My question is that, when the objective is L1 loss/regularization, $loss_i = |y_i - \hat{y_i}|$, hence $g_i$ is either 1 or -1 (assume no zero loss). Therefore the absolute value of gradient is 1 for any data instance. How to sort then and select instances for the subsample? Or does lightGBM skip the subsampling process if L1 regularization is selected?


texts: How to use MNIST dataset to make predictions on similar images (colorblindness charts)?

texts:
I am trying to use the MNIST dataset to train a convolutional neural network to classify digits written in colorblindness charts. As some people have suggested, I have tried playing with the brightness and contrast, as well as converting to grayscale, but all of these have inconsistent results because all charts are very different.

I am looking for general ideas about approaches I can try. Would style transfer make sense? Here is an example of an easy chart I want to be able to classify:

I tried various transformations and got results such as these. Although to me, a mildly colorblind person, these are easier to see, it is not nearly close to MNIST.

TLDR:
I don’t have a dataset of color blindness charts. MNIST is readily available however. I’m trying to somehow use MNIST (probably after transforming it) to make the neural network classify the limited color blindness charts I have. MNIST and the charts are different and I need to make them similar so that the NN after training on modified MNIST can predict color blindness charts.

EDIT:

I tried the suggestions and applied various OpenCV Transformations to images. At the end, everything is resized and converted to grayscale. My accuracy went up from 11% (on non processed images) to 33% on the new images.

My problem is finding a set of transformations that are universal. My transformations work well for some images:

Some are not so good:

How can I improve this? Here are the transformations I do:

image = cv2.imread(imagePath)
    image = imutils.resize(image, height=400)
    contrasted_img = CONTRASTER.apply(image, 0, 60) #applies contrast
    median = cv2.medianBlur(contrasted_img,15)
    blur_median = cv2.GaussianBlur(median,(3,3),cv2.BORDER_DEFAULT)
    clustered = CLUSTERER.apply(blur_median, 5) #k-means clustering with 5
    gray = cv2.cvtColor(clustered, cv2.COLOR_BGR2GRAY) #RESULT

SECOND EDIT:

I used written advice and calculated that the digit usually takes up between 0.1 and 0.35 ish of the image. So I increase threshold for black and white until that happens. This resulted in images like this:

My accuracy of the NN went up to 45%! Another amazing improvement. However, my biggest issue is with images where the digit is darker than the background:

This results in incorrect thresholds. I also have some noise issues, but these are less common and can be fixed with tuning my percent white and threshold:


texts: Pandas Conditional Fill NaN Forward/Backward

texts:
Updated 22 Oct. 2018:
I have the following dataset:
data = [('D',1,10,8),
           ('D',2,12,12),
           ('X',1,28,np.NaN),
           ('D',3,np.NaN,np.NaN),
           ('X',2,np.NaN,25),
           ('X',3,32,25),
           ('T',1,220,np.NaN),
           ('X',4,30,np.NaN),
           ('T',2,240,np.NaN),
           ('X',2,38,np.NaN),
           ('T',3,np.NaN,np.NaN),
           ('T',4,200,150)]

labels = ['item', 'month','normal_price','final_price']

df = pd.DataFrame.from_records(data, columns=labels)

    item    month   normal_price    final_price
0   D       1       10.0            8.0
1   D       2       12.0            12.0
2   X       1       28.0            NaN
3   D       3       NaN             NaN
4   X       2       NaN             25.0
5   X       3       32.0            25.0
6   T       1       220.0           NaN
7   X       4       30.0            NaN
8   T       2       240.0           NaN
9   X       2       38.0            NaN
10  T       3       NaN             NaN
11  T       4       200.0           150.0

I want to fill NaN in the 'normal_price', 'final_price' columns for each item with the 'normal_price', 'final_price' of its preceding month (if not available by its succeeding month). I have tried using this:
df[['normal_price','final_price']]=df[['normal_price','final_price']].fillna(method='ffill')

but it gives me this:
    item    month   normal_price    final_price
0   D       1   10.0                8.0
1   D       2   12.0                12.0
2   X       1   28.0                12.0*
3   D       3   28.0*               12.0
4   X       2   28.0                25.0
5   X       3   32.0                25.0
6   T       1   220.0               25.0*
7   X       4   30.0                25.0
8   T       2   240.0               25.0*
9   X       2   38.0                25.0
10  T       3   38.0*               25.0*
11  T       4   200.0               150.0

The problem is with cases with an asterisk (I've also tried 'bfill'). These values should be filled with based on their correct items. Ideally, I should get:
    item    month   normal_price    final_price
0   D       1   10.0                8.0
1   D       2   12.0                12.0
2   X       1   28.0                25.0
3   D       3   12.0                12.0
4   X       2   28.0                25.0
5   X       3   32.0                25.0
6   T       1   220.0               150.0
7   X       4   30.0                25.0
8   T       2   240.0               150.0
9   X       2   38.0                25.0
10  T       3   220.0               150.0
11  T       4   200.0               150.0

I have also tried the followings (from answers offered by date):
df[['normal_price','final_price']].ffill(limit=1).bfill(limit=1)

or
df[['normal_price','final_price']]=df[['normal_price','final_price']].interpolate(method='nearest')

But none of them are giving me reasonable fillna corresponding to each item. I have found this method:
df[['normal_price','final_price']]=df[['normal_price','final_price']].fillna(df.groupby(['item'])[['normal_price','final_price']].transform('mean'))

It works better, BUT it introduces unpredictable values (in this case the 'mean') for NaN values, not with the preceding or following values as I originally wanted. I am trying to combined the df.groupby(['item']) concept with '.ffill' or '.bfill', but so far no success.


texts: LSTM forcasting: time series as input and a unique value for each as output

texts:
I am sorry for the mistakes I will make in this post, I'm new to deep learning ^^'

I am trying to build an LSTM model that can help me predict some unique value according to time series indices and previous time series indices correlated with the value.

To make it simple, I have an indice that varies throughout time and "place", it will be my input, and a resulting value (the output) that changes only once a year.

Bellow is a screen shot of the data:

I thought of making this problem a classification problem, but the way I understood classification in this course, is to put (previously?) some values in an "output interval"

I have also tried to follow this tutorial, but I'm getting some weird results (since maybe I have changed the datetime into numeric data?), and a huge rmse of (above 70%)

Thank you for your time and advices:)


texts:
I have been working on an image classification tasks for which I am extracting the image frames from the video stream collected for different classes.

I have already trained an image classification model (using transfer learning) however due to the outliers (or overlap in the class distribution) accuracy of the model is poor. And not able to generalize the new images / video streams.

Could you please help me with the below queries

How the sample is distributed in each class ? Can I use any visualization techniques (for example : histogram) to see the sample distribution.
And also going through the image one by one is tedious process so is there a technique with which I can find the outliers (outlier images) from the samples. So that I can remove outliers before training the model.

Any updates on this..

Thank you


texts: How do you identify whether your RMSE score is good or not?

texts:
Im building a XGBoost regression model to predict the values in the range of -3 to 3. Im using Root Mean Squared Error to evaluate the model. With hyper-parameter tuning and everything the best scores I could get was this:

training - 0.118
validation - 0.3207 (5 folds)
testing - 0.3018

I know for the regression task, the RMSE values should be as less as possible. But however, I couldn't reach the range of 0.2 atleast for the test score.
Im wondering does my validation and test score range of 0.3 has anything to do with my -3 to 3 continuous target range?
Any quick help here is much appreciated. Thank you


texts: Multiclass classification problem with more prediction classes than real classes

texts:
Can I have a multiclass classification problem with more prediction classes than real classes?

For example: I want to predict the channel the user is going to watch. The real classes are "user didn't see tv" and "user saw channel C". The prediction classes are "user will not watch tv", "user will watch channel C", "user will watch one of two channels: C or D".

Is this a good formulation of a classification problem?

How can I calculate the performance metrics as recall, precision, and F1-score per class?

Thanks


texts: What is the best approach to find the distance of a person/people from camera, using an image?


texts:
I am new to Keras and ML and I want to create a NN that can seperate a bitmap-like image into its visual components.
My approach is to feed a two dimensional image (lets say 8 by 8 pixels) into a NN, which then outputs a three dimensional matrix (e.g 3 by 8 by 8). Every element of the first dimension represents an image of a component (see this illustration).
As a first version I created a Keras sequential model with 1 flatten layer and 2 dense layers, 192 units each (3x8x8=192). After around 5000 training images, the model performance was still mediocre.
Question: Is there a way of splitting the output of the last dense layer into three seperate images that will then be processed in a parallel manner by further dense layers? Is there a better approach to the problem I am facing?


texts: Accuracy gain vs amount of data in Neural Networks

texts:
There's a theoretical question I tackled upon in the excellent book Neural Networks and Deep Learning by Michael Nielsen, which I would love to discuss about.
The question is:

How do our machine learning algorithms perform in the limit of very
large data sets? For any given algorithm it's natural to attempt to
define a notion of asymptotic performance in the limit of truly big
data. A quick-and-dirty approach to this problem is to simply try
fitting curves to graphs like those shown above, and then to
extrapolate the fitted curves out to infinity. An objection to this
approach is that different approaches to curve fitting will give
different notions of asymptotic performance. Can you find a principled
justification for fitting to some particular class of curves? If so,
compare the asymptotic performance of several different machine
learning algorithms.

Regarding a justification for fitting particular class of curves, empirically by viewing several datasets I've seen that usually if there's an accuracy gain by more data, the accuracy gain is linear in the amount of data.
Of course that doesn't always hold truth because it depends on the model. It has to be big and complex enough to fit and "benefit" from the additional data.
The above is just an assumption, and would love to hear more solid opinions.
Regarding the second question of comparing asymptotic performance of several machine learning algorithms, I didn't fully understand the question, but here's a comparison of SVM and FC network from the book:


texts: Does variance and standard deviation both measure how spread out the numbers are?

texts:
I've heard from different sources that, standard deviation measures how spread out the numbers are. But I've also heard the same for variance.

Is it technically correct to say this statement for both std and var?


texts: Why is everybody using `mobilenetv2` for mask detection?

texts:
I was looking for good pre-trained models to be used for mask detection and I found resnet50 and mobilenetv2 (lots of times).
The implementation I liked the most is the following: https://github.com/balajisrinivas/Face-Mask-Detection/
I wonder why is this model being used too many times for problems like this? I read it's very soft. Is there any other special point?


texts: How to fix "Expected sequence or array-like"

texts:
I am trying to get the accuracy of the model and I am getting this error

  TypeError: Expected sequence or array-like, got

Here's my code sample.

%matplotlib inline
import matplotlib.pyplot as plt
import pandas as pd

dataframe = pd.read_csv("WA_Fn-UseC_-HR-Employee-Attrition.csv")

dataframe.head(2)

# spare input and target variables
inputs = dataframe.drop('PerformanceRating', axis='columns')

target = dataframe['PerformanceRating']

from sklearn.preprocessing import LabelEncoder

MaritalStatus_ = LabelEncoder()
JobRole_ = LabelEncoder()
Gender_ = LabelEncoder()
EducationField_ = LabelEncoder()
Department_ = LabelEncoder()
BusinessTravel_ = LabelEncoder()
Attrition_ = LabelEncoder()
OverTime_ = LabelEncoder()
Over18_ = LabelEncoder()

inputs['MaritalStatus_'] = MaritalStatus_.fit_transform(inputs['MaritalStatus'])
inputs['JobRole_'] = JobRole_.fit_transform(inputs['JobRole'])
inputs['Gender_'] = Gender_.fit_transform(inputs['Gender'])
inputs['EducationField_'] = EducationField_.fit_transform(inputs['EducationField'])
inputs['Department_'] = Department_.fit_transform(inputs['Department'])
inputs['BusinessTravel_'] = BusinessTravel_.fit_transform(inputs['BusinessTravel'])
inputs['Attrition_'] = Attrition_.fit_transform(inputs['Attrition'])
inputs['OverTime_'] = OverTime_.fit_transform(inputs['OverTime'])
inputs['Over18_'] = Over18_.fit_transform(inputs['Over18'])

inputs.drop(['MaritalStatus', 'JobRole', 'Attrition' , 'OverTime' ,
                'Gender', 'EducationField', 'Department', 'BusinessTravel', 'Over18'], axis=1, inplace=True)

inputs.rename(columns={'MaritalStatus_':'MaritalStatus',
                       'JobRole_' : 'JobRole',
                       'Gender_' : 'Gender',
                       'EducationField_' : 'EducationField',
                       'Department_' : 'Department',
                       'BusinessTravel_' : 'BusinessTravel',
                       'Attrition_' : 'Attrition',
                       'OverTime_' : 'OverTime',
                       'Over18_' : 'Over18'}, inplace=True)

inputs.head(2)

from sklearn import preprocessing

scaler = preprocessing.StandardScaler()
scaled_df = scaler.fit_transform(inputs)
scaled_df = pd.DataFrame(scaled_df, columns=inputs.columns)
scaled_df.head()

from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
import numpy as np

inputs = scaled_df
X_train, X_test, y_train, y_test = train_test_split(inputs, target, test_size=0.2)

model = RandomForestClassifier(ccp_alpha=0.01)

# train
pred = model.fit(X_train, y_train)

from sklearn.metrics import accuracy_score
accuracy_score(X_test,pred) #--------------> this is where I get the error

Can someone help me?


texts: Training a function that maps n-dim to n-dim

texts:
As an example, say the input is an array of numbers representing an audio snippet and the output is a transformed/filtered version of it.

What would be the proper name for that? Which are examples of algorithms for the job?

EDIT:
More specifically, I want to train audio source separation. The input is a mixed sound (spectrogram) and the output is the sound with some energy removed in certain frequencies. The function needs to recognize some pattern in the input and decide what to remove.


texts: Categorical Variable and Target Variable

texts:
Though a similar question is answered here , but I wanted to take a different approach. Assuming that I have a binary target variable 1/0 and a categorical variable Gender M/F. From this, I can have a proportion p1 for M with target 1 and p2 for F with target 1. N1 and N2 would be total number of M and F respectively.
Is it prudent to run a test for H0: p1=p2 ? If H0 cannot be rejected, doesn't it indicate that Gender doesn't have any correlation with Target


texts: Is it worth to upgrade CUDA and cuDNN while having older GPUs?

texts:
New CUDA 11.x versions add support for TF32 format, other new features for newer cards (RTX30xx, A100 etc).
Is it worth upgrading to CUDA 11.x if you have GTX 1050 or RTX 2080 (having tensor cores)?
Could it be that new features only add computational overhead (at least in the size of installation file, they do), and an older GPU won't be able to use the new features?


texts: Are there any frameworks available that allow for automated large scale supervised machine learning?

texts:
The typical steps for solving a machine learning/pattern recognition problem:

Data Analysis and splitting the data into test and train sets.
Choosing a model.
Training the model, and testing the model against the test set.
If the accuracy of the model is not acceptable, start over with a new model.

Step (2) can be automated to some extent using a grid search, but are there ways of automating the whole process?

I'm thinking in particular of large scale data applications (for example when dealing with retail data, or customer analytics on a site like Netflix, ) where there are millions of instances of similar but distinct machine learning problems that each need to be trained and validated separately.

In such a situation it is impossible for a team of analysts or data scientists to perform the above steps and some sort of automated model development framework must be used.

What are the frameworks that allow for this?


texts: Is there any logic to adding a threshold to see if two variables are related?

texts:
I have two variables $X$ and $Y$ given as tuples of $(x, y)$, and I want to see if there is a relationship between the two variables. I can do so by finding the correlation coefficient.

However, I found that by selecting an arbitrary subset of the data (e.g. $(x, y) | x > k$ ), I can get a higher correlation coefficient and a stronger result. Is doing so mathematically sound? I have no a priori reason to believe that certain data points are "more important" than others, to put it simply.


texts:
I do not undestand, how the deployment of a ML-model works in the reality.
A given dataset needs to be mostly time pre-processed (for example One Hot Encoding).
After will be a model cretead and stored.
As next will be the model with Streamlit deployed. I mean the option when the new data could be uploaded and a "Label" column will be generated - predicted values.
Here is the point. The new data are not being pre-processed (for example One Hot Encoding) and they are in the form as they were before modelling.
The model deployment is mainly used from the customers and they do not know "about pre-processing".
Have I something misunderstood or could someone bring more light into this topics?


texts: Calculating Uncertainty for categorical predictions

texts:
I am wondering what is the best way to calculate the uncertainty for my categorical predictions. I have created a model that predicts what rating a movie is getting based on certain keywords and the amount of times these key words are mentioned. I have trained the model with a small data matrix and now I am running it though a bigger data matrix that has data from over 20,000 reviews. I have used different models and different variations of my dataset (feature engineering and feature selection) and now that I have my results from my best model/dataset combination, I want to learn how to calculate the uncertainty of my results per rating. Could I use the confusion matrix or other metrics (sensitivity etc) to calculate the uncertainty?
Example Predictions:
Rating1:   1000
Rating2:   25
Rating3:   59
Rating4:   599
Rating5:   3569
Thank you!


texts: DQNs for huge or continuous state spaces

texts: Have there been occasions where DQN failed to deal with huge state spaces? Can you point out a research paper regarding it?


texts: How to perform Anomaly Detection on a force profile?

texts:
I have a set of force profiles of an industrial machine. I'm trying to develop an algorithm that tries to understand when a new profile is "anomalous" with respect to the ones in "normal operating conditions".
In the picture below you can see the force profiles (function of time). I want the blue curve found checked as anomalous.
What approaches do you suggest? I'm thinking about using some statistical distance (like mahalanobis) to check the similarity of a new curve to the "mean" of the others.
Another point: it can be useful to use approaches like PCA in this case? Then I can try to use clustering techniques to separate anomalous profile from normal ones. I'm not sure since I have a great number of observations but of the same variable and PCA is a multivariate technique.
]1


texts: Would Deep Q Learning work for a finite horizon problem?

texts:
I want to apply Deep Q Learning to a problem, which has a clear finite horizon definition, like:

$$V(s) = \mathbb{E}[r_1 + r_2]$$

Since the horizon is finite, I do not use reward discounting. My action spaces at the time steps 1 and 2 do change and also the policies are therefore clearly not stationary as well. But the state and reward transitions are deterministic as: $r_i(s_i,a_i)$ and $s_{i+1}(s_i,a_i)$. The state space is also very large, so tabular methods are out of question. So the value function can be defined as:

$$V(s) = \sum_{a_1}\sum_{a_2}\left(r_1 + r_2\right)\pi_1(a_1|s_1=s)\pi_2(a_2|s_2(s_1=s,a_1))$$

since the only randomness comes from the policy distributions. (I dropped the reward functions' depedency on the states and actions for clarity). My question is, would Deep Q Learning work for such a finite horizon case? I plan to use two separate MLPs for the Q functions at time steps 1 and 2. I know that the Bellman Optimality can be shown both for the finite and infinite cases; in the finite case, it does not even need the stationarity of the distributions. But all examples on DQNs I came accross during my research solves problems with the infitine horizon assumption. So I just wanted ask for some insight there.


texts: Information compression for variable input size

texts:
Is there a way to compress information of a variable input size?

Autoencoder requires standardized input sizes. Although I can add masks on the cost function and add dummy features to standardize input/output size, I am hesitant with the potential drawbacks.

The input structures I am interested in are graphs and images. If input sizes and shapes vary too much, padding, resizing and rescaling do not work.


texts: Loss function to compare non binary segmentation

texts: I need to compare two images corresponding to landmark locations. I was thinking of something related to Dice loss. I cannot use dice loss since the image is not binary. The background is black but the landmark locations have varying intensities depending on the distance. What would be the best loss function to compare such images? Is cross-entropy suitable for this?


texts: Autoencoders for the compression of time series

texts:
I am trying to use autoencoder (simple, convolutional, LSTM) to compress time series.

Here are the models I tried.

Simple autoencoder:

    from keras.layers import Input, Dense
    from keras.models import Model
    import keras

    # this is the size of our encoded representations
    encoding_dim = 50

    # this is our input placeholder
    input_ts = Input(shape=(2100,))
    # "encoded" is the encoded representation of the input
    encoded = Dense(encoding_dim, activation='relu')(input_ts) #, activity_regularizer=regularizers.l2(10e-5)
    # "decoded" is the lossy reconstruction of the input
    decoded = Dense(2100, activation='tanh')(encoded)

    # this model maps an input to its reconstruction
    autoencoder = Model(input_ts, decoded)

    # this model maps an input to its encoded representation
    encoder = Model(input_ts, encoded)

    # create a placeholder for an encoded (%encoding_dim%-dimensional) input
    encoded_input = Input(shape=(encoding_dim,))
    # retrieve the last layer of the autoencoder model
    decoder_layer = autoencoder.layers[-1]
    # create the decoder model
    decoder = Model(encoded_input, decoder_layer(encoded_input))

    autoencoder.summary()

    adamax = keras.optimizers.Adamax(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.01)
    autoencoder.compile(optimizer=adamax, loss='mean_absolute_percentage_error')

Convolutional autoencoder:

    from keras.layers import Input, Dense, Conv1D, MaxPooling1D, UpSampling1D
    from keras.models import Model

    window_length = 518

    input_ts = Input(shape=(window_length,1))

    x = Conv1D(32, 3, activation="relu", padding="valid")(input_ts)
    x = MaxPooling1D(2, padding="valid")(x)

    x = Conv1D(1, 3, activation="relu", padding="valid")(x)

    encoded = MaxPooling1D(2, padding="valid")(x)

    encoder = Model(input_ts, encoded)

    x = Conv1D(16, 3, activation="relu", padding="valid")(encoded)
    x = UpSampling1D(2)(x)

    x = Conv1D(32, 3, activation='relu', padding="valid")(x)
    x = UpSampling1D(2)(x)

    decoded = Conv1D(1, 1, activation='tanh', padding='valid')(x)

    convolutional_autoencoder = Model(input_ts, decoded)

    convolutional_autoencoder.summary()

    optimizer = "nadam"
    loss = "mean_absolute_error"

    convolutional_autoencoder.compile(optimizer=optimizer, loss=loss)

LSTM autoencoder:

    from keras.layers import Input, LSTM, RepeatVector
    from keras.models import Model

    inputs = Input(shape=(1, 500))
    encoded = LSTM(128)(inputs)

    decoded = RepeatVector(1)(encoded)

    decoded = LSTM(500, return_sequences=True)(decoded)

    sequence_autoencoder = Model(inputs, decoded)
    encoder = Model(inputs, encoded)

    sequence_autoencoder.summary()

    sequence_autoencoder.compile(optimizer='nadam', loss='mean_absolute_error')

To check for compression loss, I use the SMAPE formula.

I got such results. The average loss for simple autoencoder is 14.28%, for convolutional autoencoder is 8.04%, for LSTM-autoencoder is 9.25%.

My question is: is it practical to compress time series with losses using a neural network if the compression time does not matter? Perhaps i should pay attention to other methods? How can I achieve better compression?

Thanks!


texts: Why is it the Most significant feature from xgboost model is not at all significant when i build a model with same data set but using catboost?


texts: How to cluster when samples are independent and/or dimensions are not independent for each sample?

texts:
In machine learning, one can use Euclidean distance to measure a cluster

$$
\mu\in R^k,
$$

over data points
$$
\{x_i\}^N_{i=1}\in R^{k\times N},
$$
with the measure
$$
\text{error}_i = ||\mu-x_i||^2_2
$$
The total error can be calculated as
$$
\text{error} = \sum^N_{i=1}error_i
$$
To estimate the parameters, one formulates the optimization problem
$$
\min_{\mu_{new}} \big{(}\text{error}\big{)}
$$
This has two underlying assumptions: first, that samples are independent, and, second, that the dimensions are independent for each sample.

Can anyone explain how I should analyze when either or both assumptions are not satisfied?


texts: Binary Classification on small dataset < 200 samples

texts:
I have a dataset consisting of 181 samples(classes are not balanced there are 41 data points with 1 label and rest 140 are with label 0) and 10 features and one target variable. The 10 features are numeric and continuous in nature. I have to perform binary classification. I have done the following work:-

I have performed 3 Fold cross validation and got following accuracy results using various models:-

LinearSVC:
0.873
DecisionTreeClassifier:
0.840
Gaussian Naive Bayes:
0.845
Logistic Regression:
0.867
Gradient Boosting Classifier
0.867
Support vector classifier rbf:
0.818
Random forest:
0.867
K-nearest-neighbors:
0.823

Please guide me how could I choose the best model for this size of dataset and make sure my model is not overfitting ? I am thinking of applying random under sampling to handle the unbalanced data.


texts: Can neural network make from Small images make big

texts:
I have several images of one artist, but they are small size 300x200 px
can Neural Network somehow restore it to a big size like 1024x768 if, for example, I have only one big picture from this artist?


texts: Ideas on handling large quantities of "waste" data

texts:
Hello and thank you for taking time out of your day to help me. I am currently working on developing a machine vision application for production monitoring. The application handles images, about 20,000 images everyday are being processed and I want to come up with an interesting way to handle all this information.
Let me first explain what the processing of every images does. The application is being implemented on a production line, when an item passes infront of my camera, a photograph is taken and then through some image pocessing the application decides if the item is properly packaged and perfoms some actions accordingly. After that, the image is localy stored and in the end of the day every image is deleted to clear space for the next day.
Now, this system works good enough but I was thinking that maybe there is more creative way to handle the "waste" images. Do you think that creating a cloud based database with a statisticaly significant number of images of everyday could be usefull, and if yes do you have any statistical models to suggest?


texts:
I have a problem that doesn't seem to fall into a common machine learning category, and I was wondering if this still could potentially be solved with ML.

Problem: I have two signals recorded from two sensors, and would like to determine whether they are correlated (i.e. record the same physical event) or not.

The catch: I don't have access to the full signal time series of both sensors, but only one at a time - I can only exchange a small descriptor on the order of 32 bits to see if the signals match or not.

Our current approach is to calculate a bunch of numerical signal features such as mean, derivative, zero-crossings, FFT etc. and see which ones provide the best correlation - but that seems to be a lot of guesswork and doesn't work very well in any case.

So now I had the following idea:

Start with a neural network which takes a fixed window out of the signal (+ possibly the FFT of that window) as an input, and produces a 32-bit output
Pick two random correlated samples out of the pool of examples, and run the network twice, once with each sample (and its FFT)
Take the difference between the two output values as error measure and perform backpropagation as usual
Repeat from 2. until the difference for all examples is below a threshold

Here are my questions:

Does this approach seem feasible at all?
As someone relatively new to machine learning, how would I implement this?
I've had a look at Keras - would this be a suitable starting point?

Thanks in advance, and best regards, Florian

Addendum: I've found this somewhat related post (Is it possible using tensorflow to create a neural network that maps a certain input to a certain output?), but I don't think that this is the same problem, as I don't actually care what the output looks like, just that it is as unique as possible for each matching pair of samples.


texts: Two-tower net does not learn when made deep

texts:
I have been trying to train a relatively simple two-tower net for recommendation. I am using PyTorch and the implementation is the following - basically embeddings layers for users and items, optional feed-forward net for both towers, dot product between the user and items representations, and sigmoid.
class SimpleTwoTower(nn.Module):

    def __init__(self, n_items, n_users, ln):
        super(SimpleTwoTower, self).__init__()

        self.ln = ln
        self.item_emb = nn.Embedding(num_embeddings=n_items, embedding_dim=self.ln[0])
        self.user_emb = nn.Embedding(num_embeddings=n_users, embedding_dim=self.ln[0])

        self.item_layers = [] #nn.ModuleList()
        self.user_layers = [] #nn.ModuleList()

        for i, n in enumerate(ln[0:-1]):
            m = int(ln[i+1])
            self.item_layers.append(nn.Linear(n, m, bias=True))
            self.item_layers.append(nn.ReLU())

            self.user_layers.append(nn.Linear(n, m, bias=True))
            self.user_layers.append(nn.ReLU())

        self.item_layers = nn.Sequential(*self.item_layers)
        self.user_layers = nn.Sequential(*self.user_layers)

        self.dot = torch.matmul
        self.sigmoid = nn.Sigmoid()

    def forward(self, items, users):

        item_emb = self.item_emb(items)
        user_emb = self.user_emb(users)

        item_emb = self.item_layers(item_emb)
        user_emb = self.user_layers(user_emb)

        dp = self.dot(user_emb, item_emb.t())
        return self.sigmoid(dp)

I am trining with Binary cross entropy loss and Adam optimizer. When I am using only the embeddings, I see improvements from epoch to epoch (loss is decreasing and the evaluation metric are increasing). However, once I add even a single feed-forward layer, the network learns just a bit in the first epoch and then stagnates. I have tried to had code one linear layer with ReLU, to check if the issue is with the way I am creating the list of layers, but this did not change anything.
Has anybody else had a similar problem?
Edit: Here I have posted the question in the PyTorch forum and I have some replies.


texts: Which learning algorithms to use in what order - dimensionality reduction, bayesian network structure, regression?

texts:
The data is a huge set of observations of dozens of variables, all (potentially, somehow) related to a dichotomous outcome variable, and all (potentially) correlated to each other, or to unknown / unobserved things.

I think that "merely" applying logistic regressions or any related extension or improvement (such as Classification And Regression Trees etc.) might not be sufficient, because there is a "basic" bayesian network structure that seems to exist (based on expert opinion) - this means that the observed variables are somehow describing distinct nodes in a bayesian network, but the exact formula HOW they describe these nodes is unclear, as is the eventual predictive power of these nodes towards the outcome.

Here is this exact problem in a highly simplified example:

We have two runners who are going to compete in a 400m sprint against each other. We have a dataset of observations of thousands of such competitions with two runners, each observation containing, for each of the two runners, (a) age, (b) size, (c) number of training runs in the past 6 weeks and (d) average mileage per training run in the past 6 weeks.

Now, we believe that (a) and (b) will determine (e) "talent for sprinting" and (c) and (d) will determine (f) "current form", and that (e) and (f) together will give the probability of winning - but we do not know HOW (a) and (b) determine (e) or how (c) and (d) determine (f) (e.g., is it "age times size" or is it "age times 2 while age lower than 45" etc.). Further, we also do not know how (e) and (f) together determine the probability of winning...

Is there any "known" approach to such a problem, i.e. a way to determine the formula that connects the initial variables to the later variables that determine the outcome, and then further determining the predictive power of the resulting network?

Or is that unnecessary (i.e. using the original variables only in some algorithm - which one? - should give the same result)? Or is it impossible (too many unknown things)? What should I do / read about / learn to solve this problem?

And in addition, is there a way to CHECK whether my original assumption (that (a) and (b) determine (e) etc.) was correct, e.g. via some sort of clustering algorithm, PCA, ...?

What would you do to approach such a question?

Thank you very much indeed in advance for helping a rather new data science learner by pointing me into the right direction!


texts: Association Rules with Python (coded dataset)

texts:
I have this dataset which I really need to use association rules techniques on. The dataset has like 90 variables, many of which are ordinal. Thing is, the data is already coded using numbers instead of strings (e.g. bread = 4 instead of "bread") as well as some re-scaled numerics such as 1 = 1%-10%".
What I have so far:
from apyori import apriori
#Convert dataframe to list
val_list= []
for row in range(1,5530):
    val_list.append([str(data.values[row,column]) for column in range (0,90)])
    print('Row ', row, ' ok')

apr = apriori(val_list,min_support=0.1,min_confidence=0.2,min_lift=2)
result = list(apr)

Still, this way I don't get the feature names in the frequent "baskets" so it's not much use, since I have baskets like [33, 1, 8, 8, 1, 1] with no idea what the numbers might be referring to. What can I do and/or how do I prepare the data for association rule mining?


texts:
I had to convert a code written in pytorch to keras (with tensorflow backend). But there was this layer called CrossMapLRN2d which had no direct counterpart in Keras. So wanted to know what does this layer do and how to implement it in keras.
The exact line of code was

nn.CrossMapLRN2d(size=5, alpha=0.0001, beta=0.75, k=1.0)


texts: Choosing input variables for Linear Regression for higher accuracy

texts:
I have a data set describing water levels of rivers. It has following attributes:

Value                   float64
Variable Name            object
Variable Unit Code       object
Date Pulled              object
Site Code                 int64
Site Latitude           float64
Site Longitude          float64
Site Name                object

I am using Linear Regression in trying to predict the Value using Site Code, Site Longitude and Site Latitude but getting results which are no way near the actual values. I am wondering if it's my choice of independent variables that is creating this problem or should I try other types of Regression? If it's the first case, which approaches are recommended to get higher accuracy rate or more precisely, what should one do if there are not many relevant independent variables?


texts:
In the original BERT paper, section 'A.2 Pre-training Procedure', it is mentioned:

The LM masking is applied after WordPiece tokenization with a uniform masking rate of 15%, and no special consideration given to partial word pieces.

And in the RoBERTa paper, section '4.4 Text Encoding' it is mentioned:

The original BERT implementation (Devlin et al., 2019) uses a
character-level BPE vocabulary of size 30K, which is learned after
preprocessing the input with heuristic tokenization rules.

I appreciate if someone can clarify why in the RoBERTa paper it is said that BERT uses BPE?


texts: Create a single time series plot of multiple devices

texts:
I have a dataset where there are stored the measurements of 30 devices. Each device has about 4000 values and it is structured as well:
device01 0.02;0.13;1.15;0.10;8.30;........;4.20
device02 0.06;0.13;1.40;0.03;7.40;........;6.30
........
device30 0.03;0.24;1.10;0.43;4.40;........;2.30

Each value is got each X seconds (I don't have the timestamp reference, I only know they are got through time). How can I plot them to get a single time series plot like the one in this picture?

As you can see lots of the values do overlap, how can I make them consecutive in the plot?


texts: How to restructure my dataset for interpretability without losing performance?

texts:
What I am doing:
I am predicting product ratings using boosted trees (XGBoost) with a dataset in this format:

What I want to do:
I want to use SHAP TreeExplainer to interpret each prediction my model gives in terms of product attributes and user ids.
What I am getting:
My model is drawing all the conclusions based on product names and user ids, instead of product attributes and user ids.
What I tried:
I discovered that each product name has a unique combination of product attributes, i.e. by knowing the product attributes you can find its name. So my idea was to remove the product_name column, leaving only the attributes.
My reasoning was that restructuring the dataset in this way would lead to the interpretability that I wanted without any performance loss (since the product name doesn't add any new information).
What I got:
The model performance decreased a lot. Even with a great deal of hyperparameter tuning, I couldn't get near the performance I had when also using the product name.
What I think maybe going on:

My dataset is too small for the model to learn with the product attributes (10k samples, 60 attributes).

or

Maybe there are some attributes adding bias and screwing with my model ability to generalize, leading to an overfit.

I am a little skeptical about the number 2, seeing that my training loss also went up when I removed the product name.
My question:
So, how can I restructure my dataset? Does anybody have a clue why my model can't reach the same performance without using the product name? Any light or ideas on what I can try?


texts: Why does batchnorm1d in Pytorch compute 0 with the following example (2 lines of code)?

texts:
Here is the code
import torch
import torch.nn as nn
x = torch.Tensor([[1, 2, 3], [1, 2, 3]])
print(x)
batchnorm = nn.BatchNorm1d(3, eps=0, momentum=0)
print(batchnorm(x))

Here is what is printed
tensor([[1., 2., 3.],
        [1., 2., 3.]])
tensor([[0., 0., 0.],
        [0., 0., 0.]], grad_fn=<NativeBatchNormBackward>)

What I am expecting is the following:
Using hand calculation, let $x = (1,2,3)$, then $E(x) = (1+2+3)/3 = 2$ and $Var(x) = (1^2 + 2^2 + 3^2) /3 - (2)^2 = 0.9999...$, so that the final output looks like $y \approx (1,2,3) - 2/\sqrt{1} = (-1, 0, 1)$
So, I am expecting the output to the batchnorm be
tensor([[-1., 0., 1.],
        [-1., 0., 1.]])

Can someone please explain where I went wrong?


texts: Once a predictive model is in production, how it can be evaluated?

texts:
I have a data science project, predicting customer's next purchase day. Customer's one year behavioral data was split to 9 and 3 months for train and test, using RFM analysis, I trained a model with different classifiers and the best one's result is as follow:
Accuracy of XGB classifier on training set: 0.93
Accuracy of XGB classifier on test set: 0.68
This is my school's project, and I was wondering, in real world projects, how can we evaluate a model's performance after it's on production. How can I measure how successful my model was? What if the performance measures in production are much lower than my test result?


texts: Opinions on an LSTM hyper-parameter tuning process I am using

texts:
I am training an LSTM to predict a price chart. I am using Bayesian optimization to speed things slightly since I have a large number of hyperparameters and only my CPU as a resource.

Making 100 iterations from the hyperparameter space and 100 epochs for each when training is still taking too much time to find a decent set of hyperparameters.

My idea is this. If I only train for one epoch during the Bayesian optimization, is that still a good enough indicator of the best loss overall? This will speed up the hyperparameter optimization quite a bit and later I can afford to re-train the best 2 or 3 hyperparameter sets with 100 epochs. Is this a good approach?

The other option is to leave 100 epochs for each training but decrease the no. of iterations. i.e. decrease the number of training with different hyperparameters.

Any opinions and/or tips on the above two solutions?

( I am using keras for the training and hyperopt for the Bayesian optimisation)


texts: Columns with no (or nearly no) differences between rows worth keeping?

texts:
I have a dataset which contains multiple columns. On analysis, I found out that there were few columns which contain just a single value. There was 0 in all the rows of these columns. Does it hold any information or should I remove such columns from the dataset?

Few columns were also like that it contains 100k rows of type1 and 3-4 rows of type2. Should i also remove this columns or let them remain as such?

Note: Target column is different than the columns described.


texts: Implementing Complementary Naive Bayes in python?

texts:
Problem
I have tried using Naive bayes on a labeled data set of crime data but got really poor results (7% accuracy). Naive Bayes runs much faster than other alogorithms I've been using so I wanted to try finding out why the score was so low.
Research
After reading I found that Naive bayes should be used with balanced datasets because it has a bias for classes with higher frequency. Since my data is unbalanced I wanted to try using the Complementary Naive Bayes since it is specifically made for dealing with data skews. In the paper that describes the process, the application is for text classification but I don't see why the technique wouldn't work in other situations. You can find the paper I'm referring to here. In short the idea is to use weights based on the occurences where a class doesn't show up.
After doing some research I was able to find an implementation in Java but unfortunately I don't know any Java and I just don't understand the algorithm well enough to implement myself.
Question
where I can find an implementation in python? If that doesn't exist how should I go about implementing it myself?


texts: Data science Ph.D. program, what do you think?

texts:
As Yann LeCun mentioned, a number of PhD programs in data science will be popping up in the next few years.

NYU already have one, where Prof.LeCun is at right now.

A statistics or cs PhD in machine learning is probably more rigorous than a data science one.  Is data science PhD for the less mathy people like myself?

Are these cash cow programs?

There is a huge industry demand for big data, but what is the academic value of these programs, as you probably can't be a professor or publish any paper.


texts: Why is hierarchical clustering quadratic and k-means linear?

texts:
According to the internet, k-means clustering is linear in the number of data objects i.e. O(n), where n is the number of data objects. The time complexity of most of the hierarchical clustering algorithms is quadratic i.e. O(n2).

I am struggling to intuitively understand what is the difference between the two clustering approaches that causes this.

Question: What causes the difference in time complexity?


texts:
I was wondering how image classifier networks perform on images that are not photographs. For example if you were to feed a drawing of a car or a face to an image classifier that was only trained on photos would the network still be able to classify the image correctly?

Furthermore what if you were to feed more and more abstract drawings into the network. As humans we are able to recognize objects even in abstract forms (i.e. modern art) but do current image classifiers generalize well enough in order to do this?

Are there any networks that are also trained on artist renditions of objects and not just photographs?


texts: Data Exploration Led Conversion to Ordinal Variable

texts:
[I encountered this question at an interview few weeks ago and I am still not clear.]

If all the values in a categorical column fuel_mileage come from the set {poor, good, very_good}, then we can make the column ordinal due to universal and ordered relationship amongst {poor, good, very_good}, so this is kind of obvious.

However, imagine the label column in this same dataset is engine_longevity, so that we are studying all other variables in the context of their relationship with it. During data exploration, it turns out that another categorical column, manufacturer, all of whose values come from set {H, S, J, K}, has a very strong correlation with label engine_longevity, so much so that the choice of H, S, J, K in a given sample essentially dictates the label. Therefore, as for as this data set is concerned, H, S, J, K have an ordered relationship with respect to label engine_longevity. The question is:

Will you make column manufacturer ordinal? If yes, how strong should the relationship between manufacturer and the label engine_longevity be? And what metric will you use to measure it?
If you will not make manufacturer column ordinal, why?
More generally, should the choice of making a column ordinal come from the mutual-relationship of values within that column alone? Or, the relationship of values in a column with the label should be taken into consideration?

If there is no hard-and-fast rule, I would like to know how the community here will approach this situation.


texts: Recommended Tutorial Videos or Books on Feature Engineering Using Python

texts:
I will appreciate it if you guys can recommend for me a good hands-on tutorial videos or books on feature engineering using Python. I do not want videos or books that teach only the theory behind feature engineering but one that will show me some codes and how to do it.
Thanks.


texts: Ensemble modelling using model's probabilities

texts:
In a classification project, on the training sets, I ran a selection of classifiers. These give me about 20-30% accuracy at best. For each sample, I generate probabilities of each class. I want to make a new model which takes these probabilities and gives a weighted average of the probabilities which is accurate.
For this, I tried averaging probabilities. I also tried taking the best probabilities from each model (For instance, assuming Class A has better precision/recall for model1 output, I take model1's outputs for class A and so on). This also hasn't improved the accuracy much. Can you suggest some ensembling techniques?


texts: How do I crop faces with a neural network?

texts: I am looking to build a facial recognition system and realized I could probably pump up accuracy by first cropping the faces. I know I can use a Haar Cascade classifier to do this but would prefer to use a deep learning approach, as that is my current area of focus. To do this do I simply create a CNN with inputs being an image and outputs being the X, Y, Width, Height? Or is there more to it? I have tried this approach with an Adam optimizer and around 600 pictures but results haven't been great. Any ideas on how to improve this?


texts:
I'm working through excersises on Belief Propagation and the Junction Tree Algorithm and I'm stuck with the following problem. Consider the distribution P(A,B,C,D,E,F,G,H)=P(A)P(B)P(C)P(F) P(D|A,B)P(E|B,C)P(G|D,F)P(H|E,F), i.e.
The questions asked are:

Is it possible to use belief propagation on this graph (without forming a Junction Tree) to compute P(B|G,H)?
What about P(B|F,G)
Now assume only F has been observed. How would you compute P(B|F)?

Now, AFAIK this is a directed polytree (Nodes may have multiple parents, but there is at most a single path between any two nodes). Thus Belief Propagation should allow exact inference in this tree, which is why I would answer "Yes" to both 1) and 2). Is this correct? It seems like I'm missing something here. Why would conditioning change something about whether Belief Propagation can be used here?
Regarding 3, we note that B is cond. indep. of F given no other nodes, so P(B) = P(B|F), is this correct?


texts:
i tried to evaluate 6 models and after plotting , this what i get :

So i'm wondering , if those results are "Right" ?

Thank's in advance.


texts:
Hello I am building a python program for text spinning. I have read about GloVe, word2vec, doc2vec or text2vec. I understand that what they do is to represent each word as a semantic vector.

So I guess that if I train a GloVe I could use it just for finding synonyms, or something more?

EDIT: I have come up with one strategy:

First I train a doc2vec model. The goal would be to build semantic vectors of phrases
Then I use my original paragraph input. Word by word I change it randomly with a synonym (provided by nltk or other synonym db). Before finally spinning one word I compute the doc2vec for the changed paragraph and compute the distance with the original paragraph vectorized. I only change the synonym if the distance is small.

How does that sound to you?


texts: Spliting training data into multiple variables using R

texts:
So right now I am trying to create multiple variables with training data, and in the process I have reached an error

  Error in eval(predvars, data, env) : object '1.band1' not found

which is a product of these lines:

for(i in 1:length(data_split)){
 assign(paste("fit.lda",i, sep = ""), train(class~., data=data_split[i],
 method="lda", metric=metric, trControl=control))}

is it something that I did wrong or is it something that can be fixed with another methodology

EDIT:
My dataset is a data frame which was created by the following lines

validation_index <- createDataPartition(valuetable$class, p=0.80, list=FALSE)
dataset <- valuetable[validation_index,]

where valuetable is a dataframe which containes the values of multiple rasters.

Now, I wanted to split my data so my train function can be processed faster, that is because my machine's specs are pretty low, thus allowing me to re-create the later the results that would take time using other methods.

Let me know if you want more clarification.


texts: Dimension problem in keras neural networks

texts:
So every time I try to write a CNN or RNN I have problems understanding how the dimensions work.

I don't understand why does the model compile, which in my opinion means that the output dimension of the previous layer is the input dimension of the next one and so on(the layers are compatible with each other). So how is it possible that when I fit my model I get an error in the 3rd or 4th layer(or any other layer except the first one) saying that for ex it expects 3 dimensions but got 2.
On a more precise example, lets say I have the following one-shot model:

vocab_size =10000
src_txt_length =10
sum_txt_length = 100

inputs = Input(shape=(src_txt_length,))
encoder1 = Embedding(vocab_size, 128)(inputs)
encoder2 = LSTM(128)(encoder1)
encoder3 = RepeatVector(sum_txt_length)(encoder2)

decoder1 = LSTM(128, return_sequences=True)(encoder3)
outputs = TimeDistributed(Dense(vocab_size, activation='softmax'))(decoder1)
model = Model(inputs=inputs, outputs=outputs)
model.compile(loss='categorical_crossentropy', optimizer='adam')
model.fit(X,y)

I get an error in the TimeDistributed layer saying that it expects 3 dimensions but got 2.

X.shape == (20000,10)
len(y) == 20000  # y is a list of labels 0s and 1s

So if I want to calculate the dimension of each layer when I fit my data on paper, how will I do that ?


texts: how to convert multiple columns into single columns in pandas?

texts:
I have a dataframe like this

my desire format is like this

how can i do this?


texts: Embedding a categorical variable and concatenating with a numerical variable, in a many-to-one sequence problem with multiple features

texts:
I have a small data set where I track 4 variables across 4 time periods, 1 categorical and 1 numerical variable. Below is picture of data set that I am using:

cat1 - Categorical variable encoded
num1 - Numerical variable

Below is picture of my data set (to show the format; my X and Y is the same cause I am trying to learn how GRUs work and am not concerned much about the model accuracy).

Here is my GRU model code. It runs well and it predicts ok.
inputs = Input(shape=(4,2))
x = Bidirectional(GRU(256, activation='relu'))(inputs)
x = Dense(32, activation='relu')(x)
outputs = Dense(4)(x)
model = Model(inputs=inputs, outputs=outputs, name="func_loss_res")
model.summary()
plot_model(model, "my_first_model_with_shape_info.png", show_shapes=True)

model.compile(optimizer='adam', loss='mse')
my_callbacks = [tf.keras.callbacks.EarlyStopping(patience=10)]
print(X_train)
history = model.fit(X_train, Y_train, epochs=250, validation_split=0.25, batch_size=4, callbacks=my_callbacks)

My question is, if I were to Embed the cat1 variable say with 5 unique values and embedding size of 3 (I want to scale this model with 146 categories later and add 22 numerical variables), how can I concatenate cat1 and num1 to make it work for my given GRU? Any assistance is highly appreciated!


texts: Classifying hand written digits dataset using Probabilistic neural network (PNN)?

texts:
I am trying to apply Probabilistic Neural Network (PNN) on MNIST dataset using MATLAB on 28000 samples. I tried to build the model but it gave an error saying matrix dimension exceeds memory showing it will require 5.08 GB (28000x28000) matrix.
I want to know should I decrease the number of samples as input for training from 28000 to 200 samples as I read in research paper.
Should I reduce the sample size or is there something else wrong?
is pnn only applicable on small data sets?


texts: Error while using decompose and stl functions in R

texts:
I am new to time series analysis and I am trying out some codes seen in some tutorials. I am using stock prices as the time series data.

library(quantmod)
aapl=getSymbols("AAPL",from="2015-01-01",auto.assign=F)
is.ts(aapl)
ts.aapl=as.ts(aapl)
is.ts(ts.aapl)
x=Cl(ts.aapl)
head(x)
plot(decompose(s))
plot(stl(x))

But its showing these errors

> plot(decompose(x))
Error in decompose(x) : time series has no or less than 2 periods
> plot(stl(x))
Error in stl(x) : series is not periodic or has less than two periods

is the error is due to its lack of seasonality and trend?


texts: High Level Discussion: Generate synthetic sensor data using data from surrounding sensors

texts:
Let's assume I have readings coming from sensors. For every sensor, I have the following information:

all the data it reads
its location

Now, given an arbitrary sensor at an arbitrary location between these sensors, we want to predict the stream of data coming from this sensor. We would like to generate a synthetic stream of data, using the data coming from the other sensors and knowing their locations relative to our target sensor.

How would I go about solving such a problem? What is the "general idea" and "data-driven techniques" to handle such a problem?


texts: Clustering based on features of varied importance

texts:
Suppose I have a dataset that includes the following features {HairColor, EyeColor, EducationLevel, Income}. I would like to perform clustering to separate the dataset into smaller datasets that you would expect to behave similarly.  The difficulty that arises is that it is clear that EducationLevel and Income are much more important than HairColor and EyeColor but I do not know how to measure that importance for the sake of clustering.
In the example below, I would want it to be clear that Row 1 is more similar to Row 3, than to Row 2.

ID
EyeColor
HairColor
EducationLevel
Income

1
1
1
1
1

2
1
1
2
2

3
2
2
1
1