texts: Training Neural Networks with unknown length of input

texts:
I'm currently going into the world of machine learning and Neural Networks, thanks to synaptic (js) that interests me a lot.

So I read a lot, wikipedia links and synaptic's NN 101, but there's a lot of basics questions that I don't understand (but I'd like to) in the use of machine learning (NN) and the point of these technologies.

Let's say, I wan't my network to (kind of) learn (something like) gravity, so to train it I set in input 10 objects with a mass, and a position x, y (and z) and I set output the new x, y (and z) of each objects.
I guess I should give it several configurations and everything but here is the question; can it, then, be able to compute the interactions between 10000, 100000 objects?

At this stage in my learning, what I don't clearly get is what is the point of teaching/training neurons to compute XOR like it's shown in synaptic's documentation:

var trainingSet = [
    {
        input: [0,0],
        output: [0]
    },
    {
        input: [0,1],
        output: [1]
    },
    {
        input: [1,0],
        output: [1]
    },
    {
        input: [1,1],
        output: [0]
    },
];

var trainer = new Trainer(myNetwork);
trainer.train(trainingSet);

Were we just give it all the possible inputs and outputs to a XOR.

Well, as I'm all new to the technologies I think my questions are full of non-sense and everything but thanks for reading and help you might bring :)


texts: how can I visualize the bag-of-words in a scatterplot

texts:
I have a bag-of-words consisting of about 60000 features. Each feature represents a diminsion. I want to represent this bag-of-words in a reduced 2D space. How do I do it?

I have see an example here, which looks more like what I want but not really the same. In the example they have 2 transform and I only have one. Therefore as suggested I do not want to use pipeline. Below is my code which taking for ever and does not show any error message:

#myList contents about 800000 words
bag_of_words = vec.fit_transform(myList)
X = bag_of_words.todense()
pca = PCA(n_components=2).fit(X)
data2D = pca.transform(X)
plt.scatter(data2D[:,0], data2D[:,1])
plt.show()

I have not found any better option and right now it looks like I am doing something wrong.

What is the best way to visualize a bag-of-words in a scatterplot?

The bag_of_words looks like this:

(0, 548)    3
(0, 4000)   6
(0, 15346)  1
(0, 23299)  1
(0, 22931)  2
(0, 32817)  1
(0, 51733)  1
(0, 38308)  6
(0, 14784)  1
(0, 146873) 1
 ....


texts: Adding variables to Random Forest decreases performance

texts:
Let's say you have 6 variables.

A Random-Forest regression using the first 5 variables has an R^2 of 0.1. Another regression using just the 6th variable yields R^2 of 0.3. All of the first 5 variables are uncorrelated with variable 6 (<0.1 correlation in absolute value).

Why would a regression with all 6 variables have an R^2 of 0.31 or 0.29, i.e. adding first 5 variables to the 6th variable only negligibly increases performance by 0.01 or even decreases it?

Notice that all models are tuned via random search cross-validation tuning depth, number of features, splits and number of trees.


texts: sklearn RandomForestRegressor oob_score_ looks wrong?

texts:
I am a newbie to datascience. I installed the Jupyter notebook and was trying to create model for the kaggle titanic dataset. Below is the code I wrote -

from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import roc_auc_score
import pandas as pd
X = pd.read_csv("train.csv")
y = X["Survived"]
X["Age"].fillna(X.Age.mean(),inplace = True)
numeric_variables = list(X.dtypes[X.dtypes != "object"].index)
model = RandomForestRegressor(n_estimators=100, oob_score = True, random_state = 42)
model.fit(X[numeric_variables],y)
model.oob_score_

I am getting the oob_score_ output as 1.0 while the actual output as per tutorial should be 0.13420....
After that I did

y_oob = model.oob_prediction_
print "c,stat:", roc_auc_score(y,y_oob)

The output of this as well I am getting 1.0. Is there something wrong in my code or setup of Jupyter?


texts:
Today I've seen many Perceptron implementations with learning rates. According to Wikipedia:

  there is no need for a learning rate in the perceptron algorithm.
  This is because multiplying the update by any constant simply rescales
  the weights but never changes the sign of the prediction

Is there some benefit to implementing a learning rate with Perceptron? If there is not, why do so many implementations have it?


texts: How to change the names of the layers of deep learning in Keras?

texts:
I am using vgg16 to create a deep learning model. I want to know how to change the names of the layers of deep learning in Keras?

I tried this

for layer in vgg_model.layers:
    layer.name = layer.name + str("_")

But when I change the names of the layers, the model accuracy become low.


texts: Alternative method for RNN backpropagation through time

texts:
I am experimenting with a character-level LSTM model doing the standard task of predicting the next character given a sequence of characters. I am training the network using truncated backpropagation through time (BPTT).

I am trying to compare two techniques for setting up the training examples for this task. For my diagrams below, let's say the input sequence is the alphabet: "abcde..." and the horizon size for BPTT is 3. From what I understand, the standard way of training the network would be:

Training example 1 ('d' is the target character):

Training example 2:

Training example 3:

Technique #2:

Here is the alternative setup that seems reasonable to me:

Training example 1:

This training example has three separate target characters. The error from 'd' backpropagates through the full three time steps. However, error from 'c' backpropagates through two time steps and 'b' through one.

Training example 2:

Training example 3:

For training a single epoch over N characters, the second technique has a significant time complexity improvement. Increasing the horizon size has no impact on the running time. A larger horizon size will decrease the number of training examples, but each training example will have more target characters (i.e. more meaningful weight updates).

Is this second technique used at all for training LSTMs? Any relevant links? Why is the first technique more commonly used? Are there alternative training example setups which are also effective?


texts: Minimising inputs for decision tree predictions

texts:
It is common for decision trees with asymmetrical shapes to have leave nodes that come early. For example, the model can already generate a prediction if the answer to the first question is FALSE, without the need to ask the second question:

From a user experience perspective, it is advantageous to shorten the journey to get a prediction. One solution is to perform sklearn.tree.export_graphiz on the trained model, and manually script the delivery of input fields in the order the tree needs them. If a leave node is reached, the predict function is called with the inputs collected, while assigning the rest of the inputs to default values. However, this is not scaleable for large decision trees. Is there a way to order the input fields autonomously?


texts: Is it possible to use the saved xgboost model (with one-hot encoding features) on unseen data (without one-hot encoding) for prediction?

texts:
I think the question is self-explanatory. But let's say you have a data with a few features with categorical data, and when building a model for example XGBoost you one-hot encode categorical features. Now you want to do prediction based on test data using the saved model. Obviously the test data needs to be one-hot encoded and need have similar features as training set. The question is whether it is possible to find a way not one-hot encode the test data and directly use it for prediction? Would this be somehow possible?

To me it appears that whatever comes in to my saved model need to be as it was used during training i.e. one-hot encoded features! But this is not neat, especially when building widgets and dashboards!

Any comments/hints are appreciated.


texts: Is it possible to do attribute, value extraction prediction model in Machine Learning?

texts:
Previously I asked a question at here, but it doesn't seems to be at the correct place.
So, I moved it here with more details of what I've done.
Here is the sample data image to be process.
Start with what am I going to do..
What I tried to do here is to automate value extraction.
By referring from Left part which I address as "Attribute", then to extract Right part which I address as "Value". E.g. Name = Gan Chye Chung; Date of Birth = 10/01/1985; and so on
What I've tried :

Using Python hard-coded way to extract the Value after managed to identify the Attribute.So, at here I'm using Tesseract to read each string found, to match with the Attribute list that I want to extract. (E.g. Name) then to assume the right part after Name will be the value until the end of the line. This way works fine, until I realized the tedious part that need to tune the code every time there is a changes in the dataset (tilted image, new attributes, and so on).
Using sklearn Linear Regression to predict coordinates. Afterwards, my teammate give me some idea that probably Linear Regression model will work in such case. So I try it by feeding coordinates of Attributes as (X) to predict coordinates of Values as (y). Well, I managed to get TopLeft coordinate of Value pretty accurate, since I'm feeding the same layout of samples with only different value in it for my first batch. Then here I faced another problem, which it seems impossible to predict the width of Value. Since the width of Value is inconsistent, current name probably will be just 14 charactes (including space). But person names can be more or less than that 14 characters which hard to set a standard at it.
This also sklearn Linear Regression to predict the width or length. But I add one more field called data_type to differentiate the types which later used for Value's width prediction. Over here, I set the types like thisE.g. Name is a T_varchar_100, dob is T_datetime, age is T_smallint. Which then, I replace those into number to be able to fit into Linear Regression. Well, it doesn't seems to be a correct way, but I think it will work at the first place. I test to run it, turns out it failed as well to predict the width of the Value.

So, I wonder is there any other better way (or model) to do such prediction on Attribute - Value extraction?
It doesn't have to be a perfect answer.
Any help will be appreciated, link or reading or reference that I can refer to will be great as well. Just to brainstorm any idea that might works.
P.S. I'm not from statistic background, just try to take a tiny step into it. Cheers.


texts: In a assignment we are given macro economic indicators like GDP, Consumer price index, Producer Price index and Industrial production index. Also we are given Crude oil, Sugar prices and FM-CG Sales. We are required to forecast future quarter sales and give a model. As I'm new to this subject, I don't know where to start with it, or what to read. Can anyone provide me with some examples of what to do, or any PDFs which might be helpful.


texts: Can a decision in a node of a decision tree be based on comparison between 2 columns of the dataset?

texts:
Assume the features in the dataframe are columns - A,B,C and my target is Y

Can my decision tree have a decision node which looks for say, if A>B then true else false?


texts: What are your favorite sources for news about Machine Learning and Data Science?

texts:
Data Science and Machine Learning include a lot of different topics and it´s hard to stay up-to-date about all the news about papers, researches or new tutorials and tools.
What sources do you use to get all the information?
I use mostly Reddit as my first source and the subreddits Machine Learning and R.
But I would also Datatau and of course the great KDNuggets.


texts: Which hyperparameters of a neural network can be tunned independently?

texts:
The hyperparameter search is computationally expensive. I am wondering if one can tune the hyperparameters independently: tune one hyperparameter for a fixed value of other hyperparameters. For example, let's say we have two hyperparameters, A and B. We search for best value of A at fixed random B, then we search for best value of B at fixed best value of A.

This make sense if the the other hyperparameters does not interfere with the ordering of the validation loss for the hyperparameter we want to tune. In that sense, the number of units and the number of layers cannot be tuned independently. According to the Y. Bengio's paper (link), at some point the mini-batch size can be tuned independently (page 9, right column, The Mini-Batch Size).

But what about the other ones? Learning rate, activation function, dropout, ... which one can be tuned independently?


texts: Find-out abnormal behavior over the time

texts:
I want to detect abnormal behaviour in a oil pipe where the oil is flowing with some constant pressure. I have a sensor which monitors and pressure over the time and push it to my cloud server.

I got the dataset . My requirement is to do data analytics in python and find-out the abnormal pattern in the dataset over the time. And I need to suggest abnormal behaviors present in the dataset ., say for example., from 1 pm to 3:30 pm today , the pressure raises and falls may be due to some leakage in the pipe.

can we do it by simple statistical model? or machine learning is required?

Can please suggest the best suitable machine learning algorithm for this scenario?

Also please mention the web links over here.

Thanks


texts: Assigning new items to existing similarity based clustering

texts:
Given some clusters created from similarity measures between items, is there a recommended way to assign a new item to an existing cluster based on similarity alone? (i.e. avoiding re-clustering)

Measuring the similarity of a new item to all other items is fairly cheap, so I'm looking for a way of using this to assign it to the cluster it's most likely to belong to. It's also important for it to take cluster size into account (i.e. doesn't unfairly weight towards or against larger clusters).

Basically, I'm trying to sacrifice some clustering accuracy in exchange for avoiding a complete re-clustering when the occasional new item is added.


texts: How to feed in the bounding box annotations to MobileNet (preferably v3) for training using keras?

texts:
I am trying to train an object detection model using transfer learning with MobileNet as the base model in AWS sagemaker. (The answer doesn't have to related to sagemaker. I can manage to add it if I can learn a method to do it in keras/tensorflow)
In keras ImageDataGenerator, we can easily create the folder structure and feed the images to the training job using that generator for a "classification" tasks. The flow_from_directory function works fine for that. But in my case I need to use bounding box annotations. Is there a similar class or method to feed the image data along with annotations for my "object detection" task from directory ?
My dataset subfolders have the following structure:

    .
    │
    ├── train
    │   ├── 000001.jpg
    │   :   ...
    │   └── 07000.jpg
    │
    ├── train_annotation
    │   ├── 00001.json
    │   :   ...
    │   └── 07000.sjon
    │
    ├── validation
    │   ├── 01000.jpg
    │   :   ...
    │   └── 05000.jpg
    │
    └── validation_annotation
        ├── 01000.json
        :   ...
        └── 05000.json

I saw a method where annotations are put to a .csv  (basically a dataframe) and something with TFrecords. I am not sure how it's done though.
Is there a convenient method like a generator or something in tensorflow or keras ?
Can someone please explain me how to do this or direct me in the right direction?


texts: how to work with a large dataset in python?

texts:
for a training purposes, in order to start machine learning and data analysis with python I work on a pretty small dataset (20000 rows, 600MB) .But since couple days I decided to go further so  I have download a large data set of 1Go.

I wanted to do some analysis and apply machine learning on it so I tried to read the csv file with pyhton in jupiter nootebook and th file is still loading after more than 3 hours.

So I will would like to know  what are the best practices/process to follow when you have to work with large dataset ?


texts: Neural Network Multiple | Averging predictions

texts:
I am training multiple neural networks with various parameters. I am trying to average their predictions, but I am not really sure what that means, I am confused about what to average exactly. Here is what I mean: For a single observation in binary classification for example, the final node will give p a value between 0 and 1 (or -1 and 1 if you're using hyperbolic tangent Activation Function), then this p will be rounded to 1 or 0 if it's > 0.5, depending on your decision boundary.

Now, here is what I don't understand, should average p1, p2 and p3 produced by the models before rounding, or I should round the values to True/False responses and then compute the average? and how does that work exactly ?


texts: Can machine learning learn a function like finding maximum from a list?

texts:
I have an input which is a list and the output is the maximum of the elements of the input-list.

Can machine learning learn such a function which always selects the maximum of the input-elements present in the input?

This might seem as a pretty basic question but it might give me an understanding of what machine learning can do in general. Thanks!


texts:
I'm looking for the best solution to manage and host datasets for journalistic pursuits. I am assessing https://www.documentcloud.org and http://datahub.io/.

Can anyone explain the differences between them, or recommend a superior solution?


texts: Is there any ability to use two ore more inputs for Elman recurrent neural network?


texts: keras validation mean squared error always similar to 1

texts:
is there any reason why the validation mean squared error output from Keras is always very similar to 1? Thank you. All of my training results looks like:

155/155 [==============================] - 0s - loss: 6062.6136 - mean_absolute_error: 0.8344 - mean_squared_error: 1.0271 - val_loss: 0.8252 - val_mean_absolute_error: 0.8252 - val_mean_squared_error: 1.0164
Epoch 29/1000
155/155 [==============================] - 0s - loss: 5870.5280 - mean_absolute_error: 0.8324 - mean_squared_error: 1.0211 - val_loss: 0.8246 - val_mean_absolute_error: 0.8246 - val_mean_squared_error: 1.0130
Epoch 30/1000
155/155 [==============================] - 0s - loss: 5668.5083 - mean_absolute_error: 0.8311 - mean_squared_error: 1.0134 - val_loss: 0.8244 - val_mean_absolute_error: 0.8244 - val_mean_squared_error: 1.0106
Epoch 31/1000
155/155 [==============================] - 0s - loss: 5530.8119 - mean_absolute_error: 0.8288 - mean_squared_error: 1.0115 - val_loss: 0.8243 - val_mean_absolute_error: 0.8243 - val_mean_squared_error: 1.0089
Epoch 32/1000
155/155 [==============================] - 0s - loss: 5222.6773 - mean_absolute_error: 0.8283 - mean_squared_error: 1.0119 - val_loss: 0.8245 - val_mean_absolute_error: 0.8245 - val_mean_squared_error: 1.0071
Epoch 33/1000
155/155 [==============================] - 0s - loss: 5090.0273 - mean_absolute_error: 0.8273 - mean_squared_error: 1.0078 - val_loss: 0.8247 - val_mean_absolute_error: 0.8247 - val_mean_squared_error: 1.0060
Epoch 34/1000
155/155 [==============================] - 0s - loss: 4878.2420 - mean_absolute_error: 0.8272 - mean_squared_error: 1.0093 - val_loss: 0.8245 - val_mean_absolute_error: 0.8245 - val_mean_squared_error: 1.0046

note: I have standardized my input and output with sklearn standardization:

from sklearn import preprocessing
    X_scaler = preprocessing.StandardScaler().fit(X_list_total)
    X_list_total_standardized = X_scaler.transform(X_list_total)
    Y_scaler = preprocessing.StandardScaler().fit(Y_list_total)
    Y_list_total_standardized = Y_scaler.transform(Y_list_total)

Does it just mean that there is nothing to learn from the data at all?


texts:
During prediction (not training), is it normal to get different loss for different batch size?
Worst case happens when I use batch_size=1 for test dataset. The prediction performance get pretty bad. Performance get better, if I increase the batch size.
During training, both training and validation batch size was 32.
Model consist of Convolutional Neural Network and input is word embeddings.
My thought:
In my opinion, only thing change with different batch is the padding. For batch size=1, there will be no padding. Since, I used zero for padding, the weights (w) of the model will have no effect because of zero but the bias (b) will be added.
Do you think, this can change (moderately) the prediction and therefore the calculated loss?


texts: How to subset rows from a data frame with comparison operators in R

texts:
I have a data frame (a csv file) with dimensions 100x6 and I need only the columns c("X1", "X2", "X4") and the rows in which the value of "X1" is greater than 30. So I did:

  data_frame <- read.csv ("data_frame")
  data_frame [c("X1", "X2", "X4")]

The column subset problem is solved but now I need to subset rows from data_frame [c("X1", "X2", "X4")] where the values of "X1" is greater than 30. I tried:

  data_frame [c("X1" > 30), c("X1", "X2", "X4")]

But it returned the same data frame as data_frame [c("X1", "X2", "X4")].
Also tried using the function subset() with the same approach but got the same results.


texts: Predicting the next occurrence based on binary

texts:
I have no experience in statistics or machine learning. I have a True/False binary array describing occupation of open public spaces

+---------------------+
|  index   |  Value   |
+---------------------+
| 0        |  True    |
| 1        |  True    |
| 2        |  False   |
| 3        |  False   |
| 4        |  False   |
| 5        |  True    |
| 6        |  False   |
| 7        |  False   |
| 8        |  True    |
| ...      |  ...     |
+---------------------+

Without getting into dependent variables and domain specific heuristics, is there (or maybe more than one) a simple method to do infer the next False in python?

Ideally in pure python or using packages written in pure python.

My question is somewhat similar to this one, but I have more of a time series (i think).


texts: ValueError: Expected 2D array, got scalar array instead using predict method

texts:
I am trying to get a predicted value instead of whole features for a particular level using predict method.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

#Importing Dataset
dataset = pd.read_csv('C:/Users/Rupali Singh/Desktop/ML A-Z/Machine Learning A-Z Template Folder/Part 2 - Regression/Section 7 - Support Vector Regression (SVR)/Position_Salaries.csv')
print(dataset)
X = dataset.iloc[:, 1:2].values
Y = dataset.iloc[:, 2].values

# Feature Scaling

from sklearn.preprocessing import StandardScaler

sc_X = StandardScaler()
sc_Y = StandardScaler()
X = sc_X.fit_transform(X)
Y = sc_Y.fit_transform(Y.reshape(-1,1))
#Fitting SVR model to dataset

from sklearn.svm import SVR

regressor = SVR(kernel='rbf')
regressor.fit(X,Y)

#Visualizing the dataset

plt.scatter(X, Y, color = 'red')
plt.plot(X, regressor.predict(X), color = 'blue')
plt.show()

# Predicting a new Result

Y_pred = regressor.predict(6.5)
print(Y_pred)

This is my dataset, here I am trying to predict value only for level 6

Position  Level   Salary
0   Business Analyst      1    45000
1  Junior Consultant      2    50000
2  Senior Consultant      3    60000
3            Manager      4    80000
4    Country Manager      5   110000
5     Region Manager      6   150000
6            Partner      7   200000
7     Senior Partner      8   300000
8            C-level      9   500000
9                CEO     10  1000000

This is the error message I am getting:

File "C:/Users/Rupali Singh/PycharmProjects/Machine_Learning/SVR.py", line 34, in <module>
    Y_pred = regressor.predict(6.5)
  File "C:\Users\Rupali Singh\PycharmProjects\Machine_Learning\venv\lib\site-packages\sklearn\svm\base.py", line 322, in predict
    X = self._validate_for_predict(X)
  File "C:\Users\Rupali Singh\PycharmProjects\Machine_Learning\venv\lib\site-packages\sklearn\svm\base.py", line 454, in _validate_for_predict
    accept_large_sparse=False)
  File "C:\Users\Rupali Singh\PycharmProjects\Machine_Learning\venv\lib\site-packages\sklearn\utils\validation.py", line 514, in check_array
    "if it contains a single sample.".format(array))
ValueError: Expected 2D array, got scalar array instead:
array=6.5.
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.

I would be really grateful for any kind of help.


texts: Which train test split performs better: 50:50 or 60:40?

texts: I have 10,000 customer data of a supermarket. And I want to split the data into training set and testing set. So, which train test split gives me a better accuracy: 50:50 or 60:40?


texts: Comparing 2 data frames, if value is present replace with 1 or else 0

texts:
nrow(df1$v1) = 63849
    nrow(df2$v2) = 3244
ifelse(df1$v2 == df$v1, 1, 0)

I know this is an easy question but I tried different procedures but none of them are useful,

for(i in 1:nrow(df2)){
for(j in 1:nrow(df1)){
    if(df2$v1[i] == df1$v2[j]){
      df1$v2<- 1
    }
    df1$v2 <- 0
  }
}

This does the job but it take quiet sometime to get the job done.

Other Methods:
Method 1:

df1$v2 <- ifelse(df2$v1 %in% df1$v1, 1,0 )
Error in `$<-.data.frame`(`*tmp*`, v2 , value = c(1, 1, 1, 1, 1, 1,  :

replacement has 3244 rows, data has 63849

Method 2:

df1$v2 <- ifelse(do.call(paste0, df2$v1) %in%
do.call(paste0, df1$v1), 1,0 )
summary(df1$v2)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
      0       0       0       0       0       0

Do suggest me if you have any better solution.


texts: Why would one use entropy instead of Gini index in CART?

texts:
I read this question Gini Impurity vs Entropy and was wondering why would someone use entropy instead of Gini index in a decision tree with scikit-learn.

Indeed, I find these arguments legit:

  Given a choice, I would use the Gini impurity, as it doesn't require me to compute logarithmic functions, which are computationally intensive. The closed form of it's solution can also be found.

And by looking at the following I can't see much a difference between the two graphs:


texts: How to reload cell in jupyter notebook? After OSError: [Errno 12] Cannot allocate memory

texts:
I got an error:
OSError: [Errno 12] Cannot allocate memory
I deleted some files. And I have free memory. I don't want start learning again from first epoch. (I got an error in last epoch).

I use jupyter notebook.
I have 19 .h5 file in my output file.
Can I click something to save my last .h5 file?

Edit:


texts: How to handle Tokenized text content which is given in number?

texts:
i have one data set of customer review, but the text data is given is tokenized text number. I am unable to proceed thinking about how to proceed?
As I am encountering such data set the first time, so just need guide how to proceed.

As you can see text field is given in number, so how to proceed please guide?. it will predict the 0/1 +ve or -ve category.


texts: How to investigate what has been learnt in CNNs?

texts:
My CNN was trained using close-up images of dogs. For testing, we input an image with a dog next to its owner and it was classified as a dog. How is that possible?
Isn't this image a different vector?


texts: I am curious about the network architecture of AlphaZero. Can anybody tell me details about it? Is it the same as in AlphaGo Zero?


texts:
I am using Data Studio for a project and I am connecting to my BigQuery table.
My table contains following columns:

Date
Store_name
Footfall

I'd love to compare the footfall of two stores using Data Studio but apparently I can't do that!
Any hints or should I just switch to another viz tool?


texts: Implementing the Trapezoid rule without the formula for the curve

texts:
I know that if I have some function f(x) that describes a curve, I can approximate the area under the curve using the trapezoid rule as follows:
def auc(f, a, b, n):

    subinterval = (b - a) / n

    s = f(a) + f(b)
    i = 1
    while i < n:
        s += 2 * f(a + i * subinterval)
        i += 1

    return (subinterval / 2) * s

However, I am trying to implement the trapezoid rule to approximate the area under the ROC curve. I don't have the function f(x), but rather true positive rates and false positive rates at thresholds from 0 to 1 spaced by .01. I tried to implement the rule following this guide https://byjus.com/maths/trapezoidal-rule/ as so:
def roc_auc(tprs, fprs):

    y_sum = max(tprs) + min(tprs)
    for i in range(1, len(fprs)-1):
        y_sum += 2*tprs[i]

    interval = (max(fprs) - min(fprs)) / len(fprs)
    return ((interval / 2) * y_sum) / 100

However, when I test it against auc functions implemented in numpy, scikit, etc. I get different values than the one I calculate so I know I'm doing something wrong. Can anyone tell me where I'm going wrong?


texts: Can I use survival analysis model to predict probability of an item sold

texts:
I am building a model to calculate probability of items being sold( at least within a reasonable amount of time).
I know when the item hit the market and when/if the item sold.
What is the best approach here?
Nearly 70% of items in the dataset were sold and the best classification model I have come up with so far only has 72% accuracy.
Can I turn this into a survival analysis problem and output a probability of items being sold?


texts: Overfitting due to features correlating with training set generation rules

texts:
As background, I am using a Deep Neural Network built using Keras to classify inputs into 5 categories.

The current structure of the network is:

Input layer (~450 nodes)
Dense layer (750 nodes)
Dropout layer (750 nodes, dropout rate = 0.5)
Dense layer (5 nodes)

The issue I'm having if one of overfitting. My model performs well on the held-out test set (a proportion of my training set), with accuracy sitting right around 99%. However, when I look to apply the model onto unlabelled data, it is only able to classify ~67% of observations into any category, before even considering the correctness of those classifications!

I think the issue may be around my feature and training set generation process. I generated the training set using a rules-based string matching method. This generated a training set of around 3.6 million observations (10% of population).

However, one of the largest features for my input layer is an embedding of the same text used to generate the training set. Therefore, the words matched to generate the training set are also embedded and used as features. Worth noting the text is around 140 characters per observation, and I matched bigrams from that text (so there is other information in that text that would be useful as a feature).

I would remove this feature altogether, however this is the richest information associated with each observation.

Is there a way to solve this without removing that feature altogether?

Hope this makes sense and happy to provide more clarification.

Simplified explanation:

My model performs well on my training and test sets.
Performs badly on unlabelled data.
Each observation is associated with a block of text.
To label my training set I used string matching on that text.
The text is also a feature (embedding).
Is this causing my poor performance on unlabelled data (model
learning those string matches?).
If so what can I do?

EDIT: Also happy to hear if you think the issue is something else.


texts:
I am currently studying this paper (page 53), in which the suggest convolution to be done in a special manner.

This is the formula:

\begin{equation} \tag{1}\label{1}
q_{j,m} = \sigma \left(\sum_i \sum_{n=1}^{F} o_{i,n+m-1} \cdot  w_{i,j,n} + w_{0,j} \right)
\end{equation}

Here is their explanation:

As shown in Fig. 4.2, all input feature maps (assume I in total), $O_i (i = 1, · · · , I)$ are mapped into a number of feature maps (assume $J$ in total), $Q_j (j = 1, · · · , J)$ in the convolution layers based on a number of local filters ($I × J$ in total), $w_{ij}$ $(i = 1, · · · , I; j = 1, · · · , J)$. The mapping can be represented as the well-known convolution operation in signal processing.
Assuming input feature maps are all one dimensional, each unit of one feature map in the convolution layer can be computed as equation $\eqref{1}$ (equation above).

where $o_{i,m}$ is the $m$-th unit of the $i$-th input feature map $O_i$, $q_{j,m}$ is the $m$-
th unit of the $j$-th feature map $Q_j$ of the convolution layer, $w_{i,j,n}$ is the $n$th element of the weight vector, $w_{i,j}$, connecting the $i$th feature map of the input to the $j$th feature map of the convolution layer, and $F$ is called the filter size which is the number of input bands that each unit of the convolution layer receives.

So far so good:

What i basically understood from this is what I've tried to illustrate in this image.

It seem to me what they are doing is actually processing all data points up to F, and across all feature maps.  Basically moving in both x-y direction, and compute on point from that.

Isn't that basically 2d- convolution on a 2d image of size $(I x F)$ with a filter equal to the image size?.
The weight doesn't seem to differ at all have any importance here..?


texts:
I am working on traffic violation data set which contains 36 columns(variable). I have two dependent variables out of this.

Example

id, description, age, sex    alcohol     vehicle     violation level Accident

120   speed      28    M       Yes         car             severe       yes
122   win-tint   35    F       No         truck            Medium       no
123   left-turn  26    M       No          car             light        no
124  failure-veh 28    M       No          car             severe       yes

The dependent variables are "violation level" and "accident".

violation level has 3 classes -- sever medium light

Accident has 2 levels -- Yes No

I want to predict both violation level and accident. I think this can be multitarget prediction.

Can someone help me which algorithm is good for this? I have seen some articles suggesting scikit- multiout put classification and neural network with multiple out put layers.

Or Can I go ahead with two models?

predicting the traffic violation level.

Predicting the accidents using the violation levels.

Basically I am trying to predict violation levels and then predicting how these violations contribute to accidents.

Any help would be appreciated.


texts: How to predict the duration of burst given several series

texts: I've got sets of time series data collected from Weibo. It contains the number of posts under certain topics over a year. It can be found that from time to time there're bursts of discussion on certain topic. What's interesting is some burst die down quickly while others sustain on the agenda for quite a long time and die down gradually. How to predict and model the decays given a time series data. Also, I've got data of some exogenous data including time series data of links, pictures, and retweeting and original data in the community. Can I predict the duration of decay from these variables?


texts: Data augmentation for multiple output heads in Keras

texts:
I have a transfer learning based two output classification problem. So, accordingly, I have formatted my data to have X_train as a (number of samples, height, width, channels) numpy array, y_train1 as (number of samples,) numpy array and y_train2 as (number of samples,) numpy array.

As I am not training using directory structure, I am using ImageDataGenerator.flow(). However, I am not able to figure out how I can pass two label arrays because, it is taking the labels as (2, number of samples) when I send it as [y_train1, y_train2] list.

I am able to train the network without Keras data augmentation (for two outputs). But, I am not able to apply data augmentation.

I am trying to do the following:

datagen = ImageDataGenerator(horizontal_flip=True, vertical_flip=True, rescale=1./255, class_mode="multi-label")

model.fit(datagen.flow(X_train, [y_train1, y_train2], batch_size=batch_size), batch_size=batch_size, epochs=nb_epochs, steps_per_epoch=spe, validation_data=(X_val, [y_val1, y_val2]))

Also, ImageDataGenerator.flow does not have class_mode unlike ImageDataGenerator.flow_from_dataframe.

Any suggestions/help would be appreciated!

References:

Data Augmentation Multi Outputs (No answer. I upvoted this just now)
Get multiple output from Keras (Does not explain data augmentation)


texts: How many images can be trained in Google Colab?

texts:
I am using the ResNet50 pretrained model to train my images using TensorFlow. I have 70k images and upgraded to Google Colab Pro, but still I am facing a memory error. So how many images I can train in Google Colab? And how much RAM is needed for 70k images?
This is how I labeled and loaded images from the drive.
labels = []

imagePaths_generater = paths.list_images(Config.DATASET_PATH)
imagePaths = []
for item in imagePaths_generater:
  imagePaths.append(item)

for imagePath in imagePaths:
  label = imagePath.split(os.path.sep)[-2].split("_")
  #print(label)
  image = load_img(imagePath, target_size=(224, 224))
  image = img_to_array(image)
  image = preprocess_input(image)
  data.append(image)
  labels.append(label)

data = np.array(data, dtype="float")
labels = np.array(labels)```


texts: How to normalise estimate weighted on correlations (some will obviously be negative)

texts:
I would like to work out an estimate for some variable Xi using values of a bunch of other variables Xj, j in Y (I guess this should include Xi too) and how correlated they are to Xi.

So if I do weight(j)=correlation(i,j)/sum over Y of other correlations, then this sum can be zero but I feel that might not be the only problem. I could use the min-max remapping but then the most negatively correlated variables will have no or little effect, whereas in fact they should have a strong negative effect.

One obvious fix would be to ABS() the sum but I could not prove to myself that this is the correct thing to do. It may be possible that each correlation needs to ABSed in the sum (then it would resemble the way stdev is calculated - then do I need to square my correlations?).

I simplified the problem I am trying to solve for this question but I am not sure if the details would affect the solution. There is a time dimension to it, and I am trying to use changes over each variable over time and their correlations to work out the estimate for the next change or a chain of changes I then multiply with the current value. (I don't know if this step is necessary or if proper normalisation of correlations would correct for this even if I used absolute values and not relative changes).


texts: Can SageMaker DeepAR algorithm only deal with training data that resides in Amazon S3 buckets?


texts: How to determine if my GBM model is overfitting?

texts:
Below is a simplified example of a h2o gradient boosting machine model using R's iris dataset. The model is trained to predict sepal length.

The example yields an r2 value of 0.93, which seems unrealistic. How can I assess if these are indeed realistic results or simply model overfitting?

library(datasets)
library(h2o)

# Get the iris dataset
df <- iris

# Convert to h2o
df.hex <- as.h2o(df)

# Initiate h2o
h2o.init()

# Train GBM model
gbm_model <- h2o.gbm(x = 2:5, y = 1, df.hex,
                     ntrees=100, max_depth=4, learn_rate=0.1)

# Check Accuracy
perf_gbm <- h2o.performance(gbm_model)
rsq_gbm <- h2o.r2(perf_gbm)

---------->

> rsq_gbm
[1] 0.9312635


texts: Is it possible to create a rule-based algorithm to compute the relevance score of question-answer pair?

texts:
In information retrieval or question answering system, we use TD-IDF or BM25 to compute the similarity score of question-question pair as the baseline or coarse ranking for deep learning.

In community question answering, we already have the question-answer pairs to collect some statistics info. Without deep learning, could we invent an algorithm like BM25 to compute the relevance score of question-answer pair?


texts:
When training a neural network, I usually plot the accuracy obtained on the validation data (validation accuracy) as an intermediate measure of the network's performance – the final measure being test accuracy. This validation accuracy is measured one or more times during a single epoch, depending on the size of the training data.

Usually, I stop training when the validation accuracy starts dropping – a sign of overfitting.

However, for large datasets I often don't have enough computational power to wait for the validation accuracy to drop. I merely stop the training after the validation accuracy (or validation loss) stays constant for a while, like so:

Technically, I do not know if the model can improve further, but is it reasonable to assume that it won't? Again, this is only for large datasets with limited computing power available. With sufficient computing power available, I can afford to wait until validation accuracy drops.


texts: How to concatenate feature vectors of different dimensions?

texts:
I have been using different deep learning models and extracting features from different layers for the given images. My code goes like this:

#load libraries
from keras import applications
from keras.preprocessing import image
from keras.applications.resnet50 import preprocess_input
from keras.models import Model
import numpy as np

#declare models
model1 = applications.VGG16(weights='imagenet', include_top=False)
model1.summary()
model1 = Model(inputs=model1.input, outputs=model1.get_layer('block5_conv3').output)
model1.summary()
img_path = 'elephant.jpg'
img = image.load_img(img_path, target_size=(100,100))
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
x = preprocess_input(x)

block5_conv3_features = model1.predict(x) #dimensions (1,6,6,512)

model2 = applications.VGG19(weights='imagenet', include_top=False)
model2.summary()
model2 = Model(inputs=model2.input, outputs=model2.get_layer('block2_conv1').output)
model2.summary()
img_path = 'elephant.jpg'
img = image.load_img(img_path, target_size=(100,100))
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
x = preprocess_input(x)

block2_conv1_features = model2.predict(x) #dimensions (1,50,50,128)

model3 = applications.ResNet50(weights='imagenet',include_top=False)
model3.summary()
model3 = Model(inputs=model3.input, outputs=model3.get_layer('avg_pool').output)
model3.summary()
img_path = 'elephant.jpg'
img = image.load_img(img_path, target_size=(224, 224))
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
x = preprocess_input(x)

preds = model3.predict(x) #dimensions (1,1,1,248)

I'm getting feature vectors of dimensions (1,6,6,512), (1,50,50,128) and (1,1,1,128) from three different models. How can I concatenate these feature vectors in Python?


texts: Identifying important interactions between features using machine learning

texts:
Let's say I have a set of features: a, b, c, d, e, f. I'm now interested in identifying possible interactions between these features that best predict an outcome. For example, it could be that the features a, f, and the interactions a:b:g, d:f and c:e are the 5 most important factors that predict the outcome. It is not only important for an algorithm to account for feature interactions, but I also want to be able to identify these interactions.

How could I approach this problem with machine learning?


texts: ValueError: Found input variables with inconsistent numbers of samples: [2, 44]

texts:
Have a piece of code where i am cleaning the text from the 'Description' column and storing it as "cleaned"

Then i create a ML model using the above as one of my features.

       X = data[['originalname','cleaned']]
       Y = data['Total score']

       X_train, X_test, y_train, y_test = train_test_split(X,Y,
       test_size=0.2,random_state=42)

       pipeline = Pipeline([('vect', TfidfVectorizer(ngram_range=(1, 2),
       stop_words="english", sublinear_tf=True)),
       ('chi',  SelectKBest(chi2, k='all')),
       ('clf', LinearSVC(C=1.0, penalty='l1', max_iter=300, dual=False))])

       X_train.shape--->(44, 2)
       y_train.shape--->(44,)

Trying to train the model gives me the above error

       model = pipeline.fit(X_train, y_train)

How do i use both 'original description' and 'cleaned' as my feature to predict 'Total score' without the above error ?


texts: If a regression problem is reduced to classification, does minimizing the classification loss translate to minimizing regression error and hence better regression performance?


texts: What if multiple final prediction values for multi-class Neural Network are equal

texts: If, for example, your final prediction for a multi-class problem, say for ["mouse","cat","dog","lion"], is [0.1,0.3,0.3,0.3], should the neural network predict that this data is "cat","dog" or "lion"? Should this be picked randomly?


texts: Can you reduce many Naive Bayes training samples to a few big ones?

texts:
Excuse me if this is a simple question (I'm a newbie to data science).
I am trying to implement a Naive Bayes classifier in Python.

I have implemented a standard Naive Bayes classifier (Bernoulli) and this went very well.
Suppose I have 50,000 samples where the first half is labelled '0' and the second half '1'.
Then the shape of the X_train matrix will be (#number_of_unique_words/features, 50000) and shape of y_train (50000,).

From how I understand Naive Bayes, I can now also join the first 25000 documents to represent one big document with label '0' and one big document with label '1'. However, this gives me very different results when I try to predict the test set.
Is my understanding of Naive Bayes correct or is my implementation just wrong?


texts: Same SVM configuration, same input data gives different output using Matlab and scikit-learn implementation of SVM, in a classification problem

texts:
I have a classification problem with 60 data points in a 2-dimensional feature space. The data originally is divided into 2 classes. Earlier I was using Statistics Toolbox of Matlab so it was giving me fairly good results.It was giving 1 false negative and no false positive.

I used the following code:

SVMstruct = svmtrain(point(1:60,:),T(1:60),'Kernel_Function','polynomial','polyorder',11,'Showplot',true);

I am using a polynomial kernel and with polynomial order of 11.

But when I use same kernel configuration in scikit-learn SVC it does not gives the same result rather it gives very undesirable result with classifying all of them to single class.

I am using it as

svc = svm.SVC(kernel='poly', degree=11, C=10)

I have used with many values of C too. No major difference.

Why there is so much difference in results ?
How can I get same result as I got using Matlab ?
For me it is compulsory to do with python-scikit.


texts: Loss decreasing faster = lower convergence?

texts:
I am trying to optimize a generative deep LSTM network but I am unable to train each model until it converges because it would take a very long time and cost too much on AWS. I am grid searching through depths of hidden layers and number of nodes within those layers using the same same learning rate.  I intend to choose the model that produces the lowest loss for that limited training run on the 15th epoch because at that point they still have not converged.

Is this a good method? Namely, will the structure that trains the fastest converge at the lowest loss, or are there cases when a certain structure might reach a lower loss at a slower rate?


texts:
I have an exploratory script running a Databricks notebook that performs a simple arithmetic function (Pythagorean theorem) on all possible pairwise combinations of a list of pairs of floats (akin to coordinates).

The values are generated randomly, like so:

vals = np.random.rand(num_samples, 2)

The list is then converted to 2 RDDs of Rows, like so:

rdd = sc.parallelize(vals)
rows_1 = rdd.map(lambda v: Row(x=float(v[0]), y=float(v[1]), join_val=1))
rows_2 = rdd.map(lambda v: Row(x_r=float(v[0]), y_r=float(v[1]), join_val=1))

Which are then registered as tables:

sqlContext.createDataFrame(rows_1).registerTempTable('sdf_1')
sqlContext.createDataFrame(rows_2).registerTempTable('sdf_2')

sdf_1 = sqlContext.table('sdf_1')
sdf_2 = sqlContext.table('sdf_2')

Each table contains the same content, just different columns names. The two are then joined:

sdf_1.join(sdf_2, sdf_1.join_val==sdf_2.join_val).registerTempTable('sdf_join')
sdf_join = sqlContext.table('sdf_join')

With the tables joined, the following UDF is defined:

def calc_dist(x1, y1, x2, y2):
  return math.sqrt((x1-x2)**2 + (y1-y2)**2)

calc_dist_udf = udf(calc_dist, FloatType())

Finally, the operation is performed on all rows:

sdf_join\
  .select(calc_dist_udf('x', 'y', 'x_r', 'y_r').alias('dist'))\
  .filter('dist<0.05')\
  .count()

This operation completes successfully, but I have noticed that, as num_samples increases, the execution time increases exponentially. I believe I am failing to correctly parallelize the row-wise operation.

Is this assumption correct?
How can I achieve parallelization on such an operation?


texts: What is the relationship between hard-sigmoid function and vanishing gradient descent problem?

texts:
I think that the vanishing gradient problem occurs when the derivative of activation function is less than 1.
The deeper the neural network is, the f' * f' * f'... operation will result in the gradient closer to zero.

However, if we use hard-sigmoid as an activation function (its derivative is 0 or 1), the vanishing gradient problem can be solved.

I think the dying problem (derivative = 0 @ ReLU) is different from the vanishing gradient problem.

Is it right?


texts: Are calibrated probabilities always more reliable?

texts:
EDIT: Based on the answer below, I have updated the question and added more detail.

I have applied Dirichlet calibration to my fast-bert sentiment classification model, and I am struggling to really understand why/ if it is actually more reliable.

Application:

def dirichlet_scaling(train, y_train, test):
    logreg = LogisticRegression(multi_class='multinomial', solver ='newton-cg')
    logreg.fit(np.log(train), y_train)
    return logreg.predict_proba(np.log(test))

train --> validation set, size ~ 11000

test --> test set, size ~ 7000

These are the uncalibrated probability reliability diagrams:

The classwise diagrams below are based on this recent article https://arxiv.org/abs/1910.12656.

Evaluation metrics:

Confusion-Matrix:

After applying Dirichlet scaling, as per the article, I get these classwise-reliabilities:

And overall confidence calibration:

Dirichlet Evaluation Metrics:

Dirichlet Calibrated Confusion Matrix

After calibration, the minority class - mixed sentiment - is less accurately predicted. The sample below is of mixed samples with high uncalibrated probabilities (> .9).

These cases are all undoubtedly mixed (even the incorrect label). However, after calibration they are less confidently predicted.

My main question is:

The estimator over-estimates the minority class and under-estimates the majority class, which balances the class imbalance - calibrating the probabilities seems to reverse this to a certain extent.

After calibration the class imbalance appears more pronounced in the predictions. I may very well be wrong, but the miscalibration in the classwise reliability diagrams appears to be the classifier incorporating the class distributions (class priors). Is it not usually the case that when imbalanced datasets are artificially balanced, the class priors are reintroduced to balance the probabilities? Does this technique also reduce calibration?

Is it possible the classifier was already well calibrated? And that the over/under-estimation seen in the classwise-reliability diagrams is perhaps not always indicative of miscalibration if overall confidence is well calibrated?


texts: How we can Identify Specific Feature from a larger amount of Dataset?

texts:
In Machine Learning, we need to play with any kind of datasets.
In the Dataset, There are too many records and features, Some datasets had lots of features (sometimes it's called columns),
So the main problem for a data scientist is to know the behavior of the dataset and get meaningful insights from the dataset,
Let's take one example from Kaggle platform, There's a dataset for house price prediction to know what the price of a house, based on there features,
here is the link of the datasets
House Price Prediction Advance Regression Machine Learning Problem

So, The question is how to identify the meaningful feature from the dataset?


texts:
If loss is decreasing but val_loss not, what is the problem and how can I fix it?

I get such vague result:


texts:
Recently I've been trying to implement Lasso by myself in R, not using the "glmnet" package, and based on an article by Tibshirani I wrote a raw code to implement coordinate descent method, and it does successfully force many coefficients to shrink to zero. But when I try to compare my result based on a famous R built-in data set(mtcars) with glmnet package, and apply different lambdas, only to find out that while other coefficients are reasonable, one coefficient that should be insignificant remain quite large. I've been trying to figure out the reason. Specifically I have some detailed questions:

How to deal with the intercept in the formula? When the data set is scaled, it seems that the $\beta_{0}$ can just be set to the mean value of the response variable. But even the glmnet package produces a different intercept when lambda is changed in the model.
What could be the possible reason for the problem as I described above?

Hope someone could lend a hand, thanks! If any more details are needed, I can provide them.

Here's my source code in Git. I've also described my problem in detailed in README.md.


texts: What exactly means CNN is position equivariant

texts:
There is quite a good explanation which fully comply with my vision. But seems it lacks one final step. As Jean states, moving an object significantly in the input image will cause the change in which neuron is activated in the yellow layer (the one previous to the first fully connected layer). So that we see that the part of the network before FCL is  position equivariant. Then  author says that because the network detects an object at any location, the FCLs should have taken care of it.

Does the equivariance property hold for the whole network, including trailing classifier? (I've read Difference between "equivariant to translation" and "invariant to translation" but not sure I apply it to this case correctly)
Should a network detect an object in right-top corner if only trained with images with that object at left-bottom corner?

(Tried  this demo but it doesn't seem to address the second question)


texts: Any efficient way to build non-linear regression model for polynomial features?

texts:
I am trying to understand how crime frequency affect house price in certain area. To do so, I started with Chicago crime data and zillow real estate data. I want to understand the relation between house price and crime frequency and top 5 crimes in certain areas. Initially, I build up model for this specification, but it wasn't very meaningful to me. Can anyone enlighten me what should I do? any efficient approach to train regression model for potential relation between house price and crime frequency in certain areas? any heuristic idea to move forward?

example data snippet:

here is the merged data that includes annual house price and top crime type in certain areas:

Here is reproducible example data snippet

my attempt

so here is my attempt to fit regression model with above reproducible example data:

from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import StandardScaler
import pandas as pd

regDF = pd.read_csv('exampleDF')

X_feats = regDF.drop(['Avg_Price_2012'], axis=1)
y_label = regDF['Avg_Price_2012'].values

sc_x = StandardScaler()
sc_y = StandardScaler()
X = sc_x.fit_transform(X_feats)
#y= sc_y.fit_transform(y_label)
y = sc_y.fit_transform(y_label .reshape(-1,1)).flatten()
regModel = LinearRegression()
regModel.fit(X, y)
regModel.coef_

but to me, above model wasn't that efficient and needs to be done something more. I think I have to use non linear regression model for those polynomial features, and I am not sure to get this done.

Can anyone point me out how to build correct model for house price prediction over type of crimes and frequencies in certain areas? any idea? Thanks

Goal:

I want to build regression model to predict house price based on crime frequencies and types in certain areas. How can I get modeling the relationship between house price and crimes in certain areas? any thoughts?


texts: Siamese Network - Sigmoid function to compute similarity score

texts:
I am referring to siamese neural networks introduced in this paper by G. Koch et al.
The siamese net computes 2 embeddings, then calculates the absolute value of the L1 distance, which would be a value in [0, +inf). Then the sigmoid activation function is applied to this non-negative input, so the output afterwards would be in [0.5, 1), right?
So, if two images are from the same class, your desired L1 distance should be close to 0, thus the sigmoid output should be close to 0.5, but the label given to it is 1 (same class); if two images are from different classes, your expected L1 distance should be very large, thus the sigmoid output should be close to 1, but the label given to it is 0 (diff. class).
How does the use of a sigmoid function in order to compute the similarity score (0 dissimilar, 1 similar) in a siamese neural network make sense here?


texts:
I am new to data science and I am trying to understand the use/importance of accuracy, precision, recall, sensitivity and f1-score when I have a confusion matrix.

I know how to compute all of them but I cannot really understand which of them to use each time.

Could you give examples where for instance precision is a better metric that recall or where the f1-score gives essential information that I cannot get from the other terms ? In other words, in which cases should I use each of the aforementioned terms ?


texts:
I have a streaming data along with timestamp dataset that looks like this:
1.png
Timestamp can be inclusive of "seconds" too, but the data may or may not change every second. it depends on the previous values(rows i.e data which came earlier w.r.t time).
Column1, Column2 .... ColumnN correspond to the variables (they change over time) and "Label" shows the different samples. You can assume that the values tend to decrease over time for a particular label.
Labels A1,B1,C1.........A2,... M labels.
Note : Values of timeNew of a Label depends on values of  timeOld of that Label and Labels belong to its cluster.
I need to group Labels with similar behavior over time together (e.g. Label A1 and Label C1 should be put in the same cluster and B1,D2 may fall into same cluster over time as they tend to behave similar over time).
I thought of using DTW and get the similarity of each Label with respect to other Labels. but not sure, how to proceed when i have N Columns.
To be precise, i need to group Labels based on their similarities (Column1 .. ColumnN) over time and group them.
Once i group them when new data comes in i should be able to predict the values(Column1.. ColumnN) for a Label based on the previously seen data(can be just minutes closer to the current prediction) and the values associated with the Labels in its cluster and predict it accordingly.


texts: Role of a known predictor in a disease prediction

texts:
I am working on finding out whether the patient will develop the disease or not in a hospital.

Might be a basic info but I am just sharing it anyway.

Usually through historic data, I was able to see that patient who was in hospital or the icu for a longer time (more days) did develop the disease.

Similarly if he was in ventilation, it is also an indication of his health status

The distribution of hospital stay and ventilation hours between patients who developed disease and didn't develop disease are different and I verified them visually.

Now my question is

1) Do we need to include these as a predictor (input variable) in our model? I ask because am thinking by using these two variables, model might miss some other input variables which has an influence on outcome. For ex: I may not know whether his urea reading is an indication of whether he will develop disease or not.

Basically what I am trying to know is should we feed variables which we are confident that they will impact the outcome into the model?

Or is the model to help us know something which we aren't aware of .

Can you help me with this?


texts: how to know if there is a bias in data collection methods

texts:
I am collecting data for machine learning models I want to build for some application.
I started with random sampling (just simply collecting 'recent' data) but I am not getting enough records of interest, which I want to use as my target. The process is currently automated.
I don't want to use SMOTE unless it works really well with text(NLP) data.
I also don't really want to tackle this as an anomally detection either.
Idealy, I want to build a look-alike model to give "authenticity score".
I could manually search for targets I am lacking or set up another collecting method, which may be more efficient than the random sampling.
What I am afraid of is that this might introduce some biases in my data.
How can I know if there is a bias?
My data consists of text, categorical, and continuous variables.
If all the samples are mixed and shuffled nicely, would that be alright unless training data is from one sampling method and testing data is from another?


texts:
I have a big database with 40k recors and 2 classification classes. In this big database the 76% of records belong to the first class.

I've used a 70-30 split partition with stratified sampling, and the K-nn gives the best accuracy on k = 20.

1) Is it too big value for k ?

2) Is it possible that this big value for k is due to the disproportion of the 2 classes in the database , even if i used a stratified sampling ?


texts: I am interesting in knowing an approximate amount of SURF features used to do e.g. image-recognition on 224x224x3 images. Is there a standard amount of features used? Or perhaps a well-acclaimed article using SURF features that I could look at? I am not going to work with the features, but I am just trying to understand some aspects of their usage.


texts:
I am trying to transform two datasets: x_train and x_test using tsne. I assume the way to do this is to fit tsne to x_train, and then transform x_test and x_train. But, I am not able to transform any of the datasets.

tsne = TSNE(random_state = 420, n_components=2, verbose=1, perplexity=5, n_iter=350).fit(x_train)

I assume that tsne has been fitted to x_train.

But, when I do this:

x_train_tse = tsne.transform(x_subset)

I get:

AttributeError: 'TSNE' object has no attribute 'transform'

Any help will be appreciated. (I know I could do fit_transform, but wouldn't I get the same error on x_test?)


texts: Splitting hold-out sample and training sample only once?

texts:
I have a question related to evaluating out-of-sample predictions.

For my research I want to tune two parameters related to Support Vector Machines, and use these optimized parameters to predict the hold-out sample as good as possible. To evaluate my model I obviously have to split my data in a training sample (80%) and a hold-out sample (20%). When tuning the parameters I also use 10-fold CV, but this only involves the training sample.

Now I thought that my approach is not super valid, as the 20-80 split of hold-out versus training sample is only done once, and thus might be too subject to randomness. However, I feel that this approach is used quite often, for example in http://www.sciencedirect.com/science/article/pii/S095741740400096X

In my earlier research I have done leave-one-out predictions, where I used n-1 observations and predicted the remaining observation (so that every single observation is predicted once, using the model that is trained based on only the other n-1 observations). However, in my new project this is not possible, because I then also need to tune the parameters n times, which would require too much time and would mean I lose the interpretation/plots relating to these parameter values.

In short: is it a bad thing that I only split my training and hold-out sample once? Does anyone have any comments about my current approach?


texts: How does training in batches help in obtaining a better deep learning model? What should one keep in mind while splitting data into batches?


texts: How does a random forest algorithm deal with a few irrelevant input variables

texts: I have a list of variables from which I would like to train a Random Forest Algorithm. I suspect that a few of my input variables, which have noisy distributions, won't be able to predict much. Can I use them anyway, knowing the algorithm will eliminate them in the process, or should I beware that these variables may biais my model?


texts: How to effectively tune the hyper-parameters of Gensim Doc2Vec to achieve maximum accuracy in Document Similarity problem?

texts:
I have around 20k documents with 60 - 150 words. Out of these 20K documents, there are 400 documents for which the similar document are known. These 400 documents serve as my test data.

At present I am removing those 400 documents and using remaining 19600 documents for training the doc2vec. Then I extract the vectors of train and test data. Now for each test data document, I find it's cosine distance with all the 19600 train documents and select the top 5 with least cosine distance. If the similar document marked is present in these top 5 then take it to be accurate. Accuracy% = No. of Accurate records / Total number of Records.

The other way I find similar documents is by using the doc2Vec most similiar method.  Then calculate accuracy using the above formula.

The above two accuracy doesn't match. With each epoch one increases other decreases.

I am using the code given here: https://medium.com/scaleabout/a-gentle-introduction-to-doc2vec-db3e8c0cce5e. For training the Doc2Vec.

I would like to know how to tune the hyperparameters so that I can get making accuracy by using above-mentioned formula. Should I use cosine distance to find the most similar documents or shall I use the gensim's most similar function?


texts:
I have set of topics as follows.

"web based", "web-based" -> with surplus symbols
"technology","technologies" -> with singular and plural
"learned", "learnt", "learning" ->suffix stripping

Can you please recommend an accurate tool to perform aforementioned tasks.


texts: How exactly does matrix factorization help with collaborative filtering

texts:
We start with a matrix of user ratings for different movies with some elements unknow i.e the rating of a yet to be seen movie by an user. We need to fill in this gap. So

How can you decompose or factorize a matrix where all elements are not known in advance.
after you somehow decompose it, do you just multiply the matrices back to recreate the original one, but now with the missing item populated?
How do you know which factorization method like non negative,singular value,eigen to choose,without going into too much math?


texts: Clustering based on partial information?

texts:
I'm open to suggestions on how to improve the title.

My problem is this, but I think it's a more general problem. In my context, I have a lot of data which has location data (Lat/Lon) as well as various characteristics of my objects.  I'm interested in clustering based on location in such a way that it controls for effects based on weather and other local effects.

So what I want is a clustering which, after trained, determines the cluster of a new object solely based on its GPS coordinates. However, I want the clusters not to just cluster based on locational clusters (there are lots of objects in this ball in Florida, this ball in California, etc.) but to make clusters so that within the clusters, objects are more similar than you would expect.

So for example, k-means would not accomplish this; it would just identify pockets of objects based on location clustering, or it would require more information to determine the cluster of a new object (neither are desirable).

Another way would be to run k-means a bunch of times, rate the results based on comparing homogeneity within clusters, and pick the best, but this seems both computationally annoying and unlikely to get good results (it would tend to identify circular clusters, rather than e.g. picking out things within 10 miles of a coast, or whatever turns out to be useful).

I feel like someone has thought of this before and come up with something.  What sorts of algorithms (classical or not) might help with this?

I don't mind re-implementing something on my own, and I don't mind stacking/etc. classifiers.  My dataset is fairly small for now, so even if there isn't a fast algorithm for this, it should be okay (even $O(n^2 \log(n))$ algorithms are doable, I've found).


texts:
I want to setup a decent framework for building and deploying ML code on our servers. The server side code is all java. I have inherited a system that uses weka as it is based on java and makes life easy for the server side folks. But  data scientists who come in, use R/python to do their modelling and then write some glue code to make the models run in java environment on weka. This library is getting outdated(works alright though), but mainly writing this glue code is getting cumbersome/repetitive and can be done away with. I'm looking for an optimized pipeline that helps me get models deployed quickly.

How is this done in the industry? What does your pipeline look like?

I would like to be able to quickly model using one language that data science team uses(I'm tending to python) and be able to deploy/call these models for prediction on the server side without having to redo a bunch of stuff to get it working there.


texts: What is the difference between reconstruction vs backpropagation?

texts:
I was following a tutorial on understanding Restricted Boltzmann Machines (RBMs) and I noticed that they used both the terms reconstruction and backpropagation to describe the process of updating weights. They seemed to use reconstruction when referring to the links between the input and the first hidden layer and then backpropagation when referring to the links to the output layer.
Are these terms used interchangeably or are they different concepts?


texts: First i want to tell you that i'm trying to create a model in the right steps, which needs data exploration, data preprocessing (AFTER splitting the data into train/test), etc. For the part of data exploration i want to plot the data for visualizing the correlation,distribution, etc. I don't know when i supposed to do this so i decided to plot all the things i need BEFORE splitting the data into train/test. One of the plot is the correlation with seaborn heatmap, my data contains numerical and categorical values (which needs to be encoded to binary number), since i preprocess the data AFTER the splitting, the heatmap cannot include the categorical values since its not binaried yet. Now i'm really confused of how to tackle this issue, i still need to know the correlation between the categorical feature with the target but if i encode it BEFORE the split train/test, it will defies the proper manner that i want to follow. Anyone has solution or maybe opinion?


texts: How to tune an weighted voting ensemble method?

texts:
I am working on kidney cancer patients' data with 5 unbalanced labels. These codes are contained of Normalization, Oversampling on Feature Engineering part. A list of 9 ordinary  Machine Learning methods is provided which are used for the classification task. Then, I take advantage of two kinds of ensemble methods of hard voting and weighted voting methods. 10-fold CV has is exploited to validate results.
methods = ['Support Vector Machine', 'Logistic Regression', 'K Neighbors Classifier', 'Random Forest', \
          'Gaussian Naive Bayes', 'Linear Discriminant Analysis', 'Decision Tree', 'Gradient Boosting',\
          'soft_VotingClassifier','hard_VotingClassifier']

I would like to know how to tune weights for the soft voting method one?
Here, are the code and results I have right now:
from sklearn import preprocessing
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import SVC
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.ensemble import VotingClassifier

if method == 'soft_VotingClassifier':
    cl1 = LogisticRegression()
    cl2 = KNeighborsClassifier(n_neighbors=10)
    cl3 = RandomForestClassifier(max_depth=35)
    cl4 = GaussianNB()
    cl5 = LinearDiscriminantAnalysis()
    cl6 = DecisionTreeClassifier()
    cl7 = SVC(C=0.1, gamma=0.0001, kernel='poly')
    cl8 = GradientBoostingClassifier()
    estimator = [(method[0],cl1), (method[1],cl2), (method[2],cl3), (method[3],cl4),\
                 (method[4],cl5), (method[5],cl6), (method[6],cl7), (method[7],cl8)]
    eclf = VotingClassifier(estimators=estimator,
    voting='soft', weights=[5, 5, 10, 5, 6, 8, 4, 10])

if method == 'hard_VotingClassifier':
    cl1 = LogisticRegression()
    cl2 = KNeighborsClassifier(n_neighbors=10)
    cl3 = RandomForestClassifier(max_depth=35)
    cl4 = GaussianNB()
    cl5 = LinearDiscriminantAnalysis()
    cl6 = DecisionTreeClassifier()
    cl7 = SVC(kernel='linear',gamma='scale')
    cl8 = GradientBoostingClassifier()
    estimator = [(method[0],cl1), (method[1],cl2), (method[2],cl3), (method[3],cl4),\
                 (method[4],cl5), (method[5],cl6), (method[6],cl7), (method[7],cl8)]
    eclf = VotingClassifier(estimators=estimator, voting='hard')

Confusion Matrix results on test data:
[50  5  1  4  2]
[ 0 13  1  0  3]
[ 0  1  2  1  1]
[ 4  0  0  2  0]
[ 0  3  0  0  2]

Accuracy result on test data:

For those kind people who want to check my code or even run them, I gonna put the repository link.


texts: Can macro F1 score be greater than micro F1 score?

texts: I am reading about evaluation metrics, and it seems that micro scores are more useful. But I was wondering about scenarios where macro F1 score is greater than micro F1 score, and if this is at all possible.


texts: How to interpret my neural network with high accuracy but low probability in test results

texts:
I have built a classical ANN using keras which provides probability (using sigmoid function) of the outcomes (0 or 1). While the accuracy of the model is high when the model is fit ~90%, the outcome probability of the test set results is very poor. How can I interpret this?

Build ANN

classifier = Sequential()
classifier.add(Dense(activation="relu",input_dim=7,kernel_initializer="uniform", units = 4))
classifier.add(Dense(activation="relu",kernel_initializer="uniform", units = 4))
classifier.add(Dense(activation="sigmoid", kernel_initializer="uniform", units = 1))
classifier.compile(optimizer="adam", loss="binary_crossentropy",metrics=['accuracy'])
classifier.fit(X_train,y_train, batch_size=10,epochs=100)

Predict results:

y_pred = classifier.predict(X_test)

I have attached the test set results where result probabilities can be seen with respect to frequency. blue shows probabilities for 1 and orange for 0

Is this a case of overfitting? How can I tweak an ann to avoid overfitting?


texts: Market analysis using wheighted items-Data Modelling

texts:
I'm working on finding some way to return all associated skills when given a skill . Let me explain : suppose I know Java as a skill, then by association, I also do know Hibernate,Spring,J2EE..etc. I've been searching for a way so that I can implement Market analysis so that it will give me all possible associated skills of one and also the cost of hopping from one skill to another.
The last part is tricky and that's where I'm stuck.
This is what I imagine the dataset to be:
skill        association-with-cost
java         Hibernate,3
java         spring,1
java         c#,2
c#           java,3
c#           asp.net,1
c#           linq,1

Will APriori or FP tree work for what I have in mind? is the dataset example I've given good for it or should I come up with a new model? How can I do this?
Many thanks!


texts: Which words are causing wrong prediction

texts:
I have a simple multiclass text classification code which uses -
split words -> remove stop words ->apply TFIDF vectorizer -> RandomForest (or LinearRegression or NaiveBayes)

During prediction some sentences which are similar to trained sentences but have some extra words are causing low confidence for a class prediction.
I want to automatically identify those words which contributed to reducing the confidence score.
Any advice?

Example (2 class prediction):
I want to know status of this task -> Status Update (0.98 score)
I am working since morning and during and discovered that task status is incomplete so spent extra hours to complete the work. -> Status Update (0.6 score), TaskFinish (0.4 score)

In above hypothetical example, in second sentence, words like "complete" tried to change the class and words like "working", "discovered" etc which are not useful words causing low score for status update. These words I want to identify.


texts: What to give as predictors to predict future values?

texts:
I am new to machine learning techniques. I was going through few supervised machine learning model examples and i have doubt in predicting future values.

I have daily time series dataset from database where my target variable is complete noise signal like this:

To train and predict the models, we divide the dataset into train set, validation set and test set to check if the model is efficient or not. I have two independent variable and one target variable. I am using linear regression,Keras LSTM and other models.

My basic question is how do i predict for future values (for next one week, one month) when i don't have independent variables for next week? what am i supposed to give as predictors in that case?

Any information is much appreciated.


texts: Machine learning algorithm that can use many instances to predict 1 continuous outcome per person

texts:
I am trying to use movements identified from accelerometers during sleep to predict gait speed (continuous). I am trying to figure out what the best machine learning algorithms/ feature extraction methods can be used to use as much of the information as possible in this prediction.

I am currently identifying every movement that occurs during sleep for each night measured and then extracting features from the accelerations of each movement. I then average all of these features over each movement identified during the night and then each night measured for that person, resulting in 1 set of features per person. This can be easily used in machine learning algorithms to predict gait speed as a continuous variable.

However, I think by averaging over all of the movements per night and then over all of the nights collected, I am losing a lot of information about the individual movements of each person. Is there a way to include all of the movements (~200-1000) from one person as instances that can be used to predict the 1 set of outcomes per person? If not, is there a better way to condense the number of movements in into 1 instance per person without losing a ton of information from taking an average? Maybe using clustering?

Thanks for any input!


texts:
I am new in python and data science (and not great in math). I am learning machine learning. I got following normalize function. Can you please explain what does this normalize function do?

def normalize(array):
    return (array - array.mean()) / array.std()

Also please explain what this array - array.mean() does?


texts: Detecting seasonality in timestamped events

texts:
I have a program that detects events in a large amount of measurement data. When it detects an event, it writes a timestamp. I have thousands of event timestamps. What I wish to do is detect if there is seasonality in the timestamps I have. I am not well versed in the terminology used, but I think seasonality is what I want to find.

Pictures may aid my explanation. If I have a bunch of events on a timeline, as in the figure below, the events all seem to be random but there may be some kind of seasonal component to the events.

What I wish to do is detect if any of the seemingly random events are following a strict repeating interval. An illustration is given below, where we see that in the seemingly random events above there are some data points that are repeating with a fixed frequency.

I am not certain what kind of method to apply. I have looked into power spectral density, fourier transform and ARIMA, but I am still in the idea phase.

Properties of the applied method should possibly include:

A quantitative measure of how strict or fixed the intervals are, or how
certain we can be that we have detected a fixed cycle
The ability to detect seasonality on different timescales (e.g. events occurring multiple times within the same hour or multiple times during a week with a fixed pattern)

What kind of method is applicable here?


texts: Time Series Forecasting RNN: Masking Values

texts:
Suppose you have missing values in a time series E.g. :

t1 x1 y1
t2 ?  ?
t3 x3 y3
t4 ?  ?
t5 x5 y5

You are trying to forecast this time series using a recurrent neural network (say LSTM) and you decide to handle the missing values using masking.

Masking example (from Keras): set x[:, 3, :] = 0. and x[:, 5, :] = 0.   - insert a Masking layer with mask_value=0. before the LSTM layer: model =  Sequential() model.add(Masking(mask_value=0., input_shape=(timesteps, features))) model.add(LSTM(32))

I have heard people say that masking is somewhere inbetween:

A) completely removing missing values/rows, and

B) Imputing/learning missing values

Can anyone explain why masking is any different to just removing the values from the time series? The way I understand it so far is that: if all values in input tensor equal the mask value then that time step will be skipped and the state transferred (if stateful is true). How is this different to excluding the row from the time series?


texts: How to sort tensor along two axis by variance in TensorFlow?

texts:
I have a function that visualizes 2D convolutional kernels. The weights come as 4D-tensors [filter_height, filter_width, in_channels, out_channels], as specified for conv2d. Now I try to sort the kernels by their variance before visualizing them, highest variance first.

My current approach would be something like this:

#get variance along 2D kernel dims, returns tensor [in_channels, out_channels]
mean, variance = tf.nn.moments(kernel, [0,1])

#get sorted indices of output kernels by their variance
indices = tf.nn.top_k(variance, k=(variance.shape[1]), sorted=True).indices

#kernel to [in_channels, out_channels,filter_height, filter_width ]
kernel= tf.transpose(kernel, (2, 3, 0, 1))

#get indices to same dimension as kernel
indices=tf.expand_dims(indices,axis=2)
indices=tf.expand_dims(indices,axis=2)

#sort kernel according to indices
kernel=tf.gather_nd(kernel,indices)

#kernel back to [filter_height, filter_width, in_channels, out_channels]
kernel= tf.transpose(kernel, (2, 3, 0, 1))

But after trying endless approaches, I can't even get tf.gather_nd() to output a tensor with the correct  amount of dimensions.
I think I'm not entirely sure about the functions inner workings though.

Can anyone lead me into the right direction? Any help is highly appreciated!

Edit: to clarify, i would like a tensor  [filter_height, filter_width, in_channels, out_channels] where the first of the filter_heightXfilter_width kernels is the one with the highest variance, the second the one with the next highest and so on up to out_channels


texts: Reconciling a vector input with a scalar operation?

texts:
I've been studying the topic of Meta-Learning. I am particularly interested in Regularized Evolution.  I have been trying to replicate the work presented here:
https://ai.googleblog.com/2020/07/automl-zero-evolving-code-that-learns.html
In this work; the team used Evolution, or Survival of the Fittest to search for and automatically build ML algorithms that could complete a specified task.
It consists of first defining a Task. Which according to this paper here: https://arxiv.org/abs/2004.05439

Here we loosely define a task to be a dataset and loss function T =
{D, L}

The task itself drives the evolutionary algorithm to create an algorithm that can minimize the loss function (The algorithms predicted label compared to the Ground Truth). The evolutionary algorithm has access to a list of 65 mathematical operations (The full list is in the paper) ranging from basic arithmetic to calculus and statistics. There are scalar operations, vector operations and matrix operations.
OP0 no_op
OP1 s2=s3+s0
OP2 s4=s0-s1
OP3 s8=s5*s5
OP4 s7=s5/s2
OP5 s8=abs(s0)
OP6 s4=1/s8
OP7 s5=sin(s4)
OP8 s1=cos(s4)
OP9 s3=tan(s3)
OP10 s0=arcsin(s4)

Using these as building blocks; there is a lot of potential for the creation of novel ML algorithms.
So that is sort of a high level look at what is going on. In my attempts to replicate this in my own code base I am stuck because I do not completely understand how the authors of the paper handled feeding data to the wide range of operations at disposal. For example:
My input data is a 1 x 7 matrix of doubles. If the algorithm built by the evolutionary process consist mainly of vector or matrix operations, then I can easily make my data compatible. However, I'm not sure how I should handle scalar operations. Do I input individual values from the matrix, or do I somehow cast the matrix into a single value (normalization?)?
So far I have the read the paper a few times and looked through their code repository extensively. I'm not exactly sure how the authors of this experiment handled that.


texts:
I'm trying to understand where the output of the LSTM is. Please refer to the following picture:

http://colah.github.io/posts/2015-08-Understanding-LSTMs/

It seems that at each tilmestep, we output h_t and C_t which correspond to the hidden state and the memory cell.

Now suppose I'm trying to model the stock price movements which is binary [0,1], 0 for down, 1 for up which is my y_i.

I feed in x_t which is a feature vector at each tilmestep and I expect to get a 1 dimensional output y_t after the last tilmestep.

Are h_t what I'm looking for? This would imply that h_t matches the output dimension, but for some reason I though it is independent of the output dimension.


texts: Mathematical formulation of Support Vector Machines?

texts:
I'm trying to learn maths behind SVM (hard margin) but due to different forms of mathematical formulations I'm bit confused.

Assume we have two sets of points $\text{(i.e. positives, negatives)}$ one on each side of hyperplane $\pi$.
So the equation of the margin maximizing plane $\pi$ can be written as,
$$\pi:\;W^TX+b = 0$$

If $y\in$ $(1,-1)$ then,

$$\pi^+:\; W^TX + b=+1$$
$$\pi^-:\; W^TX + b=-1$$

Here $\pi^+$ and $\pi^-$ are parallel to plane $\pi$ and they are also parallel to each other. Now the objective would be to find a hyperplane $\pi$ which maximizes the distance between $\pi^+$ and $\pi^-$.

  Here $\pi^+$ and $\pi^-$ are the hyperplanes passing through positive and negative support vectors respectively

According to wikipedia about SVM I've found that distance/margin  between $\pi^+$ and $\pi^-$ can be written as,
$$\hookrightarrow\frac{2}{||w||}$$
Now if I put together everything this is the constraint optimization problem we want to solve,
$$\text{find}\;w_*,b_* = \underbrace{argmax}_{w,b}\frac{2}{||w||} \rightarrow\text{margin}$$
$$\hookrightarrow \text{s.t}\;\;y_i(w^Tx\;+\;b)\;\ge 1\;\;\;\forall\;x_i$$

Before proceeding to my doubts please do confirm if my understanding above is correct? If you find any mistakes please do correct me.

How to derive margin between $\pi^+$ and $\pi^-$ to be $\frac{2}{||w||}?$ I did find a similar question asked here but I couldn't understand the formulations used there? If possible can anyone explain it in the formulation I used above?
How can $y_i(w^Tx+b)\ge1\;\;\forall\;x_i$?


texts: Implementations of "Growing When Required" approach

texts:
Recently I discovered such thing as GWR neural networks. This is a family of neural nets that can expand dynamically: in breadth and/or in depth. As far as I understood, such networks are declared to give the best results in continuous learning tasks, because their main goal is to mitigate catastrophic interference/forgetting.

Despite that fact that this approach was new to me, it appears that it was first introduced in far 2002 - more than 15 years ago. But the most interesting here is that I can't find implementations of any of those algorithms in popular frameworks! My main tool for development is Deeplearning4j and I'm quite sure that continuous learning is not implemented there in any way. I also googled for something like "tensorflow growing when required" or "keras dynamically growing networks", but it doesn't give any results.

My question is: why there's a situation that this mechanism is not available in popular NN-frameworks out of the box, despite its promising results (at least based on fancy plots from science papers I've read)? Or maybe I'm missing something or googling using wrong terms and it's called somehow differently?


texts:
I'm totally new to the topic of real-time bidding in which I know Machine Learning algorithms are used pretty often.

Can somebody explain me the system in a plain language i.e. a language for a non-technical person?

What is the bidding? Who bids on what? Where does Machine Learning get involve?

What is cookie matching mainly about?


texts: Unbalanced data set for multi-class classification algorithms

texts:
how can unbalanced data set for multi-class classification algorithms in python
i am data set a contain three class . for example class1 , class2 and class3.
a value Counts
class 1  is 956
class 2  is 940
class 3  is 420
Thanks in advance for your feedback


texts:
I have a spreadsheet with thousands of records regarding support requests.
Case number, Issue description, etc.

Our goal is to classify these records in many categories in order to assign them the right priority.

Example:

Customer can't use pickup feature.
Customer can't dial 911 or Long Distance numbers.

For item number 1, I have decided to use a category called Best Effort and for item 2, an Urgent category.

Customer can't use pickup feature, BEST_EFFORT
Customer can't dial 911 or Long Distance numbers, URGENT

I'm planning to setup a dictionary of words.

best_effort = ['pickup','record','conference']
urgent = ['system is down','911', 'can't dial emergency','call center is down']

My goal is to use TFIDF and then cosine similarity to find best match and category. Does it makes sense? Any better recommendation to classify this type of information?


texts: Is RL applicable to environments that are totally RANDOM?

texts:
I have a fundamental question on the applicability of reinforcement learning (RL) on a problem we are trying to solve.

We are trying to use RL for inventory management - where the demand is entirely random (it probably has a pattern in real life but for now let us assume that we have been forced to treat as purely random).

As I understand, RL can help learn how to play a game (say chess) or help a robot learn to walk. But all games have rules and so does the ‘cart-pole’ (of OpenAI Gym) – there are rules of ‘physics’ that govern when the cart-pole will tip and fall over.

For our problem there are no rules – the environment changes randomly (demand made for the product).

Is RL really applicable to such situations?

If it does - then what will improve the performance?

Further details: - The only two stimuli available from the ‘environment’ are the currently available level of product 'X' and the current demand 'Y' - And the ‘action’ is binary - do I order a quantity 'Q' to refill or do I not (discrete action space). - We are using DQN and an Adam optimizer.

Our results are poor - I admit I have trained only for about 5,000 or 10,000 - should I let it train on for days because it is a random environment?

thank you Rajesh


texts: Problem with converting string to dummy variables

texts:
I'm new in data science, I have data which want to work on it, I omitted extra columns and convert it to 4 columns ( Product, Date, Market, Demand ) . in this data Product and Market are string, I know for working on this data must convert them. I want to convert the string to dummy variables but this isn't logical because I have 64 fruits in the product column.
I am confused and I don't know what can I do whit this strings.


texts:
I use a neural network with 3 inputs and 1 output with Keras. I'm using MinMaxScaler from sklearn to normalize my inputs in the range [0,1]

my input shape is (XX,3)
my output shape is (XX,1)

I don't have any input while scaling the input and output arrays

    self.scaler = MinMaxScaler(feature_range=(0,1))

    dataX = self.scaler.fit_transform(dataX)
    dataY = self.scaler.fit_transform(dataY)

    self.model.fit(dataX,dataY,nb_epoch=1000)

After the training I have able to make prediction like this :

    data    = numpy.array([[ val1,val2,val3]])
    data = self.scaler.fit_transform(data)
    prediction = self.model.predict(data)

but my output is not scaled correctly, it make sense, as I didn't call inverse transform to apply the inverse scaling

but when I call it, I get a ValueError , I tried to apply the transformation on a single prediction or on an array of prediction but the problem is the same

self.scaler.inverse_transform(predictions) # prediction shape is (18,) :

ValueError: operands could not be broadcast together with shapes (18,) (3,) (18,)

ValueError: non-broadcastable output operand with shape (18,1) doesn't match the broadcast shape (18,3)

I undestand it is a shape issue, but the error message does not help so much

(12,1) was accepted by fit_transform (18,1) seems pretty similar.. so I dont get the error.


texts:
I have a dataset with 5K records focused on binary classification. I am posting it here to seek your suggestions on project methodology

Currently what is my objective is

1) Run statsmodel logistic regression to find risk factors that influence the outcome

2) Then build a predictive model based on best features (may or may not include risk factors). because as you may know not all significant variables are good predictors.

Though I can use scikit-learn logistic regression to build a predictive model but I am planning to go with Xgboost because it provides better performance in my dataset (non-linear data slightly imbalanced)

I do step one because I have to find what are the risk factors that influence the outcome, so I am doing it. (ex: risk factors that influence customer to default in loan repayment) You know where we get p-values and find significant risk factors.

In 2nd step, I build predictive model because I realized through running the built model that not all risk factors are good predictors. So in the end, I include new set of features that help in better prediction along with risk factors

Do you think that I am right in having/approaching this as two objectives problem?

Do you think what I am doing is redundant or am proceeding in the right direction?

Do you think there is no reason to use 2 algorithms separately?

Do you have any suggestions or tips to make it easy to achieve my objective?


texts:
Excuse the potentially dumb question, I've only just started learning about data science.

How do we find out how much data we should collect before using it to start making decisions?

Is there a way of knowing if we should wait to collect more data? Or if the quantity of data can lead to meaningful results?

Background: we have a mobile app and some stats about it, shown below. It's early days and the data set is small. We want to start using this data to make better decisions about improving the product.


texts: Can I apply Feature Selection even though the number of features is smaller than the number of observations?

texts:
I was reading An Introduction to Statistical Learning when I came across the High Dimensional section, they argue that high-dimensional data causes a lot of problems. My question is, it is necesary to have more features than observations to apply any feature selection model?
If so, can you provide me with any article that talks about it?


texts: How to define a custom metric function in R for Keras?

texts:
I just started using keras and would like to use unweighted kappa as a metric when compiling my model. Following the instructions from here, I tried to define my custom metric as follows:

library(DescTools)  # includes function to calculate kappa
library(keras)

metric_kappa <- function(y_true, y_pred) {
   CohenKappa(y_true, y_pred)
}

model %>% compile(
   loss = 'categorical_crossentropy',
   optimizer = optimizer_rmsprop(),
   metrics = metric_kappa
)

However, there must be something wrong with my definition since I get a segmentation fault. Any idea on how to solve this and how to properly set up a custom metric would be highly appreciated.


texts: ReLU for combating the problem of vanishing gradient in RNN?

texts:
For solving the problem of vanishing gradients in feedforward neural networks, ReLU activation function can be used.

When we talk about solving the vanishing gradient problem in RNN, we use a more complex architecture (e.g. LSTM). In both of these, activation function is tanh. Can't we use ReLU instead of tanh in RNNs for solving vanishing gradients too rather than opting for a more complex architecture?