texts: Decision Boundary with random new observations vs observations from test set

texts:
I'm trying to plot decision boundary for Decision Tree classifier. Classifier is trained on training set, and decision boundary (contour) using random new observations and observations from test set is not same.
Here is the code:
def plot_decision_boundary(clf, X_train, y_train, X_test, y_test, contour=True):
    x1_min = np.round(X_train[:, 0], decimals=2).min()
    x1_max = np.round(X_train[:, 0], decimals=2).max()
    x2_min = np.round(X_train[:, 1], decimals=2).min()
    x2_max = np.round(X_train[:, 1], decimals=2).max()

    # Test set
    x1 = X_test[:100, 0]
    x2 = X_test[:100, 1]

    xx, yy = np.meshgrid(x1, x2)
    X_new = np.c_[xx.ravel(), yy.ravel()]
    y_pred = clf.predict(X_new)
    zz = y_pred.reshape(xx.shape)

    print(x1.shape)
    print(x2.shape)
    print(xx.shape)
    print(yy.shape)
    print(X_new.shape)
    print(zz.shape)
    print()

    # Random points
    x1_r = np.linspace(x1_min, x1_max, 100)
    x2_r = np.linspace(x2_min, x2_max, 100)
    xx_r, yy_r = np.meshgrid(x1_r, x2_r)
    X_new_r = np.c_[xx_r.ravel(), yy_r.ravel()]
    y_pred_r = clf.predict(X_new_r)
    zz_r = y_pred_r.reshape(xx_r.shape)

    print(x1_r.shape)
    print(x2_r.shape)
    print(xx_r.shape)
    print(yy_r.shape)
    print(X_new_r.shape)
    print(zz_r.shape)

    custom_cmap = ListedColormap(['#fafab0','#9898ff','#a0faa0'])
    # Plotting training points
    plt.plot(X_train[y_train==0, 0], X_train[y_train==0, 1], 'ro')
    plt.plot(X_train[y_train==1, 0], X_train[y_train==1, 1], 'bs')

    # Plotting decision boundary
    plt.contour(xx, yy, zz, alpha=0.8)

    plt.axis([x1_min, x1_max, x2_min, x2_max])
    plt.xlabel("$x_1$", fontsize=16)
    plt.ylabel("$x_2$", fontsize=16)
    plt.title("Decision Tree")

    plt.show()

Output: (not including contour)
(100,)
(100,)
(100, 100)
(100, 100)
(10000, 2)
(100, 100)

(100,)
(100,)
(100, 100)
(100, 100)
(10000, 2)
(100, 100)

Shapes of data (both random and test sets) used for predicting and plotting is same. Contour for random set is clean and as expected, but for test set, its zigzag. Is there any issue with plotting test set in this code?


texts: How to reduce dimensionality of 3.2B categorical features?

texts:
Background:

This means a dataset of 7,000 samples and 3.2B columns, which I would have to read into distributed Spark memory somehow.

Obviously I want to reduce the number of columns that gets fed into the final model, although I could start with a smaller amount of samples.

Approaches:

I keep coming back to this Multiple Factor Analysis for categorical features http://factominer.free.fr/factomethods/multiple-factor-analysis.html which looks like it would give me groups... but do I have to manually decide which groups to remove?

I've also read about autoencoders NNs being used for dimensionality reduction.

I've also read about including all inputs directly in the first layer of my desired model and then drastically reducing the number of inputs into the second layer as another form of dimensionality reduction.

More Background:

There are 3.2B positions in the human genome. Each position can contain either one of 4 different nucleotides or some larger insertions/ deletions of nucleotides.

I have 2,200 cases (prior to QC filtering) for a relatively common disease and I can and compare them to as many controls as I like so 4,800 controls seems like a reasonable balance. I've read that case-control ratios edging toward 90-10 are considered imbalanced.

What approach should I take?


texts:
I found this very informative figure, on how to split the dataset depending on how much data (or how many observations to be more precise) you have.

My question is, since "less data" is very subjective, is there a statistical test you can perform or even a rule of thumb on which split to follow?

My current problem is a classification problem with 145 observations, 22 features, 2 labels (18-True, 127-False), but I'm interested on the general approach.

Thank you.


texts:
Lets say I have text data for different documents from 2005 - 2015. I want to compare the similarity between $t$ and $t-1$ documents. So I take the document at 2006 and compare it with the document at 2005, take the document at 2007 and compare with the document at 2006 … all the way to 2015, compared with 2014.

I have computed for each year a Word2Vec model independently of other year and obtained high dimensional arrays for each Word. So I have 10 Word2Vec models from 2005 - 2015.

Whats the best way for me to compare the documents similarity from here.

Previously I used TF-IDF where I was able to for each document have a large matrix with words in the rows and documents in the columns. Then I could combine he TermDocumentMatrix at $t$ and $t-1$ and compute the cosine similarity.

However Word2Vec gives much high dimensional arrays and I cannot think how to compare W2V models from $t$ to $t-1$.

Any help would be great!


texts: Stochastic gradient descent in matrix factorization, sensitive to label's scale?


texts: How to adapt output with multiple columns for better presentation?

texts:
Recently I've started using RMarkdown in RStudio, and I don't know how to adapt the output for a useful presentation. The variables in the data set being: Cia, Riesgo, Tipo.Refa, Estado.Poliza, Tipo.Endoso, Asegurado, Moneda, Suma.Asegurada, Prima.Pesos, Premio.Pesos, Puntos, and Pesos.
The only thing I want is find a way to present the data in a more understandable format, unlike the images I attached. (dimensions of the R object are 9 rows, 12 variables)

A bit of help would be much appreciated.

```{r setup, echo = FALSE}
knitr::opts_knit$set(root.dir = '/Users/Igna43/Dropbox/Nacho/Nacho 2016/Análisis R')
```

```{r echo = FALSE}
data1<-read.csv(file = "ListadoEnero2016.csv", header = TRUE, sep = ",")
opneg1<-subset(data1, data1$Pesos. < 0, )
opneg1
```


texts:
I am trying to construct a machine learning model that predicts the difference in price from tomorrow to the day after tomorrow, using yesterday's OHLCV (open, high, low, close, volume).

My models (LSTM and DNNs), however, give me very poor outputs. I think it is because the data needs to be reformatted in some way, shape, or form (i.e. normalized, log-pct change, etc.)

What is the best way to manipulate the input? Should I use a sci-kit 'scaler', or any other method to normalize the data? If so, what range should the scaler be within?

What about the output? If I want the output to be linear (continuous, non-softmax value), should I manipulate the output data as well to be scaled, and then unscale the predicted output to match the original labels? Does it matter?


texts: Can we use embeddings or latent vectors for a recommender system?

texts:
I'm having a hard time understanding why people use any vector they find as a candidate for a recommender system.
In my mind, a recommender system requires a space where distance represents similarity. Of course, before you can construct such a space, first you need to settle on the type of distance you want to use (euclidean, angular, or anything else). Then you need a model (assuming we are talking about ML) to map your input (it could be an image, text, or anything else) to a point in that space. One major aspect of this model is that it's aware of the type of distance we've defined. If there's no notion of the distance in the model, definitely the output of the model is not going to have the attribute of "distance means similarity".
I'm asking this question because I've seen people use any vector they find to construct a recommender system. Here's an example of using a VAE's latent vectors for recommender systems:
https://developer.nvidia.com/blog/building-recommender-systems-faster-using-jupyter-notebooks-from-ngc/
I've also seen people using fastText word embeddings in the same way. I understand that all these embeddings/latent vectors form clusters in their spaces with some interesting patterns. But I don't think this is enough to assume the "distance represents similarity" requirement for a recommender system.
Please let me know if I'm missing anything here.


texts: Finding similarity between two histogram plots

texts:
I have data census(name, sex, age, capital_gain) and I want to plot all possible views (in a histogram) and find the most interesting view based on the distinction to other views.

So, I have to compute the similarity between two histogram plots, for example:

Average Capital Gain vs Sex (which computed by aggregate Capital gain to Sex)

Average age vs Sex (which computed by aggregate average age to sex)

My questions is, what kind of measurement that can be used to check the similarity between those two histogram plots? I consider the features that want to be compared are (x-axis, y-axis, and pattern of the histogram)

I really not sure about distance function that can be used for this case. I know Jaccard distance, but not sure it can be used in this situation or not.

Thank you,


texts: How to aggregate face embeddings of all photos of the same person?

texts:
I am classifying about 3000 thousand people's faces using FaceNet. Each person has about 100 photos.

FaceNet first calculates a face embedding ( a feature vector) for each photo. So each person has 100 face embeddings.

What I want to do is aggregate the face embedding of each person into one. What is the best way of doing this?

I have tried to use mean method. But I am not sure whether this is recommended way.

--
The reason I want this is because using a single SVM as classifier for 3000 labels is very slow. (I took 50+ hours and about 250G memory and it still won't finish training). So I need to divide the training data into subsets, and use multiple SVCs to get first level of results. Then I uses the aggregated face-embedding of each person and closest distance to get second level result.


texts: Improve CNN model for image classification

texts: I'm using transfer learning with VGG16 for image classification. I have 6 classes each one with more than 20k images, I'm trying to improve my accuracy but after many tests I still don't have good results. My accuracy is around 60% or sometimes less than that. I did many tests on the learning rate, batch size, epochs. In fact, I want to classify parts of cars by View ( front, back, left,...) so for example in class front I have different parts of car (door, window, trim ...) where the picture was token on the front side. Maybe this doesn't help the model to learn well because there are images similar for example in left and right classes. I hope that someone could help me to improve it.


texts: What are helpful annotation tools (if any)

texts:
I'm looking for tools that would help me and my team annotate training sets. I work in an environment with large sets of data, some of which are un- or semi-structured. In many cases there are registration that help in finding a grounded truth. In many cases however a curated set is needed, even if it just were for evaluation. A complicating factor is that some of the data can not leave the premise.

We are looking to annotate an object detection task, but I anticipate an image segmentation task, a text classification task and a sentiment detection task in the near future.

What I'm looking for is a system that can help a group make an annotation, preferably in a way that motivates the annotators by showing group progress, relative individual progress and perhaps personal inter annotator agreement.


texts: Neural Network returning training data ratio instead of probability

texts:
I am trying to create a neural network from scratch using numpy.
I have created a network that can classify iris data base to a high degree of accuracy.

I am facing the problem that sometimes instead of predicting a class given input, the network just trains to return the probability distribution of the training sample irrespective of the input.

So, after training the network on 150 samples (50 sample from each class), then on testing on a sample I would get (0.333 0.333 0.333) for each of the three classes, where the expected output was supposed to be, say, (1,0,0).

I had solved that problem by tweaking the Hyperparameters, but I am facing a similar problem with wine data set, for which I cannot do the same.

Has anybody faced this problem before? How did you solve it?


texts: Clustering a dataset and creating a model per each cluster

texts:
I was wondering if it makes sense to cluster a dataset to find closely related data points and train a binary classification model for each of this clusters as they would be minidatasets. I'll ellaborate a bit more.

I have a dataset with information about an infrequent event (e.g. a crime in one part of the city). The dataset looks like this. Note how the features are not item-specific but timestamp specific (as in the 2020-03-12 17:00 timestamp) as they represent the state of the city at that timestamp: weather, hour of the day, visibility, etc...

Timestamp         StreetID Feature1 Feature2, Feature3,... FeatureN
2020-03-12 17:00  12       4         C         5            100
2020-03-12 17:00  145      4         C         5            100
2020-03-08 19:00  145      8         D         6            98
...
2020-03-06 18:15  76       6         C         8            110

I want to train a machine learning model that predicts if this rare event is going to happen or not in a given street or set of streets. I have two options here:

Train a multiclass classification model where you will get a probability for each of the streets. The problem is that there are thousands of streets and some of them have few positive cases
Train one binary classification model for each street. Also impractical for the reasons mentioned above
Group several streets and train a binary classification model for each of these groups.

My question is related to the third option. How can we build these groups?. I assume that it would help to group together streets that share a feature distribution that explains if this unfrequent event is happening or not. Could I make that by clustering the dataset shown above (ignoring the StreetID feature obviously) and checking if inside those clusters there is a high concentration of one or more StreetID's?. Then I can train a binary model with a dataset for the StreetID's of one cluster containing the positive cases/timestamps where the event happened together with the negative cases/timestamps where the event didn't happen like in the example below (assuming we have 15 minutes timestamps and Streets 145 and 76 belong to the same cluster i.e have a high concentration of events inside one cluster). Hypothetically this would make more sense than just grouping streets by some other criteria.

Timestamp         StreetID Feature1 Feature2, Feature3,... FeatureN  EventHappened
2020-03-12 16:45  -        3         F         9            130      no
2020-03-12 17:00  145      4         C         5            100      yes
2020-03-12 17:15  -        9         F         19           30       no
....
2020-03-08 19:00  145      8         D         6            98       yes
...
2020-03-06 18:15  76       6         C         8            110      yes
2020-03-06 18:30  -        7         D         20           80      no


texts: How to get the survival duration prediction for each individual in the data by using the Kaplan-Meier method?


texts: Temporal train test split for recommender systems

texts:
When evaluating a collaborative filtering recommender system, it is practical to split the data temporally. However, by doing so, some users might be present in only either of the train or test set. For example, consider the below example:
user  year
0     2020
0     2020
0     2021
1     2021
1     2021
1     2021
2     2020
2     2021
2     2021

If we decide to split by year such that ratings after 2020 will be in the test set, then:
Train
user  year
0     2020
0     2020
2     2020

Test
user  year
0     2021
1     2021
1     2021
1     2021
2     2021
2     2021

This means that user 1 will not be in the train set at all. When using matrix factorization/latent models, since user 1 is not in the train set, when we multiply the latent factors U and V to get back the predicted rating matrix, user 1 will not be in there at all, and thus we will not be able to predict the ratings for user 1. This applies to items as well, although it is not shown here.
How does one deal with that? Does one simply remove users that are not in the train set from the test set? Wouldn't this lead to a lot of data wastage?


texts: Log loss vs accuracy for deciding between different learning rates?

texts:
While  model tuning using cross validation and grid search I was plotting the graph of different learning rate against log loss and accuracy separately.

Log loss

When I used log loss as score in grid search to identify the best learning rate out of the given range I got the result as follows:

Best: -0.474619 using learning rate: 0.01

-0.674328 (0.000482) with: learning rate: 0.0001
-0.583335 (0.003236) with: learning rate: 0.001
-0.474619 (0.004336) with: learning rate: 0.01
-0.494540 (0.008705) with: learning rate: 0.1

Accuracy

When I used accuracy as score in grid search to identify the best learning rate out of the given range I got the result as follows:

Best: 0.781958 using learning rate: 0.1

0.656220 (0.085705) with: learning rate: 0.0001
0.715279 (0.010021) with: learning rate: 0.001
0.740141 (0.007927) with: learning rate: 0.01
0.781958 (0.003770) with: learning rate: 0.1

In both cases I got different learning rates that I should use to tune my model. When the score is log loss, I got optimum setting for learning rate as 0.01. When score is accuracy, I got optimum setting for learning rate as 0.1.

In such cases, what score should I use for my model?


texts:
I'm working on image classification using CNN, my dataset contains more than 50 classes (50 folders) which represent the types of car parts, and in each folder we have vehicle brands, each vehicle folder contains the images of the part taken in different sides: front, back, left , right, top, bottom. For example: a folder 'door' wil contain different brand of cars, in each car we will have images of the door tooken by different sides.
I have to create a model to recognize the type of the car part and also the side that the picture was taken.
Please, can anyone suggest how can I use all these folders for my training dataset and how to recognize those two things ( type and side).


texts: How to read Feature Based Grammar from a string

texts:
To read Context Free Grammars we can use

nltk.CFG.fromstring("""
S-> NP VP`...
    """)

But it can't read a feature based grammar which is like?

S -> NP[NUM=?n] VP[NUM=?n]

So how to read feature based grammar from a string without actually having to save it in a fcfg file to load it later?


texts: How to plot two columns of single DataFrame on Y axis

texts:
I have two data frames (Action, Comedy). Action contains two columns (year, rating) ratings columns contains average rating with respect to year. The Comedy data frame contains the same two columns with different mean values.
I merged both data frames into a total_year data frame.
The output of total_year:

Now, I would like to plot total_year on a line graph in which the X axis should contain the year column and the Y axis should contain both the action and the comedy columns.
I can plot only 1 column at a time on Y axis using following code:
total_year[-15:].plot(x='year', y='action' ,figsize=(10,5), grid=True)

How can I plot both columns on the Y axis?
I took this photo from google just to let you know guys I want to draw graph in this way:


texts: Distance metric for graph layout of correlation matrix?

texts:
I have a handful of N "things" for which I have a NxN symmetric correlation matrix.  The correlations are a based on a custom scheme for quantifying similarity between complex things, with a value range 0% to 100%, though the values for the N things are all high (95.4%-97.4%). I want to visualize the similarities and differences using force directed graph visualization [1].
I was going to use 1 minus the correlations as a distance metric.  Is this an appropriate for correlations, particularly when values are all high?
Googling hasn't been my friend when searching for the intersection between correlation matrices and graph visualization.  My thought is that 1 minus the correlation plots the N things with distances somewhat proportional to their very small differences.  An alternative might be to use the reciprocal of the correlation as the distance, but since the values are close to one, the Taylor expansion would show such a distance would be similar to 1 minus the correlation.  The two schemes would differ much more for near-zero correlations, where the reciprocal would yield a distance approaching infinity (not practical for graph visualization).  I can tweak the 1 minus distance scheme by raising to various powers, greater or less than 1 depending on whether I want to emphasize the small distances or the great distances.
[1] I would be using Matlab's layout tool, which uses Fruchterman & Reingold's "Graph Drawing by Force-directed Placement." Software — Practice & Experience. Vol. 21 (11), 1991, pp.  1129–1164.


texts: amount of data required for a good ann mdel

texts:
I am new to deep learning and started with ANN. I have a dataset with 15 parameters and 2100 rows. r2_score with Multi linear regression and random forest models is around 85% but when i try an ANN with this dataset r2_score is very less, around 32% . Is it because the number of rows are less? If so atleast what number of rows are required to go for ANN. I have added the code too for reference

import pandas as pd

dataset = pd.read_csv('chiller-2_runningdata_withcommon_parameters.csv')

dataset = dataset.drop(['DateTime','delta','KWH','RunStatus','OP Hours'], axis=1)

X = dataset.iloc[:,:-1 ].values
y = dataset.iloc[:,15:16].values
# Splitting the dataset into the Training set and Test set
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0)

# Feature Scaling
from sklearn.preprocessing import StandardScaler

sc_X = StandardScaler()

sc_Y = StandardScaler()

X_train = sc_X.fit_transform(X_train)

X_test = sc_X.transform(X_test)

y_train = sc_Y.fit_transform(y_train)

# Importing the Keras libraries and packages
import keras
from keras.models import Sequential
from keras.layers import Dense

# Initialising the ANN
classifier = Sequential()

# Adding the input layer and the first hidden layer
classifier.add(Dense(units = 8, kernel_initializer = 'uniform', activation = 'relu', input_dim = 15))

# Adding the second hidden layer
classifier.add(Dense(units = 8, kernel_initializer = 'uniform', activation = 'relu'))

# Adding the output layer
classifier.add(Dense(units = 1, kernel_initializer = 'uniform'))

# Compiling the ANN
classifier.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics = ['accuracy'])

# Fitting the ANN to the Training set
classifier.fit(X_train, y_train, batch_size = 32, epochs = 500)

# Predicting the Test set results
y_pred = classifier.predict(X_test)
y_pred = sc_Y.inverse_transform(y_pred)
y_pred_train = classifier.predict(X_train)
y_pred_train = sc_Y.inverse_transform(y_pred_train)
y_train = sc_Y.inverse_transform(y_train)

#calculate r2_score
from sklearn.metrics import r2_score
score_train = r2_score(y_pred_train,y_train)
score_test = r2_score(y_pred,y_test)


texts: How to implement one vs rest classifier in a multiclass classification problem?

texts:
I have a dataset which contains 750 data points with 8 classes in the target variable. I tried implementing simple machine learning models and also did hyperparameter tuning but they results were not much impressive. The best log loss that i could get was 1.52 with a misclassification rate of 53%. What are the other methods that i could apply to improve the models performance ? Also, I want to implement onevsrest classifier with the hope that, it would improve the results, but implementation of the same is not clear to me although I looked in the internet for some clear codes.

best_alpha = np.argmin(cv_log_error_array)
clf =OneVsRestClassifier(RandomForestClassifier(n_estimators=alpha[int(best_alpha/2)], criterion='entropy', max_depth=max_depth[int(best_alpha%2)], random_state=42, n_jobs=-1))
clf.fit(X_traindata, y_train)
sig_clf = CalibratedClassifierCV(clf, method="sigmoid")
sig_clf.fit(X_traindata, y_train)


texts: Boosting with highly correlated features

texts:
I have a conceptual question. My understanding is, that Random Forest can be applied even when features are (highly) correlated. This is because with bagging, the influence of few highly correlated features is moderated, since each feature only occurs in some of the trees which are finally used to build the overall model.

My question: With boosting, usually even smaller trees (basically "stunps") are used. Is it a problem to have many (highly) correlated features in a bagging approach?


texts: Validation Accuracy not going beyond 60% for image classification with 5 species of snake

texts:
My dataset has about 17000 images belonging to 5 classes. I am using 16000+ images for training(about 3k/class) & 500 for validation(100/class). Training accuracy is very good but validation accuracy doesn't seem to go beyong 60%. I have about 140k parameters in my model with 100 epochs. Could there be any overfitting happening? Is the dataset too small to train any sort of neural network? I have seen people use transfer learning on smaller datasets so I gave it a try. I am totally new to this being my first project. Any sort of help would be appreciated. I am sorry if I sound completely noob here :(
This is how I preprocessed the images for training
from keras.preprocessing.image import ImageDataGenerator

# this is the augmentation configuration we will use for training
train_datagen = ImageDataGenerator(
     rescale=1. / 255,
     data_format="channels_last")

train_generator = train_datagen.flow_from_directory(
    train_data_dir,
    target_size=(img_width, img_height),
    batch_size=batch_size,
    class_mode='categorical',
    shuffle=True)

validation_generator = train_datagen.flow_from_directory(
    validation_data_dir,
    target_size=(img_width, img_height),
    batch_size=batch_size,
    class_mode='categorical',
    shuffle = True)

Here is the output to the above code
Found 16389 images belonging to 5 classes.
Found 500 images belonging to 5 classes.


texts: Keras LSTM Input Shape - Batch Size and Time Step

texts:
So I have 82 different sets of data, each with varying length where each point has one feature and a label (0 or 1).

I'm trying to use Keras LSTM to be able to predict the class of a point depending on the previous values before it.

Currently I've padded each set to be of the same length and plan on using a masking layer.

What I'm confused with after reading many examples is how I should reshape my data and supply the correct input shape to the model.

If each set is of length 55, and I want to consider the previous 5 (just throwing a number out there) points to predict the class at each step, how would I do that?

Should I manually use a moving window to break each set into arrays of length 5, then have input_shape=(5, 11, 1)
since 55/5 = 11??

Is this the best approach? How would this work for the remaining 80+ sets?

Just to clarify:
say one of the 82 sets looks like this [1.1, 4.8, ... 2.1] (length of 55) I want the model to predict the class of each point [0, 1, ... 1] (x55) except for the padded points (-1).


texts: Possible?: Split pdf file on pages where object detection algorithm finds custom object

texts:
I have scanned documents in large pdf files consisting of many individual documents. Each document begins with a exhibit number sticker, much like this one: example. The files are scanned in greyscale.

I am looking to detect this numbered sticker object in the files. On top of that, my current understanding is that I would need to crop this sticker if I want to run another NN on it to recognize the handwritten digits on the sticker.
I wish to split the files on the pages where an object is detected and rename those individual files based on the handwritten digit on the sticker.

So far I know I must split the pdf pages into individual images with pdf2image like in this example, and I should label the training data beforehand with something like labelimg. After reading more into it, it looks like tensorflow and keras will suit my project fine.

My questions are as follows:

Assuming the image detection model generates a valid bounding box, how can I crop the bounding box?
What NN would best suit each need (detecting single object + handwriting OCR)
What output file should I expect from the model? Should I expect any obstacles in using the output file as a basis to process the corresponding pdf files with something like PyPDF2?

The short of it is I'd want to do what's described in this article and edit my pdf file based on the results.


texts:
It's not entirely clear what pre-trained networks are out there in deep learning. Of particular interest to me are the deep convolutional networks.

I'm only aware of the VGG series like VGG16/VGG19. I searched the internet for Microsoft's famous ResNet that won ImageNet 2015, but couldn't find it. I suppose the corporate giants don't really share their models, eh?

Any pointers to maybe a comprehensive list that I have simply overlooked or there isn't one?


texts: Fitting an arimax model on out of sample dataset

texts:
I have built an arimax model where we have sales data across time as the response variable and price is one of the external variables. I used the below code to build a simple arimax model. I had data points from 1 to 24, I have kept only 1 to 20 data points in the training dataset

library(stats)
fit=arima(window(tssales, end=20), order = c(0,1,1), xreg = window(tsprice, end=20))
summary(fit)
fcast=forecast(fit, h=5, xreg = window(tsprice, end=20))
plot(fcast)

Now when I try to fit the model from the training dataset in out of sample dataset (last 4 data points) I use the below code

library(stats)
out_of_sample=arima(window(tssales, start=21), xreg = window(tsprice, start=21), model=fit)

I am getting the following error

  Error in arima(window(tssales, start = 21), xreg = window(tsprice, start = 21),  :
    unused argument (model = fit)


texts: How to determine number of leaves in decision tree analysis?

texts:
Would be grateful if some expert on the forum can help me understand how to decide optimum number of leaves in a decision tree analysis.

I am using SAS and if I supply leaves=6 in my model then miss-classification rates for validation & training data sets are 18.6% & 18.8% respectively. And SAS lists 5 variables which are significant.

And if I don't supply leaves count in the code and let SAS decide it, then SAS after pruning takes 10 as leaves count and miss-classification rates for validation & training data sets are 17.5% & 16.9% respectively. And SAS lists 6 variables which are significant.

Now that the miss-classification rates have reduced & trees after pruning have increased from 4 to 10, is it a good thing or it indicates overfitting?

Looking forward to opinions of experts in this group.   Thanks


texts:
I am trying to make a model which would take an image of shoes as an input and output a meaningful N-dimensional embedding of the shoes, so that they could be searchable/comparable/clustered and used in a recommender system.
My first guess was to employ a siamese CNN (Densesnet + 1 extra fully connected layer for the 32-dimensional embedding generation) with online hard mining triplet loss. So the idea was to train the network on making prediction if shoes on images belong to the same shoe model based on the euclidean distance between the outputs.
However, it failed to generalize well and has shown poor result on the test data.
My current dataset is ~4k images of ~500 different sneaker models.
What are my options in this situation?


texts: How to normalize data without knowing the min and max values?

texts:
I have a Lending club dataset from Kaggle;
it contains many different columns: there are for example dummy variables, years, amount of the loan...ect
I want to normalize the data in the training and test set but I have to use the Min and Max of the train set to prevent data leakage from the test set.
My question is: if there is, in the test set or even when I try to predict new data point, a value that is greater that the Max value or lower than the Min value and I normalize it using the same values from the train set, is it correct? can I the model process this value normally?

this is the code that I use to normalize

    from  sklearn.preprocessing import MinMaxScaler

    scaler = MinMaxScaler()
    X_train = scaler.fit_transform(X_train)


texts: How to write a LSTM model with 3 dimensional X_train and Y_trains?

texts:
I have X_train and Y_train with [2160,24,3] dimensions. But when I try a simple LSTM like this:

model = Sequential()
model.add(LSTM(24 , return_sequences = True))
model.add(Dropout(dp))

It gives me this error message:

  ValueError: Error when checking target: expected dropout_192 to have
  shape (24, 24) but got array with shape (24, 3)

And when I try to change my code like this:

model = Sequential()
model.add(LSTM(24 , return_sequences = False))
model.add(Dropout(dp))

model.add(RepeatVector(24))

I get:

  ValueError: Error when checking target: expected repeat_vector_55 to
  have shape (24, 24) but got array with shape (24, 3)

Also:

model = Sequential()
model.add(LSTM(24 , return_sequences = False))
model.add(Dropout(dp))
model.add(RepeatVector(24))
model.add(TimeDistributed(Dense(1)))

Or:

model = Sequential()
model.add(LSTM(24 , return_sequences = False))
model.add(Dropout(dp))
model.add(RepeatVector(24))
model.add(LSTM(24 , return_sequences = True))
model.add(Dropout(dp))
model.add(TimeDistributed(Dense(1)))

Gives me:

  ValueError: Error when checking target: expected time_distributed_56
  to have shape (24, 1) but got array with shape (24, 3)

I am confused and don't know how to fix it? What I want is train a LSTM model with X_train and Y_train of size [2160,24,3]. How can I do that?


texts: Scientific explanation of Skip Connections

texts:
I am searching for scientific work on skip-connections.

Everybody is talking about improving gradient flow through the Network, which feels like it makes sense. But I would love to understand and read what that means when it comes down to the math.

Other than the resnet paper I haven't found anything in depth about those magical skip-connections.

I am looking for any hints, papers or equations.


texts: I am going to train a network to label a given image whether it belongs to category A or category B (a binary classifier). I use flow_from_directory to generate train, validation and test data. I keep all the images belong to category A in one folder. However, category B has sub folders. Lets say we have a folder which contains ship images (category A) and another folder which contains all the other images (cats, dogs, buildings, etc.) which are non ships (category B). In my application I keep all the non-ship images in different folders, according to their content, under non_ship folder. At the end I want a binary result (ship or non-ship). In this case I am actually not sure about how keras handles this. Does not it consider all the sub folders and just consider all the non-ship images are in one folder or consider that they belong to different subclasses?


texts: Parameters optimization algorithms in Weka

texts:
In Weka, I used the Grid and Random search parameters tuning algorithms but unfortunately, their performance (in terms of better prediction accuracy) is observed worst when we use the ML algorithms (Support Vector Regression, Linear Regression etc) without any optimization algorithms.
I wonder how it is possible? I mean one algorithm (Grid or Random search) should perform better or worst when compared with each other but they have worst performance compared to without any parameter optimization algorithms. I even tried the hybrid of both in Weka with MultiSearch option, but even the hybrid of these two have worst performance.
Kindly if someone could provide comments based on their experience in this regard.


texts: Filling missing data with other than mean values

texts:
What are all the options available for filling in missing data?

One obvious choice is the mean, but if the percentage of missing data is large, it will decrease the accuracy.

So how do we deal with missing values if they are are lot of them?


texts: What is the state of the art for DNNs to infer that they need more neurons than in the initial setup? Can they be acquired dynamically from the underlying "resource substrate"?


texts: Best parameters to try while hyperparameter tuning in Decision Trees

texts:
I want to post prune my decision tree as it is overfitting, I can do this using cost complexity pruning by adjusting ccp_alphas parameters however this does not seem very intuitive to me.
From my understanding there are some hyperparameters such as min_samples_split, max_depth, min_impurity_split, min_impurity_decrease that will prune my tree to reduce overfitting.
Since I am working with a larger dataset it takes a long time to train therefore don't want to just do trial-error.
What are some possible combinations of above mentioned hyperparameters that will prune my tree? + reasoning behind choosing particular combination will be helpful.
Thanks in advance!


texts:
I just started using machine learning and was wondering if anybody have cool ideas for a Startup project, I've seen this website Treato  and was amazed by it.
Thanks.


texts: Grouping probabilities of full model vs fitting model on subset of variables

texts:
Suppose I want to predict some event probability with a set of features. Some of the features may be gender or some other categorical variable. Assume the probabilities are well calibrated.

Now, say, I need to report the average predicted probability for males and females (and maybe some other subset of features).

Is there a difference between

averaging the probabilities of the full model or
simply fitting a model on only the subset of variables?

And if so why? The full model would outperform the simpler model by a large margin. An example would also be appreciated.


texts: How do I find the minimum value of $x^2+y^2$ with a genetic algorithm?

texts:
I want to find $(x,y)$ which minimizes $x^2+y^2$ with GA to apply it for another function.

Does anyone know any example of GA with deap (Python) like that?


texts: Need a little help Understanding how to build model's in Keras

texts:
I am trying to make a CNN in Keras, and to test the validity of my model i am trying to get it to train on MNIST dataset, so i am sure that everything is working fine, but unfortunately model is barely training and i suspect that nothing updating.
My model is :

model=Sequential()

#conv1_1
model.add(Conv2D(128,kernel_size=3, strides=1,
                 padding='SAME', use_bias=False,
                 activation='relu',name='conv1_1',input_shape=(28,28,1)))
#conv1_2
model.add(Conv2D(128, kernel_size=3, strides=1,
                 padding='SAME', use_bias=False,
                 activation='relu',name='conv1_2'))
model.add(MaxPooling2D(pool_size=2,strides=2))

#conv2_1
model.add(Conv2D(64, kernel_size=3, strides=1,
                 padding='SAME', use_bias=False,
                 activation='relu',name="conv2_1"))

#conv2_2
model.add(Conv2D(64, kernel_size=3, strides=1,
                 padding='SAME', use_bias=False,
                 activation='relu',name='conv2_2'))
model.add(MaxPooling2D(pool_size=2,strides=2))

model.add(Flatten())
model.add(Dense(1024, activation='relu',name='Dense1'))
model.add(Dropout(0.2))
model.add(Dense(512, activation='relu',name='Dense2'))
model.add(Dense(10, activation='softmax',name='output'))

Compiled with:

model.compile(loss='categorical_crossentropy', optimizer='adadelta', metrics=['accuracy'])

model.fit(X_train,y_train,batch_size=10,validation_split=0.2,epochs=10)

My X_train and y_train look like:

plt.imshow(X_train[0].reshape(28,28))
plt.show()

y_train[0]
array([0., 0., 0., 0., 0., 1., 0., 0., 0., 0.])

Here are the Results of first 3 epochs:

Epoch 1/10
48000/48000 [==============================] - 45s 927us/step - loss: 14.2813 - acc: 0.1140 - val_loss: 14.4096 - val_acc: 0.1060
Epoch 2/10
48000/48000 [==============================] - 44s 915us/step - loss: 14.2813 - acc: 0.1140 - val_loss: 14.4096 - val_acc: 0.1060
Epoch 3/10
48000/48000 [==============================] - 44s 924us/step - loss: 14.2813 - acc: 0.1140 - val_loss: 14.4096 - val_acc: 0.1060
Epoch 4/10
48000/48000 [==============================] - 45s 930us/step - loss: 14.2813 - acc: 0.1140 - val_loss: 14.4096 - val_acc: 0.1060

This is my first Keras Model, and i think i am missing something important here.


texts: How to model non-linear demand function?

texts:
I am trying to build a dynamic pricing algorithm on intermittent data (a lot of zeros between non-zero values). I have on average 100 non-zero data points for each product. However, it seems to be quite difficult to predict the right price due to various reasons (non-regular demand, intermittent data, noises etc) so for the first step, I am trying to forecast the demand of each product using MAPA from  Nikolaos Kourentzes. I hope to use this demand forecast and price-demand relation to predict the optimal price.

However, in practice these price-demand relationships tend to be non-linear so I was wondering if anyone here could guide me to papers/resources that I can follow to train non-linear model(s) on the price-demand relationship with intermittent demand data. Ideally, the trained model should be optimizable for price.

[Update 23/03/2019 : Added images/tables for elaborating Dynamic pricing]

The input is a intermittent sales data that looks something like the example shown below for most products.

(Intermittent demand image courtesy to)

Using Dynamic pricing, I would like to model the price-demand relation like the one shown below. Important : Linear demand are not realistic and in my case, the demand for my case non-linear.

(Linear demand image  from)

In the end, I hope to create a model that can tell me what the optimal price is given a certain demand.

(Optimal demand image from


texts: Feature Selection algorithm/library for CRF

texts:
I am using the Conditional Random Fields CRF suite scikit-learn wrapper algorithm. I have read on the literature various approaches for feature selection, but I cannot find any on that package or, generally, available ones for CRF. Would you know any libraries (Python preferred) or easy to implement algorithms for this purpose?

Update

I tried using the scikit-learn feature selector's library but does not work for 2 reasons: 1) the CRF takes as an input list of lists of dicts whereas the other take tabular data 2) the CRF does not have a .coef or a feature_importances_ attribute in order to perform feature selection.


texts: Pandas get_dummies() rows dropping after joining back with X

texts:
I'm having an issue that I can't explain and am hoping I am missing something simple.
I have a large dataset of shape(45Million+, 51) and am loading it in for some analyses (classifiers, deep learning, basically just trying a few different things as some research for work).
I take a few steps when I load it in:

dropna() to get rid of all rows with an na (only about 6K out of the 45M)
Use pandas get_dummies() to change a categorical variable with about a dozen classes into dummy variables (have also used sklearn's onehotencoder for this and had the same issue outlined below)

When I would run a RandomForest on a subset of the data (about 4 million rows, made using train_test_split) I would get the following error:
<ValueError: Input contains NaN, infinity or a value too large for dtype('float64').>

Since I dropped NAs at the start, this confused me, so I went back and checked the length of each output.
When I drop NAs, I get a length of 45356082
When I split off the categorical variable and one hot encode it, it has a length (and every variable within it does) of 45356082. We'll call this Dummy.
Here's where it gets weird-- when I join Dummy back to my original X as Xnew, Xnew has a length of the same as above, but the dummy variable columns now have length 45351726.
The join process is dropping like 4500 rows from the dummy columns.
Any idea why this would happen?
Here's the code I'm using:
choice_data_sub = pd.read_csv("predData.csv")
# Drop NAs
choice_data_sub = choice_data_sub.dropna()

X = choice_data_sub[["Columns1", "Column2", "Column3", "Categorical"]]
y = choice_data_sub[["NextPurchase"]]

choice_data_sub = choice_data_sub.reset_index()

gametype_df = pd.get_dummies(choice_data_sub.Categorical, prefix="Game")

# merge with X
gametype_df = gametype_df.reset_index()
# X = X.reset_index() <-- This breaks it in a different way, was tried as a fix
X = X.join(gametype_df)

Also! Just discovered that the following code works, but I'd still like to know why this didn't.
X = X.reset_index()
X = X.merge(gametype_df, left_index=True, right_index=True)


texts:
Please pardon me as the title might not be very accurate

I am trying to create a model that learns the word representation and then is able to predict word representation in another piece of text. An example will make it more clear. Please see below for the example:

Model Training Text:

I lived in *Munich last summer. *Germany has a relaxing, slow summer lifestyle. One night, I got food poisoning and couldn't find !Tylenol to make the pain go away, they insisted I take !aspirin instead.

Model Predicts:

['Munich','Germany','Tylenol','aspirin']

Evaluation Text:

When I lived in Paris last year, France was experiencing a recession. The nightlife was too fun, I developed an addiction to Adderall and Ritalin.

Output:

['Paris','France','Adderall','Ritalin']

The question is that what sort of NLP technique will be helpful in such a case. I don't even know what are these kind of problems called. Can you please tell what are these problems called?

One approach I can think of is to train the RNN with Embedding Layer to predict the position of * and ! since * will be prefixed to the name of the country and ! will be prefixed to the name of the drug but my challenge is that how can I structure my data for such training. Is it a feasible approach?

Is there any resource/material I can refer to and draw the inspiration from?

I would very much appreciate any help or suggestions. Thanks a lot in advance.


texts: Can somebody explain me this method? CNN Keras - starter

texts:
def ReadImages(Path):
    LabelList = list()
    ImageCV = list()
    classes = ["nonPdr", "pdr"]

    FolderList = [f for f in os.listdir(Path) if not f.startswith('.')]

    for File in FolderList:
        for index, Image in enumerate(os.listdir(os.path.join(Path, File))):
            ImageCV.append(cv2.resize(cv2.imread(os.path.join(Path, File) + os.path.sep + Image), (224,224)))
            LabelList.append(classes.index(os.path.splitext(File)[0]))
            print(FolderList)
    return ImageCV, LabelList

I got this method, and I'm not understanding properly what it do line by line, can someone help me?


texts: Target Encoding: missing value imputation before or after encoding

texts:
I want to perform a target encoding for my categorical features although I am not sure when to perform the data imputation if any of them has missing values.
Let's say I have a few continuous features, Cnt1-Cnt5 (without NA's) and two categorical features, Cat1 and Cat2, with Cat2 having missing values. Let's also assume that I want to use Random Forest as an imputation method. Which approach would be the correct one?

Impute Cat2 treating Cat1 and Cnt1-Cnt5 as predictors in RF and then perform target encoding on categorical variables.
Target encode Cat2 for non missing and Cat1, build RF and impute missings for Cat2 (which is now numeric, not categorical).
Any other approach?

We can generalize this question and ask whether we should impute missings for any kind of variable (including continuous) before or after target encoding?

I see at least one benefit of imputation after target encoding - if there are unseen levels of categorical variable present in the test data (which will result in NA's in the test set after performing target encoding), those would be easily imputed by RF built on training data, without any potential error due to new levels.


texts: Instead of one-hot encoding a categorical variable, could I profile the data and use the percentile value from it's cumulative density distribution?

texts:
I have a categorical variable which has thousands of values, for a dataset which has millions of records. The data is being used to create a binary classification model. I am in the early steps of feature selection, but I am trying out Random Forest, Boosted Trees, and Logistic Regression to see what works.

If I find the frequency of each category and sort that, I see that about 50 values make up the top 80%. Is it valid to condense this feature as a binary on whether the value is in that set of values or not. By 'valid', I mean is it likely that this sort of transformation retain any useful information for a model? I have a concern that sorting these categorical values which do not have any order to them creates some incorrect assumptions.

The frequecy distribution looks a little like this:

A;10%
D;5%
E;1.2%
B;1.1%
...
Z;0.004%
W;0.0037%
...

Going one step further, is it valid to profile each class in my dataset and do the same? Say Categories A-F comprise the top 80% of class 0 and Categories D-H are the top 80% of class 1. I would convert:

data_id;cat_var
1;B
2;F
3;H
4;Z

to

data_id;cat_var_top80class0;cat_var_top80class1
1;1;0
2;1;1
3;0;1
4;0;0

Adding picture to hopefully clear up this idea. In yellow are the pre-calculated distributions of cat_var (***_id in the picture) for classes 0 and 1 based on the training set. On the right shows how the transformation would be applied:


texts: Machine learning on data with only time stamps

texts:
I'm working on trying to predict physical traffic volume over a network of intersections.

The data I have is sorted per intersection and consists of the time (in sec after epoch) they passed and the direction someone came from. A mock sample here (I have months worth of data for 60 intersections):

   Intersection 1:{
     (1456140051, 1),
     (1456140066, 2),
     (1456140070, 2),
     (1456140073, 3),
     (1456140078, 1),
     (1456140091, 1)}
   Intersection 2:{
     (1456140051, 1),
     (1456140058, 4),
     (1456140060, 3),
     (1456140063, 2),
     (1456140067, 2),
     (1456140071, 4)}

(Note: the data is not for the complete network, there are many smaller intersections for which no data is gathered)

So far I've done some fourier analysis which gives me strong weekly and daily periodicity (obviously) and thus calculated daily/weekly averages/noise etc. for each intersection, and I've looked for some correlations between some neighbouring intersections. The next step for me is to use the data from all the intersections to estimate the traffic volume at one intersection, and then do the same for the traffic volume 10 minutes in the future.

I have found various machine learning methods that should work well with time series with constant time intervals and differing values (e.g. 1 data point of varying value every minute), and I can aggregate my data so I get a value of traffic volume each minute or so (typical traffic is between 5 and 100 /minute /intersection). However, I feel I would lose a lot of information that way.

So what I'm looking for it a method that can work with the data I have (or some derivative of that without much information loss) as input. Can anyone recommend some articles or point me in the right direction?


texts:
I am comparing 2 neural network models. I have used the model to make predictions on unseen data.

One model returns an error of 20.9% for y1, 36.6% for y2,
4.53% for y3 on unseen data, and a CV(RMSE) of 19.3.
The other returns an error of 15.5% for y1, 33.8% for y2
and 4.83% for y3 on unseen data, and a CV(RMSE) of 31.5.

I'm struggling to interpret the results. A lower CV(RMSE) is better, yet why do I get a much higher error on unseen data?


texts: User based recommendation factoring in user data

texts:
The question is:
what algorithms (and libraries) should i use if i want to build a recommender system with the following data in mind representation:

USER_ID  ZIP  Movie1  Movie2  Movie3
1  2483  5  0  3
2  2483  4  1  5
3  2345  3  1  5

Basically i want to factor in user data into a recommendation of a movie.

(zips can be transformed to long/lat but thats another question and out of scope now)

i was searching the internet for hours with no success. So i will be grateful if someone can point me in the right direction.


texts: Multi label text classification from thousands of labels

texts:
I don't have machine learning experties, but I'm working on a project that has text classification requirements in it. The easiest approach I was able to understand was using fasttext.. It worked, but the accuracy is terrible. The problem is that we are trying to classify academic publications with multiple labels out of thousands of labels. I've followed all the best practices to clean the training data and followed the fasttext tutorials, but still the prediction quality is 15% max.

So my question is, is fasttext the right solution to this problem? What other solutions can be used if otherwise? Can you guide me to any examples or tutorials similar problems (multi-label out of VERY LARGE set of LABELS)?


texts: Spacy tokenizer that strips HTML/XML keeping the positions

texts:
I'm new to Spacy and infact new to Data Science. I would like to process some XML files for NER and then mark that in the original XML. I would like to know how to tokenize the XML for the NER. I think I should be filtering out the XML tags/code and then feeding the remaining text while retaining positions so that I can get the positions of the NER results in relation to the original XML. This would help me identifying and tagging the exact locations in the XML.
Any ideas and guidance on this would be most appreciated. Thanks


texts: Backpropogation through Maxpool and relu

texts:
Why is backpropagation through maxpool and relu needed?

Purpose of backpropagation is to update weights while on the other hand maxpool and relu only perform a simple operation on the input. They don't really have any weights or any filters. So why is there a need to backpropagate through them?


texts: Item based and user based recommendation difference in Mahout

texts:
I would like to know how exactly mahout user based and item based recommendation differ from each other.

It defines that

User-based: Recommend items by finding similar users. This is often harder to scale because of the dynamic nature of users.

Item-based: Calculate similarity between items and make recommendations. Items usually don't change much, so this often can be computed off line.

But though there are two kind of recommendation available, what I understand is that both these will take some data model ( say 1,2 or 1,2,.5 as item1,item2,value or user1,user2,value where value is not mandatory) and will perform all calculation as the similarity measure and recommender build-in function we chose and we can run both user/item based recommendation on the same data ( is this a correct assumption ?? ).

So I would like to know how exactly and in which all aspects these two type of algorithm differ.


texts: Algorithm to calculate nerual network training time?

texts:
Before starting a new machine learning side project, it would be very useful to estimate how long it will take to run 1, 10, 100, 1k epochs. A crude estimate is more than sufficient (i.e. 1 epoch would take 1 second, 10 seconds, 1 minute, 1 hour, etc..).

Given the variables below, can you recommend any heuristics that could provide an estimate?

Problem type (e.g. Image Segmentation)
Model type (e.g. PyTorch Unet)
Dataset (e.g. 10k images, 512x512)
Compute (e.g AWS p2.xlarge)
Library (e.g. PyTorch)

Is an empirical method (e.g train on smaller subsets of the data and scale accordingly) a better approach to solving this problem?


texts: I've looked at hmmlearn but I'm not sure if it's the best one.


texts:
In classification tasks, we can interpret the output vector as how "confident" the model is that the input has a certain label. For example,
y = [0.01  0.20  0.99  0.10]

would mean the model is 99% certain the input has label with index 2 and 1% certain it has label with index 0.
My question is: Is there an equivalent for regression tasks?
If my output looks like
y = 0.33

is there a way to measure how "certain" the model is that the output really is 0.33?


texts:
I working on a binary classification task. The dataset is quite small ~1800 rows and ~60 columns. There are no duplicates in the rows. I am comparing different classifiers amongst the canonical ones: random forest, logistic regression, boosted tree and SVC. I am training the hyperparameters by a CV on 90% (train) with 10% held out to measure the generalization error (test). The dataset is slightly unbalances (1 to 3 ratio of classes) hence I used a stratified fold for all splits. I also use roc-auc as a metric for my CV.

I get the following results for roc-auc score and accuracy:

 DummyClassifier
Train
ROC-AUC score: 0.50000
Accuracy: 0.69705
Test
ROC-AUC score: 0.50000
Accuracy: 0.69545

 LogisticRegression
Train
ROC-AUC score: 0.88459
Accuracy: 0.78666
Test
ROC-AUC score: 0.72559
Accuracy: 0.69545

 RandomForestClassifier
Train
ROC-AUC score: 1.00000
Accuracy: 0.99695
Test
ROC-AUC score: 0.81748
Accuracy: 0.80455

 XGBClassifier
Train
ROC-AUC score: 1.00000
Accuracy: 0.99949
Test
ROC-AUC score: 0.80617
Accuracy: 0.79545

 SVC
Train
ROC-AUC score: 0.89900
Accuracy: 0.83248
Test
ROC-AUC score: 0.73515
Accuracy: 0.73182

There is always a significant gap between train and test scores. I am clearly overfitting. I guess it is a consequence of the low number of rows but I am not sure about what to do about that? Force the CV grid search for hyperparameters to a range with strong regularization?


texts: Are there trained neural networks, that can distinguish a book's author point from what he stands against?

texts:
Let's take some specific book to narrow this example. Atlas shrugged by Ayn Rand. When i'll be saying something like "Ayn Rand's ideas", i'll mean only those, which are clearly stated in the book.

Humans who read this book, can clearly distinguish sentences, that describe Ayn Rand's point of view from sentences, that oppose it.

Are there trained neural networks, that could distinguish sentences with Ayn Rand's ideas vs sentences with opposing ideas?

If not, are there any reasons, why it's impossible to train such network, if each sentence in such book would be marked as pro-, contra- or neutral to Ayn Rand's ideas? Successfully trained network should be able to distinguish author's point of view in books, where humans can clearly do it.


texts: Model parameters & Hyper parameters of Neural Network & their tuning in training & validation stage

texts: I am new to Machine learning in that Artificial Neural Network. I am using nnet package in R for training of Neural Network. What is the difference between model parameters and hyperparameters? I have heard that the hyper parameters are set before we set the model parameters. What are the hyper parameters and model parameters of a Artificial Neural Network and when they are actually tuned? i.e. Are hyper parameters tuned in Training stage of Neural Network and model parameters in validation stage?


texts: Single real number evaluation metric for VAE for a regression problem

texts:
I've set up a VAE for a regression problem.
vae.compile(
   optimizer=Adam(lr=0.0005),
   loss=kl_loss_func(z_mean, z_log_var),
   metrics=['mae', 'accuracy'])

history = vae.fit(
   X_train, X_train,
   epochs=50,
   validation_data=(X_test, X_test))

Using the information in history I can plot accuracy and mae against epochs. However, in order to compare multiple regressors, I'm interested in a single real number evaluation metric.
The evaluate method returns 0.0 for accuracy and mae:
print(f"Evaluation:\t{vae.evaluate(X_val, return_dict=True)}")

{'loss': 0.0012786508304998279, 'mae': 0.0, 'accuracy': 0.0}

accuracy: 0.0 is expected as my application is a regression problem and not a classification problem. However, I'm not sure about mae: 0.0. Should not this metric be used as a single-number evaluation metric?


texts: Selecting the right algorithm for match probability prediction

texts:
Looking for assistance kick-starting a new machine learning scenario. In this case I need to pair one entity (ex. person) with a group of entities (ex. other people) given a history of matching patterns over time.

For example: Assume we have people who like other people with a combination of traits such as height, weight, education level, etc... This notation of 'like' is bi-directional when calculating probabilities. When a new person arrives I'd like to isolate a group of people (ranked by probability) to recommend them too.

In this case the input would be a single person with all their traits and the output would be a list of people with ranked probabilities of matching. There seems to be two main sets of data

Known people
Traits known people have liked or not with the traits of people they have liked or not

I am relatively new so not sure which types of training model to use. For example, is this something that can be accomplished most reasonable with a linear regression model, binary classification, boost-tree's, matchbox, etc, etc, etc..? Possibly I need a combination.

I am using the Azure Machine Learning studio which has lots of options but I am just not sure where to start in this scenario. Any suggestions or pointers would be appreciated in helping


texts: How to determine percent contribution to classification in logistic regression?

texts:
I'm trying to understand the contribution of each feature towards a specific classification.  Let's say I have clients as my data and their features are income, debt and age.  We're trying to predict whether they will default on some loan.  client 1 has been classified as they will default with probability 95%.  I want to determine how much each feature has contributed to this client's classification result with the underlying goal of showing that income primarily drove client 1's default classification and debt primarily drove client 2's default classification for example.

Logistic regression is defined as

A single client's classification is

I want to understand how much each feature (b_i) has contributed but we can't split apart the features and calculate their portion of the probability unless we get rid of the exponential.  We can look to the contribution to log odds though

but now the terms on the right hand side can be positive or negative so I cannot say b_0 contributes 20% of the classification result for example.

this seems like a simple question but the answer isn't clear.  The log odds are above.  If we sum up -2, -0.4 and 0.3 and solve for p we get the probability of default.  We know that the largest number contributes the most to this classification but how do I compare 0.3's contribution to the -2 and -0.4?


texts:
I'm doing science project on my university to make an app, which uses AI to detect cyber crimes. I'm looking for sites to make my data set. Do you know any with ads like tobacco, alcohol, prescriptions, drugs, forgery? I'm using LASER embedder, so the site can be in almost any language. I'm looking for something like OLX, which allows illegal content. Dark web is fine as well.

I'm aware this question can somehow violate stackexchange terms of use. Please don't post particular site with an add, just a domain. I'll find posts myself. Something like avito.ru will do.


texts: How to enumerate the hypothesis search space in case of discrete valued input vector and binary label?

texts:
Some definitions used by the author in the example below:
X - It is a set of training examples, where each example is a feature vector with 6 features/attributes(Sky, AirTemp, Humidity, Wind, Water and Forecast)
H - set of all possible hypotheses that estimate the target function
f:X->y.
EnjoySport is the output label which is either Yes or No.
Consider, for example, the instances
X and hypotheses H in the EnjoySport learning task. Given that the attribute
Sky has three possible values, and that AirTemp, Humidity, Wind, Water, and
Forecast each have two possible values, the instance space X contains exactly 3.2.2.2.2.2 = 96 distinct instances. A similar calculation shows that there are
5.4.4.4.4.4 = 5120 syntactically distinct hypotheses within H. Notice, however,
that every hypothesis containing one or more "$\phi$" symbols represents the empty
set of instances; that is, it classifies every instance as negative. Therefore, the
number of semantically distinct hypotheses is only 1 + (4.3.3.3.3.3) = 973.
This excerpt is taken from Machine learning by Tom M. Mitchell. Ch-2, pg-23,24
I was unable to understand why 5 is there instead of 6 and also I am unable to follow semantically distinct hypotheses.


texts: Correlation coefficient and non linear association

texts:
Correlation and Linear Regression

Author:

Lisa Sullivan, PhD

Professor of Biostatistics

Boston University School of Public Health

Says there  may be non-linear association which the correlation  coefficient does not detect. I disagree with this idea. The Karl Pearson correlation coefficient actually computes the degree of relative changes in two variables which may be constant or varying.


texts: Select samples from a dataframe in python

texts:
I have a data set (pandas dataframe) with a variable that corresponds to the country for each sample. I have to take the samples that corresponds with the countries that appears the most.

thanks


texts: Difference between Data Engineer and Data Scientist

texts: I'm very confuse with the term Data Engineer and Data Scientist. There are lot of jobs available for both roles in current market with almost same technical skills requirement. Are they same or different ? Which role is recommended for someone from coding background (C++, Java, Python) along with RDBMS knowledge ?


texts: How to do webscrapping in R on this webpage?

texts:
I am quite new to R and I am trying to learn webscraping. I basically need to extract documents from this website.

Ideally, the data needs to be structured in three columns: YEAR, DATE, and
INTRODUCTORYSTATEMENT_CONTENT. Can anyone help with the coding?


texts: Standardizing Vegas odds for a randomForest

texts:
I'm sorry I don't have reproducible code, but I have a pretty specific question that I can't find an answer to.

I'm using randomForest to project NBA statistics. Vegas-odds are incredibly useful because it's provides the wisdom of the crowd. Intuitively I feel like they need to be standardized for analysis, but maybe randomForest is good enough.

The reason why I feel it needs to be standardized is because it's disjoint. If a team has a moneyline of -125, that means that you must pay \$125 to win \$100 (payout of \$225). If a team has a betting line of +110, that means you need to bet \$100 to win \$110 (payout of \$210). Therefore, it's disjoint in that there would never be scores in (-100, 100) since +100 or -100 are both even odds.

With that said, would you recommend reshaping the vector in some way so that the random forest can learn "better"? E.g. -125 is a (125/(100 + 125) 55.6% chance of winning and a +110 is a (100/110+100) 47.6% chance of winning. Would changing the moneylines to percentages help performance? I know the only surefire way to check would be to run models, but I really don't have time for it at the moment, and this question will help me to determine in general if/when standardizing is necessary.


texts: I observed that Adaboost or Bagging ensemble classifiers present in sklearn only work for single label training data. How do I use these for multilabel data?


texts: Role of Image Resolution in Deep Learning

texts:
I have multiple image datasets about the same topic that I want to use for a classification task utilizing Deep Learning. The datasets differ in the resolution of images (i.e. some pictures are 128x128px, some 512x512, others 2048x2048).
If I used the dataset with the highest resolution for training my intuition would be that it'll be harder to classify images with a lower resolution as the network learns patterns that it may not recognize in lower resolution pictures. On the other hand, if I use the low resolution dataset for training, the patterns learnt are more crude and it performs better on any new data as higher resolution images can be scaled down easily. Is my intuition right or am I missing something? What would be the best approach in my case for selecting proper training data?


texts: Downloading table in latex format from Stargazer using python

texts:
I have installed Stargazer to accumulate all the result and obtained the results in a well-organized way. I have used the following command to get result in latex format. But, the output generated from following command does not work in latex.
from stargazer.stargazer import Stargazer
from IPython.core.display import HTML

stargazer = Stargazer([results1,results2,results3,results4])
stargazer.as_latex()


texts: Transforming multiclass problem into binary problem to improve accuracy

texts:
I found a code for a classifier for a  6 class problem where the classes are integers from 0 to 5.
I found this code that I tested and it improved my accuracy highly.
Can anyone explain the spirit behind the idea of transforming the problem into a binary classification and how it improves the results?

class Classifier(BaseEstimator):
    def __init__(self):
        self.clf1 = BernoulliNB()
        self.clf2 = BernoulliNB()

    def fit(self, X, y):
        y1 = y.copy()
        y1[y==0] = 1; y1[y==2] = 1; y1[y==3] = 4; y1[y==5] = 4
        self.clf1.fit(X, y)
        y2 = y.copy()
        y2[y==1] = 0; y2[y==2] = 0; y2[y==3] = 0; y2[y==4] = 1; y2[y==5] = 1
        self.clf1.fit(X, y1)
        self.clf2.fit(X,y2)

    def predict(self, X):
        self.clf1.predict(X)
        self.clf2.predict(X)

    def predict_proba(self, X):
        A = np.array([[0,0],[1,0],[0,0],[0,0],[0,1],[0,0]])
        proba = self.clf1.predict_proba(X)
        B = self.clf2.predict_proba(X)
        #return proba_modified(np.array(list(map(lambda x: A@x, proba))),B)
        return np.array(list(map(lambda x: A.dot(x), proba)))


texts: Fast k-means like algorithm for $10^{10}$ points?

texts:
I am looking to do k-means clustering on a set of 10-dimensional points.  The catch: there are $10^{10}$ points.
I am looking for just the center and size of the largest clusters (let's say 10 to 100 clusters); I don't care about what cluster each point ends up in.  Using k-means specifically is not important; I am just looking for a similar effect, any approximate k-means or related algorithm would be great (minibatch-SGD means, ...).  Since GMM is in a sense the same problem as k-means, doing GMM on the same size data is also interesting.
At this scale, subsampling the data probably doesn't change the result significantly: the odds of finding the same top 10 clusters using a 1/10000th sample of the data are very good.  But even then, that is a $10^6$ point problem which is on/beyond the edge of tractable.


texts: How to Proceed with Tokenized text content which is given in number?

texts:
I have one data set of customer review, but the text data is given is tokenized text number. I am unable to proceed thinking about how to proceed?

As I am encountering such data set the first time, so just need guide how to proceed.

As you can see text field is given in number, so how to proceed please guide?.
it will predict the 0/1 +ve or -ve category.


texts: How does permutation of training data improve convergence time when training a perceptron or neural network model?

texts:
I'm currently studying some basic concepts regarding Deep Learning and Neural Networks with this material.

When discussing the training algorithm for a perceptron, the author states that looping through the training data in constant order is not always a good idea, specially in data sets where some labels could be somewhat ordered.

For example, say the model is being trained on a set of 1000 examples, the first 500 have a "positive" label, and the last 500 have a "negative" label. If these examples are iterated in their original order within the data set, in order to determine the weights of the perceptron, then after evaluating the error of the first few examples (which are "positive") the model would likely decide that every example is "positive", and would stop learning anything. It would perform well for a while, until it begins to evaluate the "negative" examples, when it would begin predicting everything as "negative". In the end, the model would've learned from only a handful of examples.

This makes perfect sense to me, and I understand the need to permute the data before iterating over it in the training algorithm. However, it is said that the convergence could be achieved even faster (about twice as fast), if the data is re-permuted in each iteration.

How does this improvement actually happen? Is there a way to actually quantify the difference in the number of epochs needed for convergence when the data is permuted once and when it is re-permuted with each iteration (maybe a statistical model, or something of the sort)? Also, are there any special cases where permuting the data with each iteration is actually ill-advised?


texts: What is the correct formatting of the input tensor for multi-variate LSTM on Pytorch?

texts:
I am working on a LSTM to predict a financial time series using 10 other financial time series. The 10 financial time series form my training dataset.

Several examples I have seen for univariate prediction use a rolling window approach. This would make sense as for each target value, only a certain amount of the training data is likely to have a predictive ability (say the 100 time steps prior to it).

What I would like to have is an LSTM which would have the 10 features and a window size of the 100 time-steps, which is associated with one target value. And then this would be rolled along for the full duration of the financial time series say (5000 values). But I'm having issues with this tensor dimensions.

In the Pytorch documentation, it describes the input_size arg as the "number of expected features". Which I would have thought would be the number of time series (10). But I haven't seen many multi-variate implementations of LSTMs and pretty much all of them have used the window size (e.g 100) as the input_size arg.

In Keras it would be as simple as shaping your input into [samples, timesteps, features]. Any advice or thoughts would be appreciated thanks!


texts:
In the PyTorch nn module there are 2 types of dropouts:

A normal Dropout - During training, randomly zeroes some of the elements of the input tensor with probability p using samples from a Bernoulli distribution. Each channel will be zeroed out independently on every forward call.
A Dropout2d - Randomly zero out entire channels. Each channel will be zeroed out independently on every forward call with probability p using samples from a Bernoulli distribution.

From Documentation

My question is when we use CNN, which of the dropout we use in the Convolutional layers. In this paper it is being said we use the first kind. But I am not so sure what is the general practice.


texts: Which time series analysis is appropriate for pooled time series data analysis?

texts: I have two data sets one is cross sectional census data with 10 years interval and another one is time series data  (monthly) for several years. Now I want to perform statistical time series analysis on this both dataset in a single model. Is it possible?


texts:
I am working on a project that requires the logit outputs (pre-softmax layer) on an accurate model for the CIFAR-10 dataset. I have been looking at Densenets since it has a softmax output layer. However, the current implementations that I've found either require training, or do not seem to be targeted towards CIFAR-10 (for instance the torchvision Densenets implmentation).

Are there any pretrained neural network models that are targeted for the CIFAR-10 dataset that would be easy to borrow and spit out the pre-softmax logit data? Preferably something like Densenets or as high accuracy as possible. Thanks!


texts: XGBoost failing on highly imbalanced data!

texts:
I am working on a classification problem, where I am trying to predict a fraud login. The data is highly imbalanced i.e.
0 = non fraud logins , 1 = fraud logins
0 : 4538076
1 : 365
I have been trying to model an XGBoost on this data . I have around 30 features. One such feature has the distribution as follows : (Most of the features have a distribution like this where we can clearly see the numbers are higher for frauds but there is also a considerable amount of overlap among two groups. I cannot put all features here so just an example of one feature)

Also attaching a KDE plot for another feature to give an idea about the data.

So,first I tried a borderline SMOTE, and undersampled few from the majority class too , the final distribution for train data is as follows:
After OverSampling, counts of label '1': 4877
After OverSampling, counts of label '0': 97540
With this , I have train, test and validation data(test and validation data are not resampled) and trained xgboost with the following parameters with a RandomizedCV.
# A parameter grid for XGBoost
params = {
        'min_child_weight': [1,5,10],
        'gamma': [0.5,0.4,1,2,2.5,1.5],
        'subsample': [0.8,0.95, 1.0],
        'colsample_bytree': [0.6,0.7 ,0.8,0.9,1.0],
        'colsample_bylevel':[0.6,0.7,0.8,0.9,1.0],
        'max_depth': [3, 4, 5],
        'alpha':[0,0.01,0.04,0.05,0.1,0.2,0.4,0.5],
        'lambda':[0,0.02,0.2,0.3,0.2,0.5],
        'eta':[0.007],
        'n_estimators':[100,300,500],
        'max_delta_step':[1,4,6],
        'scale_pos_weight':[1,100,20,500,1000]
        }

xgb = XGBClassifier(objective='binary:logistic', nthread=-1 , eval_metric = 'logloss')
folds = 10
param_comb = 150

skf = StratifiedKFold(n_splits=folds, shuffle = True, random_state = 1001)

random_search = RandomizedSearchCV(xgb, param_distributions=params, n_iter=param_comb,scoring='f1_weighted', n_jobs=-1, cv = skf.split(X_train,y_train), verbose=3, random_state=1001 )

random_search.fit(X_train, y_train ,early_stopping_rounds = 10, eval_metric = 'logloss' , eval_set = [(df_val,y_val)])

Results :
Best estimator:
XGBClassifier(alpha=0.05, colsample_bylevel=0.6, colsample_bytree=0.8,
              eta=0.007, eval_metric='logloss', gamma=0.5, lambda=0.02,
              max_delta_step=6, max_depth=5, n_estimators=500, nthread=-1,
              subsample=0.8)

 Best normalized gini score for 10-fold search with 150 parameter combinations:
0.9920305979778199

 Best hyperparameters:
{'subsample': 0.8, 'scale_pos_weight': 1, 'n_estimators': 500, 'min_child_weight': 1, 'max_depth': 5, 'max_delta_step': 6, 'lambda': 0.02, 'gamma': 0.5, 'eta': 0.007, 'colsample_bytree': 0.8, 'colsample_bylevel': 0.6, 'alpha': 0.05}
1

Results on train data :

Probability distribution on train data :

(I have even tried running a simple logistic regression with the labels and probabilities to calibrate it, doesn't work)
Results on test data  :
# confusion matrix
matrix = confusion_matrix(y_test,preds_test , labels=[1,0])
print('Confusion matrix : \n',matrix)

# outcome values order in sklearn
tp, fn, fp, tn = confusion_matrix(y_test,preds_test,labels=[1,0]).reshape(-1)
print('Outcome values : \n', tp, fn, fp, tn)

# classification report for precision, recall f1-score and accuracy
matrix = classification_report(y_test,preds_test , labels=[1,0])
print('Classification report : \n',matrix)

What am I doing wrong? How do I improve the model?
P.S. i am also exploring Isolation Forest / LOF but I dont think they aren't working out well in my case either.
Sorry, about the long question but I needed to make sure I give enough context.


texts: How to go about fine-tuning BERT using a next-sentence task

texts:
I've got a large corpus of documents, and I want to use bert to generate embeddings for a variety of predictive tasks.  The documents are multi-sentence, in a non-standard domain, and have labels at the document level, rather than at the sentence level.
Given that the labels don't map to sentences, I want to augment them with a next-sentence prediction task, which is what I understand that BERT is originally trained on.
My basic question is this: is it just as simple as setting the label to be the next sentence, tokenized into a vector of token IDs, and then one-hot-encoded?
So, if the max sequence length is 512 and the vocab length is 30K, should I simply load pre-trained bert and append a layer with shape None (batchsize), 512, 30K?  Or is there some more efficient way of achieving the same result?
I've been looking at the Bert repo on github, but it's all heavily optimized code and hard to follow, so I'd appreciate a high-level summary in answer to this question; it'd guide my eye as I look through the code.
Lastly:  I'm aware of huggingface and how easy they make it to do this.  But I'm trying to stick with tensorflow 2.0, to maintain compatibility with other parts of my stack.


texts: How to iterate and modify rows in a dataframe( convert numerical to categorical)

texts:
I have a pandas dataframe like this

0    15.55
1    15.55
2    15.55
3    15.55
4    20.84
Name: Y1, dtype: float64

I want to convert the values of Y1 to categorical (i.e) if its greater than 18.25, I want it 1 else 0

Can someone please help me on how to do it

This is what i tried so far

for temp in TRAIN_ID1:
    train_ID1.loc[(train_ID1['Y1'] > 18.250000), 'Y1'] = 1
    train_ID1.loc[(train_ID1['Y1'] < 18.250000), 'Y1'] = 0

But im getting an error

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

TypeError: an integer is required

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
<ipython-input-118-2cccb791d834> in <module>()
      1 for temp in train_ID1:
----> 2     train_ID1.loc[(train_ID1['Y1'] > 18.250000), 'Y1'] = 1
      3     train_ID1.loc[(train_ID1['Y1'] < 18.250000), 'Y1'] = 0

~\Anaconda3\envs\deeplearning\lib\site-packages\pandas\core\series.py in __getitem__(self, key)
    621         key = com._apply_if_callable(key, self)
    622         try:
--> 623             result = self.index.get_value(self, key)
    624
    625             if not is_scalar(result):

~\Anaconda3\envs\deeplearning\lib\site-packages\pandas\core\indexes\base.py in get_value(self, series, key)
   2558         try:
   2559             return self._engine.get_value(s, k,
-> 2560                                           tz=getattr(series.dtype, 'tz', None))
   2561         except KeyError as e1:
   2562             if len(self) > 0 and self.inferred_type in ['integer', 'boolean']:

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_value()

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_value()

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

KeyError: 'Y1'


texts: NLP: Compare tags semantically with machine learning? (finding synonyms)

texts:
Let's say I have multiple tags that I need to compare semantically. For example:

tags = ['Python', 'Football', 'Programming', 'Handball', 'Chess', 'Cheese', 'board game']

I would like compare these tags (and many more) semantically to find a similarity value between 0 and 1.
For example, I want to get values like these:

f('Chess','Cheese') = 0.0  # tags look similar, but means very different things
f('Chess', 'board game') = 0.9 # because chess is a board game
f('Football', 'Handball') = 0.3 # because both are sports with a ball
f('Python', 'Programming') = 0.9 # because Python is a programming language

So what is the state of the art approach to get a function f like this? I know that machine learning might be doing this, but this area is huge and overwhelming for me (on the first glance it looks like NLP focuses on other problems).
So what would be the best approach for this specific problem?


texts:
What I met a problem is I do time-series clustering, and I found the clustering result isn't ideal.

I can't use elbow method to know what clustering result is good, that means I have no ways to watch the clustering results and tune parameters.

What I want to do is cluster the same trends data into the same group.

But I met a problem like below:

>> arr # It's a time-series array with 10 time intervals.
array([[  0,   0,   100, 0,   0,   0,   0,   0,   0,   0],
       [  0,   0,   0,   0,   0,   0,   0,   0, 100,   0],
       [100,   0,   0,   0,   0,   0,   0,   0,   0,   0]])

>> model = KM(n_clusters=2).fit(arr) ; model.labels_
array([1, 0, 0], dtype=int32)

It clusters arr[1] and arr[2] the same group!

But!With our eye watching, we all know arr[0] and arr[2] should be the same group because they have 100 in the near time intervals.

How to do time-points clustering? And specify random_state=N is useless becuase there is always one corner case to let me fail.

UPDATE:

There's one algorithm to solve this and its name is KShape. It can be found in tslearn github.

It clusters time-series data based on the shape of each data, so it match my needs.


texts: Is there a way to add a API in Orange Data Mining Tool to access a new website? An example would be can I create and add an API for CNN or MSNBC or FOXNEWS? Currently in Orange in the text mining add-on they only have a couple of websites plus twitter. I am looking to add some of my own API to text mine.


texts: Dot product for similarity in word to vector computation in NLP

texts:
In NLP while computing word to vector we try to maximize log(P(o|c)). Where P(o|c) is probability that o is outside word, given that c is center word.

Uo is word vector for outside word
Vc is word vector for center word
T is number of words in vocabulary
Above equation is softmax. And dot product of Uo and Vc acts as score, which should be higher the better. If words o and c are closer then their dot product should be high, but that is not the case with dot product. Because of following:
Consider vectors A=[1, 1, 1], B=[2, 2, 2], C=[100, 100, 100]
A.B = 1 * 2 + 1 * 2 + 1 * 2 = 6
A.C = 1 * 100 + 1 * 100 + 1 * 100 = 300
Vectors A and B are closer compared to A and C, however dot product A.C >A.B
So Dot product is acting as distance and not as similarity measure. Then why is it used in softmax. Please help me make my understanding better


texts: Scikit-compatible network lasso implementation

texts:
Is anyone aware of a scikit-compatible network Lasso (nLasso) implementation?

These papers have source code as well:

  D. Hallac, J. Leskovec, and S. Boyd, “Network lasso: Clustering and optimization in large graphs,” in Proc. SIGKDD, 2015, pp. 387–396.

Code: https://riken-yamada.github.io/localizedlasso.html

  [2] M.Yamada, T. Koh, T. Iwata, J. Shawe-Taylor, and S. Kaski, “Localized Lasso for High-Dimensional Regression,” in Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, vol. 54. Fort Lauderdale, FL, USA: PMLR, Apr. 2017, pp. 325–333.

Code: https://github.com/davidhallac/NetworkLasso

But these particular implementations are not very general and can't be used as a part of the pipelines.

I would like to use nLasso to do something like this (the example picked from the scikit tutorial):

import matplotlib.pyplot as plt
import numpy as np

from sklearn.datasets import load_boston
from sklearn.feature_selection import SelectFromModel
from sklearn.linear_model import LassoCV

# Load the boston dataset.
boston = load_boston()
X, y = boston['data'], boston['target']

# We use the base estimator LassoCV since the L1 norm promotes sparsity of features.
clf = LassoCV(cv=5)

# Set a minimum threshold of 0.25
sfm = SelectFromModel(clf, threshold=0.25)
sfm.fit(X, y)
n_features = sfm.transform(X).shape[1]


texts:
So I was working on a classification task with the help of a NN. The data-set was normalised, weights random between 0-1, and all the activations were sigmoid function.

Now, when I used a 2 hidden layer model the accuracy was 50%, whereas, when I used 1 hidden layer model the accuracy was 99%. Isn't this contrary to intuitive understanding about NN's. I knew more layers means better fitting even over-fitting, but apparently something different is happening in this case (maybe the values outputted by the second hidden layer is too small for the output layer to discern). So what exactly am I missing?


texts: Input standartization for Deep Learning - Proper Scaling

texts:
Typically the input to neural network (NN) is transformed to have zero mean and 1 std.

I wonder why std scale should be 1? What about other scales? 10? 100?
Doesn't it make sense to provide NN with input of wider range so that NN can separate different clusters easier and deal with loss function for each cluster in more simple and robust way?
Did someone here tried different scales and can share his experience?

If answer depends on the activation function - in my case I use Relu.

Thanks a lot!


texts:
I am not an experienced user of Excel because I usually use python but I am obliged to use it this time.

This is the big picture of my case.

The problem is:

I have a sheet with a column with some words, another column with the translation of these words in a non standard written way because they are written using numbers (such as "sale" will be "5413") and they are aligned with the corresponding standard written words (i.e. "sale").

In another sheet I have the same non standard written words (first column) that are sorted from 0 to 9 and there are also some numerical values associated to them.

My goal is:
take the values of the second sheet and put them in the order of the first sheet so that I can have the values of the non standard written words in the same order of the standard written words.

Someone suggested me to use the VlookUp function.

=VLOOKUP($D2;$'leet_nopunct (2)'.$2:$1000;**97**;0)

It works! But I have to change the second last number (here 97) every time for each column of the second sheet and it is a very long process.

I know that Excel can use some macro: is there a way to loop this number in order to be automatically incremented it?


texts:
Have you ever seen a neural network without matrices?

I'm asking, because I'm currently building one for educational purposes.


texts: Which book is a standard for introduction to genetic algorithms?

texts:
I have heard of genetic algorithms, but I have never seen practical examples and I've never got a systematic introduction to them.

I am now looking for a textbook which introduces genetic algorithms in detail and gives practical examples how they are used, what their strengths are compared to other solution methods and what their weaknesses are.

Is there any standard textbook for this?


texts: How to standardize or normalize in neurons

texts:
Question
Please help understand where and how to do standardization (convert values in-between -1 and 1) at a neuron and the rationals, or please correct if my understanding is wrong. I am assuming any data to process is better standardized or normalized but not 100% sure.
Neuron input initialization (2)
For example, image pixel values as input are normalized between 0 and 1. Why not standardize? Even when weights are standardized, if the inputs are not, it can have a bias towards plus. Please shed a light.
Product sum in neuron (3)
I think there is no normalization by $\frac{1}{m}$ (where m is number of inputs into neuron) on the product calculation result in a neuron. After the Batch Normalization (BN) is invented and used in front of the activation function, it will not be required but without BN, why not?
$\sum_{i=1}^m w(i) * x(i) $ instead of $\frac{1}{m}\sum_{i=1}^m w(i) * x(i) $
Also when normalized, I think the input value area of the activation function will stay in the area where saturation is less likely, hence it seems the saturation issue could be avoided.
Activation output (4)
Activation function is an on/off binary switch-ish. The value depends on the function but it goes into the next neurons as input. I suppose there should be a standardization mechanism to map ON/Off values into proper neuron input values, but seems there is not. Please help understand why.


texts: Are units sold,order quantity,Profit,loss continuous or discreet?

texts:
Continuous :Cant be counted and has no limit
Discreet : Can be counted
Please help to understand if my assumption is right or wrong linear regression is used to predict only continuous variables so can i use linear regression to predict any one of below variable ?
Are no of units sold ,Profit,loss continuous or discreet?
no of units sold : For example if i have a shop that sells only toys  and i want to  predict the no of units sold every month  should i consider  no of units sold as   continuous or discrete
1.In reality no of units sold will have a limit based on no of  unit of toys i  purchased
2.Theoretically there is no limit to the units that my shop will sell if purchased unlimited units of toys,so if i list down all possible values for the units that i might sell it will be endless it can be 100,101,102 ....10000000000001.... (i have second though on whether this statement is correct )


texts: Excluding "mislabeled" examples in training set based on out of time data

texts:
This question is specifically regarding imperfect labels.

I'd like to understand the theoretical and practical implications of removing examples from the training set based on information obtained outside the performance window.

For example, let's say we're looking to identify fraudulent accounts in our data, so we define our label as "is this account flagged as fraud in the next 30 days"? Despite using this, for many accounts we have performance data beyond 30 days, and for some substantial subset of these accounts it turns out they're flagged after more than 30 days (and let's just assume for this exercise that they were already fraudulent when we sampled for the training dataset). Keeping these examples in the training data introduces noise, which can significantly impact the model's performance if very precise labeling is necessary.

What is wrong with removing these examples from our training data (since we don't want to change the label, but know that the label is incorrect)? Given that we have a completely separate test set (out of time and out of sample).


texts:
I am trying to make a convolutional neural network that classify images in two categories: with cats and without cats. It's the first time I am doing something like this and it seems I am having a problem.

What I was doing was to take some pictures, resize them to 100 by 200, apply a sobel filter to detect edges which worked, than make a pooling layer of size 2 by 2 which would reduce the complexity. I don't want high accuracy, for a beginner I think it's enough if this works for a few photos.

After applying the pooling layer, I created a fully connected layer. It had 2500 input neurons, 1000 hidden neurons in one layer and one output neurons. I used the sigmoid activation function. In the end I have to minimize this function in order to train my network: 2(y-sig(sig(X*W1)*W2))^2.

y is the expected result, X is the input vector, W1 and W2 are the two sets of weights and sig() is the sigmoid function. The design I thought of was that if the output was greater than 0.5 than there was a cat in the image otherwise not.
The problem I am having is that after I generate the W1 and W2 in matlab using rand() all my values in the hidden layer are equal to 1 which is obvious as W1 always has positive values for some reason. After making some calculations in my head I realized that the output will always be equal to 1 because sig(X*W1)*W2 is a big number so, if I am correct, the network will identify a cat in almost every photo and training will not change much as the error is already small. My question is am I doing something wrong on my design or should I just find a way to have both negative and positive values for my weights? Thanks, and sorry for such a long text!


texts:
This is a captcha where you have to select the side that makes more sense.
I am trying to use machine learning trying to break it.
My approach is to use Google's vision AI to extract keywords from those images and then use a markov chain and machine learning to predict the correct order.

Is there a better approach in your opinion? What are the chances to get it working?