texts: Extremely stochastic validation loss/accuracy

texts:
I am working on training DNNs on satellite data. The class distribution in the data is extremely imbalanced, so I train the neural networks using random majority undersampling to artificially balance the number of training examples from each class. During validation, I don't resample the data in any way.

In the graphs above, orange is training performance and blue is validation.
The validation accuracy and loss values are much much noisier than the training accuracy and loss. Validation accuracy even hit 0.2% at one point even though the training accuracy was around 90%. Why are the validation metrics fluctuating like crazy while the training metrics stay fairly constant?
Some additional information on the model/data:
I'm segmenting satellite images into three land cover classes using a U-Net model. The U-Net model has half the number of filters at each layer compared to the original paper, and I apply batch normalization after each convolution and before the activation. The model is trained from scratch with the Adam optimizer with an initial lr=0.001 and a step decay schedule, optimizing (unweighted) categorical crossentropy. The U-Net has l2 weight regularization applied on each convolutional layer with constant 0.01.
The training and validation data are all floating point images with values in the range 0-2. The images for the two datasets were sampled from the same spatial region. I don't do any preprocessing or data augmentation because of the computational overhead.
The validation set has 317k pixels in class 1, 26.5M pixels in class 2, and 27.8M pixels in class 3 (this is the "true" data distribution). The number of training instances which contain pixels in class 1 is ~200 out of 3500. The training set (in which class 1 is spatially oversampled to artificially balance the training set), consists of 58M class 1 pixels, 166M class 2 pixels, and 127M class 3 pixels. Out of 20k training instances, approximately 8200 contain class 1 pixels.
When feeding data to the model, I sample from the three datasets with probability 0.4, 0.3, and 0.3 for classes 1, 2, and 3 respectively.
EDIT: Problem solved! When working with neural networks and geospatial data it's very important to maintain the same map projection between the test/train/validation sets. I thought the software I was using to extract the test/train/validation data was ensuring the same projection between sets, but I was wrong. Making the projections of the test/train/validation data the same fixed my problem. Essentially I was validating on a different dataset than I was training on, which is not how machine learning models are supposed to be validated.


texts: How to keep track of assumptions/design choices and avoid errors in complex projects

texts:
I am part of a small team data analysts that has been trying to optimise the process we use to keep track of the assumptions and design choices that are made over the lifetime of a complex data science project (several months) - ultimately we want to (i) catch (conceptual) errors when they are made (ii) ensure that it is at all times clear what assumptions underlie the results being produced.

Our current process relies on code-generated and manually maintained excel spreadsheets - the process is (i) time consuming/results in too many files, (ii) introduces a new source of error (the copying of results into the spreadsheet) and (iii) is not dynamic, i.e. cannot easily be adapted as a project evolves.

While there are a large number of questions/answers focused on version control, reproducible research and project organisation more broadly we are not looking for a new suite of tools but a light-weight process that addresses a very specific problem.

The ideal process:

Is easy to  maintain & as much as possible automated to reduce
errors
Captures all design choices and assumptions made
At a glance reflects the impact of any changes, i.e. reflects (changes
in) the 'key metrics' that resulted from the change
Links all changes with the related code file versions
Can easily be shared with other analysts & Principal Investigators


texts: As RELU is not differentiable when it touches the x-axis, doesn't it effect training?

texts:
When I read about activation functions , I read that the reason we don't use step function is because, it is non differentiable which leads to problem in gradient descent.
I am a beginner in deep learning , as Relu is almost a linear function and also non differentiable where it touches x-axis , why it performs so much better than tanh or sigmoid functions. And why is it so widely used in Deep learning.
As it is non differentiable doesn't it affect in training?


texts:
I'm studying Logistic Regression for the first time.

My book explains how to fit the weights $w$ using the likelihood function $L(w)$, assuming that the individual samples in our dataset are
independent of one another.

I know what independent events are. But I don't understand the meaning of the individual samples in our dataset are
independent of one another in the Logistic Regression case. What does this mean? And why?


texts: I am currently using TF 2.1. When using model.fit with tf.data.Dataset, it shows me that the number of iterations is 2044/2043. So my question is, how many batches of data are actually produced? 2044 or 2043?


texts: What is the structure of a MLP that allows for arbitrary decision boundaries?

texts: I'm just doing some reacher for my machine learning class, I found out it must be three layers MLP. But why? Is there a proof/paper or I'm wrong?


texts:
I am looking for a thesis to complete my master M2, I will work on a topic in the big data's field (creation big data applications), using hadoop/mapReduce and Ecosystem ( visualisation, analysis ...), Please suggest some topics or project that would make for a good masters thesis subject.
I add that I have bases in data warehouses, databases, data mining, good skills in programming, system administration and cryptography ...

Thanks


texts: Is there any conceptual relationship between 'kernel' in SVM and 'kernel' in convolution neural net?

texts:
In SVM, we have kernel function that maps an input raw data space into a higher dimensional feature space

In CNN, we also have a 'kernel' mask that travels the input raw data space (image as a matrix) and map it to another space.

Given the fact that both these to methods are called 'kernel', I am wondering what is the connection between them from a mathematical perspective.

My guess is that it might have something to do with functional analysis.


texts: How to read the labeled enron dataset categories?

texts: I am trying to use the labeled Enron dataset (link) but I am really confused about the labeling system they use. I understand the Cat_[1-12]_level_weight is some form of confidence level. This dataset labeled by multiple students. Cat_[1-12]_level_weight increases with the number of the same label assigned by multiple students to a certain row (sample). But, what is for instance the Cat_1_level_2? From the overview (in the link), I guess Cat_1 mean "Course genre" and level_2 mean  "Purely Personal (49 cnt.)" ? If so, why the Cat_1_level_2 has values like 1, 2, 3, 4, etc.? I am really confused.


texts: How to implement large-scale Poisson Regression in Python

texts: I am trying to implement a Poisson Regression in Python to predict rates. I am dealing with a ton of data (too much to store in a DataFrame), which means that using the standard statsmodels.api GLM Poisson Regression won't work. I know that sklearn has a partial_fit() method with the SGDRegressor and SGDClassifier classes for Minibatch learning, but I cannot figure out how to implement a Poisson Regression with these classes. Does anyone know how to implement a Poisson Regression with large-scale data in Python?


texts: Is there any standard pattern recognition algorithm in predicting an item which a user will be buying next, given I have the history of the purchases

texts:
I am having a list of 10 different items a user has bought in the past. Each item has been bought multiple times. I would like to find a pattern in which the user buys a particular item and predict what he will be purchasing next.

For example I am running a cloth factory and sell clothes to different retailers. I would like to identify what clothes a particular retailer might be asking me next, based on the history of how the retailer has purchased different clothes.

We cannot use time series as the date of purchases has no equal intervals. Also I feel simply passing the sequence of purchased products ordered by date to a neural network will not be of much help.

Is there any standard algorithm to identity a pattern?

I am having two columns in my data: 1. item_id  2. date_of_purchase

I would like to create a model which will predict top 3 items which the user will be buying again next. Input to my model will be present date. Output should be top 3 items which the user might be interested in buying.


texts: Embedding a Machine Learning Model into a Web Application

texts: I'm looking  to  find some  tutorials  involving embedding a Machine Learning Model into a  Web Application eventualy with some storage and database behind( with things like slack  etc)  .


texts: Binary classification model for unbalanced data

texts:
I have a dataset with the following specifications:

Training dataset with 193,176 samples with 2,821 positives
Test Dataset with 82,887 samples with 673 positives
There are 10 features.

I want to perform a binary classification (0 or 1). The issue I am facing is that the data is very unbalanced. After normalization and scaling the data along with some feature engineering and using a couple of different algorithms, these are the best results I could achieve:

mean square error : 0.00804710026904
Confusion matrix : [[82214   667]
                   [    0     6]]

i.e only 6 correct positive hits. This is using logistic regression. Here are the various things I tried with this:

Different algorithms like RandomForest, DecisionTree, SVM
Changing parameters value to call the function
Some intuition based feature engineering to include compounded features

Now, my questions are:

What can I do to improve the number of positive hits ?
How can one determine if there is an overfit in such a case ? ( I have tried plotting etc. )
At what point could one conclude if maybe this is the best possible fit I could have? ( which seems sad considering only 6 hits out of 673 )
Is there a way I could make the positive sample instances weigh more so the pattern recognition improves leading to more hits ?
Which graphical plots could help detect outliers or some intuition about which pattern would fit the best?

I am using the scikit-learn library with Python and all implementations are library functions.

edit:

Here are the results with a few other algorithms:

Random Forest Classifier(n_estimators=100)

[[82211   667]
[    3     6]]

Decision Trees:

[[78611   635]
[ 3603    38]]


texts:
How to decide what threshold to use for removing low-variance features?

Particularly, I have 100000 features and the variances look like:

Could I e.g. take the average and use it to split this to ~half?

Or some other method of grouping?


texts: Non-uniform class occurances in input data for classification task - how to tackle it?

texts:
So, I gathered political articles for my thesis, now I want to be able to classify given text. Though the classes distribution is actually crazy.

Class 1: 964 docs
Class 2: 37,020
Class 3: 640
Class 4: 2,675
Class 5: 793
Class 6: 23,160
Class 7: 2,665

Such a skewed data is obviously going to favor classes 2 and 6, though I thought about elevating the difference from last layer for classes with lower observations, is that worth a shot? Or it will actually create overfit for these classes? Unfortunately I can't scrap more data, the websites with articles doesn't have any more (at least now). Of course any data augmentation is not possible.


texts: Why does prediction by a consensus of classifier work better than prediction by a single classifiers?

texts: I have seen that consensus of classifiers (taking say 5 separate classifiers) and obtaining the final labeling of the unknown sample based on the voting method (whichever class gets the predicted the most is the class of the unknown sample) works better than taking a single classifier for predicting the class of a sample. Why so? Any articles which show why this happens?


texts: How can I figure out section headings in a document?

texts: I have a large number of different XML documents with associated style sheets (SEC financial filings). Are there any natural language processing (or other) tools that I can use to try to automatically extract section headings?


texts: Need help in understanding a small example

texts:
Pardon me, I agree the title of the question is not clear. I would like to know the understanding of below steps which are picked from the textbook "Hands on machine learning".

 >>> housing['income_cat'].value_counts()
 >>> 3.0    7236
     2.0    6581
     4.0    3639
     5.0    2362
     1.0     822

  If I am not wrong, the above step is to get the counts for each class. For example, for class '3' there are 7236 instances. Likewise, for class '2' there are 6581 instances.

>>>  housing['income_cat'].value_counts / len(housing)
>>>  3.0    0.350581
     2.0    0.318847
     4.0    0.176308
     5.0    0.114438
     1.0    0.039826

  Next, I was not clear, what was the intention behind the above step. By doing the above step, what am I suppose to learn ?. and

 >>> from sklearn.model_selection import StratifiedShuffleSplit

 >>> split = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
 >>> for train_index, test_index in split.split(housing, housing["income_cat"]):
         strat_train_set = housing.loc[train_index]
         strat_test_set = housing.loc[test_index]

 >>>  strat_test_set['income_cat'].value_counts() / len(strat_test_set)
 >>>  3.0    0.350533
      2.0    0.318798
      4.0    0.176357
      5.0    0.114583
      1.0    0.039729
      Name: income_cat, dtype: float64

  How come strat_test_set['income_cat'].value_counts() / len(strat_test_set) results are almost same to the results of housing['income_cat'].value_counts / len(housing) ?


texts:
I have studies the gans in depth and some of its type like cycle, pix2pix, cgans. Now I want to generate random images from random distribution from generator. So I am creating a dataset with no distribution similarity between images like sky, space, designs, mobiles, and all random pictures so will generator will be able to produce random things or will it fail to understand anything ?
And what GAN's to use for this purpose ?


texts: What is a suitable loss function and evaluation metric for a classification model with large number of unbalanced target classes?

texts:
I am building a multiclass classifier to predict the "Intent" of a question.
There are some 100 classes in the target variable and each target class contains an unequal proportion of observations/questions varying from 3 % to 40 %.

Questions

What would be a good evaluation metric for this classifier?
What is the best proxy loss function for optimizing the suggested evaluation metric?

EDIT: This is not a ranking problem. The model should predict only 1 target class.
The cost of misclassification is the same for each target class.


texts: Can The linearly non-separable data be learned using polynomial features with logistic regression?

texts:
I know that Polynomial Logistic Regression can easily learn a typical data like the following image:

I was wondering whether the following two data also can be learned using Polynomial Logistic Regression or not.

I guess I have to add more explanation. Assume the first shape. If we add extra polynomial features for this 2-D input (like x1^2 ...) we can make a decision boundary which can separate the data. Suppose I choose X1^2 + X2^2 = b. This can separate the data. If I add extra features I will get a wavy shape (maybe a wavy circle or wavy ellipsis) but it still can not separate the data of the second graph, can it?


texts: What is an 'old name' of data scientist?

texts:
Terms like 'data science' and 'data scientist' are increasingly used these days.
Many companies are hiring 'data scientist'. But I don't think it's a completely new job.
Data have existed from the past and someone had to deal with data.
I guess the term 'data scientist' becomes more popular because it sounds more fancy and 'sexy'
How were data scientists called in the past?


texts: Decision trees, categorizacion and oversampling

texts:
I want to create a model to predict the propensity to buy a certain product. As my proportion of 1's is very low, I decided to apply oversampling (to get a 10% of 1's and a 90% of 0's).
Now, I want to discretize some of the variables. To do so I run a tree for each variable against the target.
Should I define the prior probabilities when I do this (run the trees), or it doesn't matter and I can use the over-sampled dataset just like that?


texts: Machine learning algorithms for suggesting new baby names

texts:
https://www.lexalytics.com/lexablog/machine-learning-natural-language-processing
https://towardsdatascience.com/named-entity-recognition-ner-meeting-industrys-requirement-by-applying-state-of-the-art-deep-698d2b3b4ede
https://medium.com/mysuperai/what-is-named-entity-recognition-ner-and-how-can-i-use-it-2b68cf6f545d
https://blog.skyl.ai/understanding-deep-learning-algorithms-for-ner
https://www.bounty.com/pregnancy-and-birth/baby-names/baby-name-search
Can Machine learning algorithms assist in suggesting new Baby names?
Separate input dataset for all the male and female existing names.
Separate input dataset for male and female gender names.
Entity Category : Person First Name.
Note : Every Person Name has a meaning.
Thanks & Regards,
Prashant S Akerkar


texts:
I'm learning how to use Keras and I've had reasonable success with my labelled dataset using the examples on Chollet's Deep Learning for Python. The data set is ~1000 Time Series with length 3125 with 3 potential classes.
I'd like to go beyond the basic Dense layers which give me about 70% prediction rate and the book goes on to discuss LSTM and RNN layers.
All the examples seem to use datasets with multiple features for each timeseries and I'm struggling to work out how to implement my data as a result.
If for example, I have 1000x3125 Time Series, how do I feed that into something like the SimpleRNN or LSTM layer? Am I missing some fundamental knowledge of what these layers do?
Current code:
import pandas as pd
import numpy as np
import os
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM, Dropout, SimpleRNN, Embedding, Reshape
from keras.utils import to_categorical
from keras import regularizers
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt

def readData():
    # Get labels from the labels.txt file
    labels = pd.read_csv('labels.txt', header = None)
    labels = labels.values
    labels = labels-1
    print('One Hot Encoding Data...')
    labels = to_categorical(labels)

    data = pd.read_csv('ts.txt', header = None)

    return data, labels

print('Reading data...')
data, labels = readData()

print('Splitting Data')
data_train, data_test, labels_train, labels_test = train_test_split(data, labels)

print('Building Model...')
#Create model
model = Sequential()
## LSTM / RNN goes here ##
model.add(Dense(3, activation='softmax'))

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

print('Training NN...')
history = model.fit(data_train, labels_train, epochs=1000, batch_size=50,
    validation_split=0.25,verbose=2)

results = model.evaluate(data_test, labels_test)

predictions = model.predict(data_test)

print(predictions[0].shape)
print(np.sum(predictions[0]))
print(np.argmax(predictions[0]))

print(results)

acc = history.history['acc']
val_acc = history.history['val_acc']
epochs = range(1, len(acc) + 1)

plt.plot(epochs, acc, 'bo', label='Training acc')
plt.plot(epochs, val_acc, 'b', label='Validation acc')
plt.title('Training and Validation Accuracy')
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.legend()
plt.show()


texts: Linear vs Non linear regression (Basic Beginner)

texts:
So my doubt is basically in Linear regression, We try to fit a straight line or a curve for a given training set. Now, I believe whenever the features (independent variable) increases, parameters also increases. Hence computing these parameters is computationally expensive. So, I guess that's the reason we move to Non linear!?  Is my understanding right?

And my next doubt is, in overfitting for linear regression, we say that the model memorizes. What I understand is that the parameters are adjusted in such a way that it can only predict for the given input and will give bad results for output
 And the reason is because we haven't fitted a generalized line or curve. But we are plotting a line or curve that passes through all output values. Is my understanding correct?


texts: Does YOLO give preference to color over shape or vice-versa while detecting an object?

texts: If you train your YOLO model only on grayscale images to detect car, then would it able to recognise a car in a colored image also. If so, then can I assume that YOLO consider only object shape not color? Kindly clarify.


texts: What is the difference between domain randomization and data augmentation?

texts:
Domain randomization (https://arxiv.org/abs/1703.06907) is used to create a synthetic dataset with enough variance that it will encompass unseen real data, as just one variation.

I am trying to understand how this is different from applying data augmentation techniques to a synthetically generated dataset.


texts:
This is the first time I am working with OCR. I have an image and want to extract data from the image. My image looks like this:

I want to extract the parameters and the values against them. Can someone guide me on how to do this? I know we can extract text from an image using tesseract and PIL libraries if the image contains some simple text. What to do in this case where there are several parameters?


texts: Probability distribution in input-output pairs

texts:
This question might sound silly. But I have been wondering why do we assume that there is
a hidden probability distribution between input-output pairs in machine learning setup ?

For example, if we want to learn a function $f: \mathcal{X} \rightarrow \mathcal{Y}$, we generally tend to assume a probability distribution $\rho(x,y)$ on $Z=\mathcal{X} \times \mathcal{Y} $ and try to minimize the error
$$
\mathcal{E}(f) = \int (f(x)-y)^2 \ d\rho(x,y)
$$

Is the probability distribtution $\rho$ inherent to the very nature of $Z$ or depends on $f$ ?

Can anyone please provide a good intuitive explanation for this ?


texts: Having problem in back propagation part for dimension

texts:
I was trying to build a neural network with single hidden layer from scratch. In back propagation part some problems have raised.
For calculating gradient of loss function with respect to weight in layer-1 the equation becomes:

$$ \frac{dL}{dW1} = \frac{dL}{dA1}*\frac{dA1}{dZ1}*\frac{dZ1}{dW1}$$
where,

$$ \frac{dL}{dA1} = \frac{dL}{dA2}*\frac{dA2}{dZ2}*\frac{dZ2}{dA1}$$

Should I calculate it with np.dot() or np.multiply()?

I was trying to do it with np.dot() and having problem with dimension. dL/dW1 dimenion doesn't fit when I go to update W1.

Here is my code and pardon me as it is little bit messy in back propagation part.

https://gist.github.com/ipritom/30fcad0c74ab59e5b31e1daac1c1d1e7


texts:
I'm looking at trying to build a model to score an archery end from a photo but i'm not sure about the optimal ConvNet architecture.  What output neurons would I use given I can have an unknown number of arrows in a photo?  I could do a regression type solution of 'total score' I guess but i'd like to get a per arrow score (or perhaps the x,y coordinates of each arrow on a projected grid).

Any thoughts, wisdom, etc would be appreciated.  I'm likely not quite thinking about this the right way.


texts: Is it feasible to use decision tree algorithms for sensor fault detection?

texts:
The gist is me wanting to separate system faults from sensor faults given some dataset from a wireless sensor network using a machine learning algorithm.

For instance, if I have some temperature sensors in a given area and their corresponding readings from every sort of time interval, I would like to know whether an abnormal value is due to an actual fault, or due to a faulty sensor. Of course, it would be a given that the training set would have such entries tagged with either sensor fault or system fault.

I have thought of just using something like linear regression but I would like it to work even if the system could not be modeled like that. Decision tree seemed to me like a more appropriate algorithm for this.

Lastly, there is also some consideration for the time it takes for training and classification as I wish to see if it can be used for systems which respond really quickly to such anomalies.

Sorry if it's a bit wordy but I wasn't sure how much information I should put since this is my first time posting (I'm not even sure if this is the right stack exchange to post this). Anyway, thanks in advance for the answers!


texts: How to use ARIMA to predict time series?

texts:
Time            Volume
9/30/2019          2000
10/1/2019          1800
10/2/2019          1600
10/3/2019          1400
10/4/2019          1200
10/5/2019          1000
10/6/2019           800
10/7/2019           600
10/8/2019           400
10/9/2019          2000
10/10/2019         1800
10/11/2019         1600
10/12/2019         1400
10/13/2019         1200
10/14/2019         1000
10/15/2019          800
10/16/2019          600

  I have this dataset with Volume consumption of a tank over period of time. The volume decreases with time and on 10/9/2019 the tank has a load-up and is full tank. I have data for over a year and would like to use ARIMA to predict when would the next load-up happen based on previous consumptions.


texts:
The disadvantage of using transfer learning is that it cannot be layered to reduce the number of parameters.
In that statement what are the layers of transfer learning and the number of parameters?


texts: Is there any consensus on choosing an appropriate ML approach?

texts:
I am studying data science at the moment and we are taught a dizzying variety of basic regression/classification techniques (linear, logistic, trees, splines, ANN, SVM, MARS, and so on....), along with a variety of extra tools (bootstrapping, boosting, bagging, ensemble, ridge/lasso, CV, etc etc). Sometimes the techniques are given context (eg. suitable for small/large datasets, suitable for a small/large number of predictors, etc) but for the most part, it seems like for any regression or classification problem there exist a dizzying array of options to choose from.

If I started a job in data science right now and was given a modelling problem, I don't think I could do any better than just try all the techniques I know with basic configurations, evaluate them using cross-validation and pick the best. But there must be more to it than this.

I imagine an experienced data scientist knows the catalogue of techniques well and follows some mental flowchart to decide which techniques to try, instead of mindlessly trying them all. I imagine this flowchart is a function of a) number of predictors; b) variable types; c) domain knowledge about possible relationships (linear/non-linear); d) size of the dataset; e) constraints around computation time and so on.

Is there any such agreed on, conventional flowchart to follow, to choose the techniques? Or does it really boil down to "try lots of things and see what works best on the desired measure eg. cross-validation"?


texts: Is stratified sampling necessary (random forest, Python)?

texts:
I use Python to run a random forest model on my imbalanced dataset (the target variable was a binary class). When splitting the training and testing dataset, I struggled whether to used stratified sampling (like the code shown) or not. So far, I observed in my project that the stratified case would lead to a higher model performance. But I think if I will use my model to predict the new cases which would highly probably differ in the distribution of target class with my current dataset. So I inclined to loosen this constrain and use the unstratified split.
Could anyone can advice to clarify this point?

train,test=train_test_split(myDataset, test_size=0.25, stratify=y)


texts: After the training phase, is it better to run neural networks on a GPU or CPU?

texts:
My understanding is that GPUs are more efficient for running neural nets, but someone recently suggested GPUs are only needed for the training phase and that once trained, it's actually more efficient to run them on CPUs.

Is this true?


texts: 2 numerical inputs, 1 numerical output, which model should I use?

texts:
I'm aware that this question might be too ambiguous to be welcome, but this is the only place I can ask.
Well I'm thinking about this over and over to no avail.
I got 2 numerical inputs, which are start_time and end_time, and 1 numerical output is the 9-digit or 10-digit number expressed in barcode. Just like the picture I drew below.

In this case, what should be my model?
Maybe the model itself would be simple NN model, but with how many hidden neurons? and with how many layers?
Do you think I should check many possibilities by checking validation loss during training?
Then GreedySearch or Bayesian hyperparameter tuning must be needed?
Any, if any advice would be appreciated. Thanks.


texts: Churn rate prediction based on sequencial data

texts:
I am trying to build a machine learning model that can predict if a certain user will churn based on its historical static and dynamic data. The data looks like below:

1) timestamp, user1, user info (static), event info(dynamic), 0
2) timestamp, user1, user info (static), event info(dynamic), 0
3) timestamp, user1, user info (static), event info(dynamic), 1

4) timestamp, user2, user info (static), event info(dynamic), 0
5) timestamp, user2, user info (static), event info(dynamic), 0
6) timestamp, user2, user info (static), event info(dynamic), 0
7) timestamp, user2, user info (static), event info(dynamic), 0
8) timestamp, user2, user info (static), event info(dynamic), 0

There are many user_id in the dataset and each could be a varied-length sequence. The features for each user are two parts. One is user info, which can be regarded as constant for each user. The other part is the event info, which includes features that change over time.

Please let me know what should be my approach to handle such a problem either in machine learning/deep learning, it is better to have a detailed step by step tutorial in pytorch.


texts: How to use binary numbers and plain floats in the same neural network input for reinforcement learning?

texts:
I'm modeling a system whose configuration can be represented by a binary array ([1,0,0,1] or [0,1,0,0], for example), and the agent can move on a 2D space (thus having 3 DoFs), and the action the agent takes depends on the position (it has to follow a predetermined trajectory).
What I did was: input the configuration concatenated with the position and angle, like this:
input = [1,0,0,1,0.57,-0.2,1.2],

and the output (action) is a 4D vector, which will be one-hot encoded
according to the max value in the output (therefore, if the output has its maximum value as the 2nd element, the action itself will be [0,1,0,0]).
It doesn't learn, and one of my suspicions is that the configuration is normalized ([1,0,0,1] is between 0 and 1), but I can't normalize position data, because it would basically render the trajectory information useless.
Is there another way to do this, am I doing it wrong, or the problem is somewhere else?


texts: Identifying poker card value and suit using iOS live-capture

texts:
I'm working on a hobby project that requires me to be able to use a front facing camera to effectively scan a poker card, identifying the value and suit, using live-capture. Think dealing cards to players, and hovering the card over the camera. The card would be very close up (as to not show the value to other players), so only a portion of the card might be seen. Typically, the top left or bottom right of most playing cards has all the info you'd need, Value over Suit.

I've tried CreateML, turi-create OSOD, object detection with annotated localization coordinates, etc. But I'm never really getting the accuracy I'm hoping for. Leaving me to think that this might be a little too much for ML, at least for live-capture. I've also tried scanning an entire deck, using the full card face, as well as using only the corner.

Any thoughts, tips, ideas, recommendations, etc would be extremely appreciated.


texts: Is it common to preprocess image data before sending it through a deep net?

texts: I'm curious as how convolutional neural network are used in practice for object recognition. Is it common to perform data preprocessing before providing the data to the input layer ? If so, what types of preprocessing are common ?


texts: Max pooling has no parameters and therefore doesn't affect the backpropagation?

texts: I feel this is a question that has a lot of variations already posted but it doesn't exactly answer my question. I understand the concept of max pooling and also the concept of backpropagation. What i cannot understand is that why is it said that max pooling doesn't affect the process of backpropagation? Is it because max-pooling "doesn't have any parameters"? Also is this statement - max pooling has no parameters- right? Why or why not?


texts: K.gradients gives type error where both arguments are tensors

texts:
On lines 142 and 143 of: https://github.com/nyck33/openai_spinup_my_implements/blob/master/continuous/mountaincar/my_ddpg_ac.py

I have:

self.get_action_gradients = K.function(inputs=[self.model.input[0], self.model.input[1], \
            K.learning_phase()], outputs=[action_gradients])

Which tells me:
line 143, in build_model
K.learning_phase()], outputs=[action_gradients])

TypeError: Can not convert a list into a Tensor or Operation.

action_gradients are calculated on line 140 via:

action_gradients = K.gradients(Q_value, actions)

so I did not think that is a problem but when I take the brackets off of the output argument for K.function like so:

self.get_action_gradients = K.function(inputs=[*self.model.input, \
            K.learning_phase()], outputs=action_gradients)

Now I get a slightly different error mentioning Nonetype rather than list:

rning-copied/_1my_imps/continuous/mountaincar/my_ddpg_ac.py", line 143, in build_model

TypeError: Can not convert a NoneType into a Tensor or Operation.

Printing out the Q_values and actions show they are tensors:

Q_values Tensor("q_values/BiasAdd:0", shape=(?, 1), dtype=float32) actions Tensor("actions:0", shape=(?, 1), dtype=float32)

But printing out the action_gradients and type(action_gradients) just confuses me more:

action_gradients [None]
action_gradients type <class 'list'>

calling:  K.gradients() on two tensors should work shouldn't it?

This DDPG code is originally from here:  https://github.com/nyck33/autonomous_quadcopter

and I am trying to adapt it for MountainCarContinuous-v0


texts: How to use Kaggle Api in Google Colab for directly using dataset?

texts:
I know that we can use Kaggle's api directly in google colab which downloads the dataset. The commands are :

!mkdir .kaggle
!echo '{"username":"somename","key":"apikey"}' > /root/.kaggle/kaggle.json
!chmod 600 /root/.kaggle/kaggle.json
!kaggle competitions download -c bluebook-for-bulldozers -p /content

But I need to do this process of making .kaggle file and pass the apikey in google colab gpu everytime. And sometimes the echo command run saying no file called .kaggle but after say 2 mins without restarting the kernal, it works. It sounds funny but yes this is true. Can't I just make a .kaggle file(only once) and use these commands just once and download the dataset just once and store it somewhere and use it even for later purposes. I have used google drive mount process but it's hectic and takes lot of time uploading the datasets in drive. It would also be ok if I need to download the dataset everytime using just this command rather than making .kaggle file and writing the api key and username every time in it:

    !kaggle competitions download -c bluebook-for-bulldozers -p /content

The success rates of previous commands are pretty less in one go and wastes lot of time.


texts: Keras: do `sample weights` take part in the derivatives

texts:
According to Keras documentation, sample_weight can be used in order to give any sample in the training data a different importance in the loss.

I have googled around but have not found the answer to my question as follows:

Do sample weights take part in the derivatives? In other words, does Keras use these weights to train the model or they just give rise to a different loss value? i.e., the derivatives are "still" computed w.r.t. to the "unweighted" loss?

Because the loss function is not actually defined based on the sample weights, its rather passed in (as an argument) to the fit function in Keras.


texts: Is it a good practice to evaluate model performance by comparing the metrics of rescaled (inverse transformed) predictions and true target values?

texts:
I am now working with a Linear Regression for a time-series regression problem (I am sorry that I cannot say too much about the problem and feature vector due to NDA).
I scaled both the input values and target (X and y) with scikit-learn StandardScaler().
The reasoning of scaling both X and y because since it is a time series regression problem, we are using the previous y value (target value) as the feature (X) for the next datapoint.
Thus, there is a relation between y and X.
So, I thought that in order to maintain the dependence of X and y (the previous target value), I need to scale them both.
After training the model, I got a low RMSE on my validation and test sets. Note that, the validation and test sets contain the data with scaled/normalized X and y values.
However, when I tried to rescale (inverse_transform) the predictions, and evaluate the RMSE of the predictions against the true target value of the test set (the true target value is already rescaled as well), I got a high RMSE error.
My question is, is it a good practice to evaluate RMSE/accuracy of rescaled (inverse transformed) predictions against the true target values?


texts: Using classification of previous sample in neural networks

texts:
I am trying to classify the state of a machine using different features coming from a set of sensors.

I am treating the problem like a time series, so I windowed the stream of the sensors each X seconds so that every sample has size ( X * s_f ).

I am obtaining fairly good results using a CNN but I would like to increase performances using some kind of approach that understands the patterns between different states.

For example if going from state A to B is more frequent than A->C I want the algorithm to include this in the classification.

My idea is to classify each sample taking into account the classification of the previous sample. How can I do this?


texts:
I'm working on a classification problem. I'm trying to build a model which can predict if a bank client will get a loan or not. Some of clients have co-borrower and the majority don't.

I also have information on co-borrowers like salary, etc. but as the majority of clients don't have co-borrower, I have missing values.
How can I impute this missing data ?

thank you


texts: Application of ideas from graph theory in machine learning

texts: I work with neural networks (ConvNNs, DeepNNs, RNNs/LSTMs) for image segmentation and recognition and Genetic Algorithms for some optimization problems. Recently I started to learn some deep graph theory ideas (random graphs, chromatic numbers, graph coloring). I'm familiar with combinatorics at undergrad level. Are there any existing interesting applications and areas of research of graph theory and combinatorics in ML?


texts: Voting classifier using grid search for Time Series

texts:
I have three models:

Arima
Auto ARIMA
Double Exponential Smoothing

I would like to apply an ensemble method - a voting method and allow the classifier to learn weights for these three models.

I have checked the votingclassifier present in scikit learn. It requires:
fit(x,y) to run.

Time series object that is present in series object don't have y.

How do you apply a voting classifier and learn weights through grid search?


texts: Using the Stanford Named Entity Tagger in R

texts: I am experimenting with the Stanford Named Entity Tagger  here http://nlp.stanford.edu:8080/ner/process and I feel it would be useful in my research. Does anyone know of a example that I could follow so that I could do the analysis in R? Ideally I'd want to provide a string and get back a count (as a list) of the number of organisations, persons, etc recognised in the string. Thanks.


texts: how to use the hierarchical coefficients?

texts:
I am trying to understand whether I can use hierarchical coefficients obtained on different methods.
I got agglomerative coefficients for methods like: "single", "complete", "ward", "average". I also got a divise coefficient as well.
Questions:

I believe the higher the cluster coefficient the better? And that leads to picking the best agglomerative coefficient with the best method? I got the highest coefficient for ward method.

Is the divisive coefficient comparable with agglomerative coefficient? agglomerative coefficient gave me 0.77 and divisive coefficient gave me the 0.74. I picked the agglerative coefficient - ward method as I this has found a better as the amount of clustering found is higher compared to the divisive one.

Does someone has an idea about this ?


texts: How to cluster text-based software requirements

texts:
I'm beginner in deep learning and I'd like to cluster text-based software requirements by themes (words similarities/frequency of words) using neural networks. Is there any example/tutorial/github code of unsupervised neural network that groups texts based on themes and words similarities?

Thank you very much for your answers!


texts: k-means cluster analysis for pitch curves in r

texts:
so I have many pitch curves time normalized.
I though of using k-means as tool for automatically cluster these.
I have the hypothesis that they correlate to three patterns but I
want to go for an exploratory way.

This is (a piece of) my data frame

time            1       2       3       4       5...
 1             155     139.5   NA      124     148
 2             NA      139.3   NA      127     156.1
 3             120.6   139.5   131.9   130     159.6
 4             117     139     133.9   135     169.5
 5             NA      138.3   138.7   137.4   169.1
 6             120     138.9   137     139.8   168.4
 7             NA      139.5   135.2   142.1   172.9
 8             119     137.1   135.9   143     176.3
...

Can you help me in how to actually run the k-means clustering for the pitch curves?


texts: regression by grouping the dependent variable

texts:
I have a large dataset exploring the effects of the independent variables on the dependent variable using Poisson regression since the dependent variable is a count variable.
However, the range of the dependent variable is too large. Hence, I was thinking of grouping the dependent variable, like low, medium and high values, and then use the Poisson regression for each group. My question is, does this makes sense? I mean, grouping the dependent variable and then running the tests separately for each group.
One question may be what is a low, medium and high value. I noticed some clustering algorithms and I (have used) and will use one of them(k-means seems to be fine) to divide the dependent variable into three groups.
My initial analysis shows that the effects of independent variables are different for each group, but I am not sure if this is the correct way of doing the analysis.
Can you please comment?
Thank you,
EDIT:
There are 454270 samples is the dataset and the range of the dependent variable is between 0 and 160 (with mean 1.293136 and std 2.681311) and 90% of the values are in the range between 0 and 3. The histogram of the variable is as follows:

A more detailed histogram of the dependent variable and two scatter plots of the IVs and the DV is as follows (there are also other independent variables but they are control variables)


texts: How do I calculate the partial derivative with respect to x?

texts:
I am trying to implement CNN using python Numpy.
I searched so much, but all I found was for one filter with one channel for Convolution.

Suppose we have an X as Image with this shape: (N_Height, N_Width, N_Channel) = (5,5,3)

And Let's say I have 16 filters with this shape: (F_Height, F_Width, N_Channel) = (3,3,3) , stride=1 and padding=0

Forward:

Output shape after conv2d will be

(
math.floor((N_Height - F_Height + 2*padding)/stride + 1 )),
math.floor((N_Width- F_Width + 2*padding)/stride + 1 )),
filter_count
)

So the output of this layer will be an array with this shape: (Height, Width, Channel) =  (3, 3, 16)

BackPropagation:

Suppose $dL/dh$ is the input for my layer in backpropagation with this shape: (3,3,16)

Now I must find $dL/dw$ and $dL/dx$: $dL/dw$ to update my filters params and $dL/dx$ to pass it as input to the previous layer as Loss respect to the input X.

From this answer Error respect to filters weights I found how to calculate $dL/dw$.

The problem I have in BackPropagation is I don't know how to calculate $dL/dx$ having this shape:(5,5,3) and pass it to the prev layer.

I read lots of articles in Medium and other sites but I don't get how to calculate it:

How Backpropagation works in a CNN
The best explanation of Convolutional Neural Networks on the Internet!
Backpropagation In Convolutional Neural Networks
How to propagate error back to previous layer in CNN?

Thanks in advance :)


texts: Value extraction from a python dataframe [ problem statement specific ]

texts:
This is an update on one of the older posts of a friend of mine (That post stopped getting responses as I got a partial solution to my problem) and I'm reworking that problem statement. Link to that post: Coding Problem - Extracting values from a column and forming a new dataframe [edited]

While extracting the weather parameters from the 'hourly' column entries, It only extracts the first value of a particular weather parameter. Each entry in the 'hourly' column is a list of weather parameters. Each parameter has 24 different values denoting the weather parameter value for 24 hours in a particular day. So, the code I used gives me 31 different entries in the new data frame while it should have been 31*24 entries for the month of January. Can you guys please help me out ?

Notebook link (edited) is provided here.

data set link is presented here.

[code]

import json
from pprint import pprint
import pandas as pd

with open(r'C:\Users\Murali\Desktop\Downloads\2016-1.json') as f:
    s = f.read()
jansixteen = json.loads(s)
#print(jansixteen)

a  =jansixteen['data']['weather']
b = ((pd.DataFrame(a)))
b.head()

def extract_col_as_df(df, column_name):
    data = [datum[0] for datum in df[column_name]]
    df = pd.DataFrame(data = data)
    return df
df_hourly = extract_col_as_df(b, 'hourly')
    print(df_hourly)

df_hourly.drop(['FeelsLikeF', 'HeatIndexF','HeatIndexC','weatherIconUrl','WindGustMiles','weatherDesc','weatherCode'], axis=1)


texts: Predicting contract churn/cancellation: Great model results does not work in the real world

texts:
I'm busy with a supervised machine learning problem where I am predicting contract cancellation. Although a lengthy question, I do hope someone will take the time as I'm convinced it will help others out there (I've just been unable to find ANY solutions that have helped me)

I have the following two datasets:

1) "Modelling Dataset"

Contains about 400k contracts (rows) with 300 features and a single label (0 = "Not Cancelled", 1 = "Cancelled").

Each row represents a single contract, and each contract is only represented once in the data. There are 350k "Not Cancelled" and 50k "Cancelled" cases.

Features are all extracted as at a specific date for each contract. This date is referred to as the "Effective Date". For "Cancelled" contracts, the "Effective Date" is the date of cancellation. For "Not Cancelled" contracts, the "Effective Date" is a date say 6 months ago. This will be explained in a moment.

2) "Live Dataset"

Contains 300k contracts (rows) with the same list of 300 features. All these contracts are "Not Cancelled" of course, as we want to predict which of them will cancel. These contracts were followed for a period of 2 months, and I then added a Label to this data to indicate whether it actually ended up cancelling in those two months: 0 = "Not Cancelled", 1 = "Cancelled"

The problem:

I get amazing results on the "Modelling Dataset" (random train/test split) (eg Precision 95%, AUC 0.98), but as soon as that model is applied to the "Live Dataset", it performs poorly (cannot predict well which contracts ends up cancelling) (eg Precision 50%, AUC 0.7).

On the Modelling Dataset, the results are great, almost irrespective of model or data preparation. I test a number of models (E.g. SkLearn random forest, Keras neural network, Microsoft GbmLight, SkLearn Recursive feature elimination). Even with default settings, the models generally perform well. I've standardized features. I've binned features to attempt improving how well it will generalize. Nothing has help it generalize to the "Live Dataset"

My suspicion:

In my mind, this is not an over-training issue because I've got a test set within the "Modelling Dataset" and those results are great on the test set. It is not a modelling or even a hyper-parameter optimization issue, as the results are already great.

I've also investigated whether there are significant differences in the profile of the features between the two datasets by looking at histograms, feature-by-feature. Nothing is worryingly different.

I suspect the issue lies therein that the same contracts that are marked as "Not Cancelled" in the "Modelling Dataset", which the model trains to recognize "Not Cancelled" of course, is basically the exact same contracts in the "Live Dataset", except that 6 months have now passed.

I suspect that the features for the "Not Cancelled" cases has not changed enough to now make the model recognize some of them as about to be "Cancelled". In other words, the contracts have not moved enough in the feature space.

My questions:

Firstly, does my suspicion sound correct?

Secondly, if I've stated the problem to be solved incorrectly, how would I then set up the problem-statement if the purpose is to predict cancellation of something like contracts (when the data on which you train will almost certainly contain the data one which you want to predict)?

For the record, the problem-statement I've used here is similar to the way others have done this. And they reported great results. But I'm not sure that the models were ever tested in real live. In other cases, the problem to be solved was lightly different, e.g. hotel booking cancellations, which is different because there a stream of new incoming bookings and booking duration is relatively short, so no bookings in common between the modelling and live dataset. Contracts on the other hand have long duration and can cancel at any time, and sometimes never.


texts: Determining position of anchor boxes in original image using downsampled feature map

texts: From what I have read, I understand that methods used in faster-RCNN and SSD involve generating a set of anchor boxes. We first downsample the training image using a CNN and for every pixel in the downsampled feature map (which will form the center for our anchor boxes) we project it back onto the training image. We then draw the anchor boxes centered around that pixel using our pre-determined scales and ratios. What I dont understand is why dont we directly assume the centers of our anchor boxes on the training image with a suitable stride and use the CNN to only output the classification and regression values. What are we gaining by using the CNN to determine the centers of our anchor boxes which are ultimately going to be distributed evenly on the training image ?


texts: I understood that gradient descent is needed to find the local extremum of any function, but how is it applied to the linear classifier (single matrix, two classes case, for example)? How does it step by step modify the elements of this matrix?


texts: Is it possible to pull a complete list of all IPv4 addresses and put them into a text file? Since there are 4,294,967,296 IPv4 addresses and each one takes about 4 bytes I would guess the file would be about 17.179869184 GB. Is it possible to download all of these addresses into one .txt file?


texts: Classification accuracy based on top 3 most likely classifications

texts:
My goal is to recommend jobs to job seekers based on their skill set.

Currently I'm using an SVM for this, which is outputting one prediction, e.g. "software engineer at Microsoft". However, consider this: how significantly different are the skill sets of a software engineer at Microsoft and a software engineer at IBM? Probably not significantly different. Indeed, by inspection of my data set I can confirm this. Hence, the SVM struggles to discriminate in situations like this, of which there are many in my data set, and my classification accuracy is about 50%.

So I had an idea.

In SK Learn, once you've trained some model, you can compute the probability a particular input X belongs to each class.

So for each input X in my test set, I took the the top 3 most likely classifications. Then I tested whether or not the correct label was in the top 3 predictions. If it was, then I considered the prediction to be correct. In doing so, the classification accuracy increased to over 80%.

So my question is: is this a valid approach to measuring classification accuracy? If it is, then does it have a name?

In my mind, it is valid given my intended application, which is to recommend a selection of jobs to a job seeker, which are relevant to their skill set.

Cross posted from CS SE:
https://cs.stackexchange.com/questions/117695/classification-accuracy-based-on-top-3-most-likely-classifications

I'm interested to know what perspective data scientists have on this.


texts:
I am working in python3 cleaning data. I have a large number of midi files scraped from a variety of sources using beautiful soup.  Many of the files may be duplicate musical pieces. I can change the the keys of the midis so that they are the same and change the instrument to piano (they are monophonic files).

So, then it should be possible to check if the song content (in terms of the midi encoding) is similar. It is particularly pressing as some of the files only have numbers as names on them. So, to be clear, it is the content of the files that I have to check for duplication. Also, I am not looking for exact matches I am just looking for percent similarity.

My current approach is to use SequenceMatcher from Difflib. I am checking if a buffer of 600  on   each seperate file has a SequenceMatcher ratio >.9 Then, I am flagging them so I can compare the duplicates by listening to them.

This approach works in test scenarios with songs in midi format. However, it is extremely slow. So, I am wondering if there is anyone who has faced this issue before. Can anyone offer any insight on optimizing this algorithm for comparing the files or paralleling the code to gain speed or any other approach aside from hashing (I am not looking for exact file matches). Any help would be appreciated.

Here is my code:

import sys
import os
import hashlib
from difflib import SequenceMatcher

def similar(a, b):
    return SequenceMatcher(None, a, b).ratio()

count=0
for dirName, subdirs, fileList in os.walk('path to my directory of files'):
    print('Scanning %s...' % dirName)
    for filename in fileList:
        path = os.path.join(dirName, filename)
        in_file = open(path, 'rb')  # Provide a path to disk or ISO image
        data = in_file.read(600)
        in_file.close()
        for filename2 in fileList:
            path = os.path.join(dirName, filename2)
            in_file2 = open(path, 'rb')
            data2 = in_file2.read(600)
            in_file2.close()
            if filename==filename2:
                pass
            else:
                s=similar(data,data2)
                if s>0.9:
                    print(filename +filename2+" "+str(s))

`


texts: Hyper-parameters for character-level RNN models

texts:
I am try to recreate Karpathy's great tutorial on RNN character-level text generation using TensorFlow - Promise to contribute the code back to the community once it works. Meanwhile, my code runs, but the model does not converge. It stays stuck on the same level of error and emits rubbish. I suspect that the code is correct but my hyper-parameters are off.

If someone previously trained a text-generating RNN successfully, I would be glad to get some hints on the parameters below. The input I use currently is NLTK Bible corpus, about 4MB of text, but I can use something else.

number of steps, i.e. sequence length
batch size
number of hidden units
number of layers
Optimizer and optimization parameters - I'm using Adagrad with decaying learning rate, trying to mimick the original script, but it is not a must
Any other tricks for the network architecture or optimization
Most important, the number of epochs (full scans of the data) needed before the model start emitting something readable


texts: XGBoost probability distribution tending towards the extreme

texts:
I am using an XGBoost classifier to make risk predictions, and I see that even if it has very good binary classification results, the probability outputs are mainly under $0.05$ or over $0.95$ (like 60% of them).

I have tried calibration methods (from the sklearn API) but it reduces the problem only slightly.

My dataset has 1800 training points and I test it on around 500 datapoints. It is quite balanced. I also use Bayesian optimisation to tune the hyperparameters of the model. There are 19 features for my model.

Does anyone know a solution to get more regularly distributed probabilities? Does the problem dwell in the fact I have too few datapoints? Should I set my hyperparameters differently? Do I have too many/few features?


texts: computational time and precision trade off

texts: I know that there are a number of predictive models (generized linear ones, trees, neural network, support vector machines, knn, Naive Bayes, ...) that have been proposed to perform various analytical tasks. Now I am striving to find appropriate references about their performance when the data becames "Big". In other words, how is their performance when the data becames really big. Does the training time increase more than linear? Is there any comparative benchmark between computational time and precision when the data becames high (for the various predictive models).


texts: How to obtain causal inference using double machine learning?

texts: How can Double machine learning be used for causal inference in treatment effects? What is the intuitive working of Double machine learning?


texts: How to assign a weight for classifiers when using weighted majority voting?

texts:
I am trying to apply weighted majority voting on an ensemble as a combiner method. I read different papers and articles, however, I am still a bit lost on:

How the weighted majority voting works

How to assign a weight for every ensemble base classifiers when using weighted majority voting.


texts: Training with data from multiple tfrecords files simultaneously

texts:
I am training a neural network in TensorFlow using several distinct groups of data, lets call them Data A, Data B and Data C. I want to have them mixed during training, ie, I do not want to train on A, then B, then C, but instead I want to use all the data simultaneously. I wish to do that using .tfrecords files.
I know that this can be done by shuffling the data before training starts, that is, construct the .tfrecords files so that they already contained mixed data. However, I wonder if something else is possible:
The most convenient way I can think of achieving what I want is to split the data into the files Data_A.tfrecords, Data_B.tfrecords, Data_C.tfrecords, and then read from them simultaneously during training. My question is, is this possible? If it is, can you give a very rough explanation of how that's possible?


texts: Node values in Boltzmann machines (0/1 vs -1/1). Are they the same?

texts:
Boltzmann machines were introduced by Hinton and Sejnowski as taking values in $\{0,1\}$. The Wikipedia entry also uses this convention. However, Hopfield Networks, which are the deterministic version of Boltzmann machines, are usually introduced as taking values in $\{-1,1\}$. Ising models also follow this convention.

With the energy function being defined equivalently in both models as
$$ E(x) = \sum_i b_ix_i + \sum_{i<j}w_{ij}x_ix_j$$
it seems that the two conventions would behave differently. For example, how would the $\{0,1\}$ model learn a preference for checkered patterns?

More generally, in an undirected graphical model where the nodes take values in $\{a,b\}$, we can define the interaction (energy) between two neighbouring nodes $x$ and $y$ as
$$E(x,y) = \begin{cases}
      w_{aa}, & \text{if}\ x=a,y=a \\
      w_{ab}, & \text{if}\ x=a,y=b \\
      w_{ba}, & \text{if}\ x=b,y=a \\
      w_{bb}, & \text{if}\ x=b,y=b
    \end{cases}$$
or equivalently represented by the matrix
$
\begin{bmatrix}
    w_{aa}       & w_{ab}  \\
    w_{ba}       & w_{bb}
\end{bmatrix}
$.

Boltzmann machines restrain the interactions between two neighbouring nodes to being described by a single scalar $w_{ij}$. In the case when our values $\{a,b\}$ are $\{0,1\}$, we get
$$ E(x_i,x_j) =
\begin{bmatrix}
    w_{ij}       & 0  \\
    0       & 0
\end{bmatrix}
$$
If we set $\{a,b\}$ to $\{1,-1\}$, we instead get
$$ E(x_i,x_j) =
\begin{bmatrix}
    w_{ij}       & -w_{ij}  \\
    -w_{ij}       & w_{ij}
\end{bmatrix}
$$

Are these two formalisms really equivalent? It seems unlikely...


texts: Coefficient of determination of linear regression in Azure ML studio differs significantly from regression R square in Orange 3 for same dataset why?

texts: I tried exactly same dataset in Orange 3 and Azure ML Studio for linear regression. Coefficient of determination in Azure ML studio is .71 while Orange gives R square as .45. Why such a huge difference?


texts: Efficient solution of fmincg without providing gradient?

texts:
I'm working on multiclass logistic regression model with a large number of features (numFeatures > 100). Using a Maximum Likelihood Estimation based on the cost function and gradient, the fmincg algorithm solves the problem quickly. However, I'm also experimenting with a different cost function and do not have a gradient.

Is there a good way to speed up the calculation process? E.g., is there a different algorithm or fmincg setting that I can use?


texts: Which is better for Data Science, a double major in Math & CS or Physics & CS?

texts:
My dream occupation is to work as a Data Scientist.

I'm soon going to be completing my first semester as a physics student, as part of an incredibly prestigious physics program. We get to run an experiment at CERN in our third year! I'm currently ranked highly in this program (top 25%), and I could push myself to top 10% if needed.

Thing is, I'm not really interested in working as a physicist.

Should I switch to Math & CS? (I'd be in a mediocre program.)

Note - I'm currently in both physics and CS, not just physics. I'm asking if I should switch the physics major with a math one.


texts: Choosing the right data mining method to find the effect of each parameter over the target

texts:
I am dealing with a lot of categorical data right now and I would like to use an appropriate data mining method in any tool [preferably R] to find the effect of each parameter [categorical parameters] over my target variable. To give a brief notion about the data that am dealing with, my target variable denotes the product type [say, disposables and non-disposables] and I have parameters like root cause,symptom,customer name, product name etc. As my target can be considered as a binary value, I tried to find the combination of values leading to the desired categories using Apriori but, I have more than 2 categories in that attribute and I want to use all of them and find the effect of the mentioned parameters over each category. I really wanted to try SVM and use hyperplanes to separate the content and get n-dimensional view. But, I do not have enough knowledge to validate the technique, functions am using to do the analysis. Currently I have like 9000 records and each of them represents a complaint from the user. There are lot of columns available in the dataset which is what I am trying to use to determine the target variable [ myForumla <- Target~. ] I tried with just 4 categorical columns too. Not getting a proper result.

Can just the categorical variables be used to develop a SVM model and get visualization with n hyper planes? Is there any appropriate data mining technique available for dealing with just the categorical data?


texts:
I am building an anomaly detection model using keras upon videos. There are total 179 frames. The original dimension of each frame is given below:
h,w,c=cv2.imread(FramesFile[178]).shape  #h=240, w=30, c=3

Then, I have applied the ResNet-50 on these video frames to extract features, it gives me (1,7,7,2048) features. And to reduce down these features I have applied conv2D layer on it to get (1, 7, 7, 512) features as shown in 2nd line of function of FramesTrain()
Here's my code for training. I have done it through convolutional LSTM
def FramesTrain():
 seq = Sequential()
 seq.add(Conv2D(512, (5, 5), padding="same"))
 seq.add(ConvLSTM2D(filters=40, kernel_size=(5, 5), input_shape=(None, 40, 40, 1), padding='same', return_sequences=True))
 seq.add(BatchNormalization())
 seq.add(ConvLSTM2D(filters=40, kernel_size=(5, 5), padding='same', return_sequences=True))
 seq.add(BatchNormalization())
 seq.add(ConvLSTM2D(filters=40, kernel_size=(5, 5), padding='same', return_sequences=True))
 seq.add(BatchNormalization())
 seq.add(ConvLSTM2D(filters=40, kernel_size=(5, 5), padding='same', return_sequences=True))
 seq.add(BatchNormalization())
 seq.add(Conv3D(filters=1, kernel_size=(5, 5, 5), activation='sigmoid', padding='same',
       data_format='channels_last'))
 seq.compile(loss='binary_crossentropy', optimizer='adadelta')
 return seq

 model=FramesTrain()
 model.fit(FramesFeatures[0], batch_size=32,epochs=10, verbose=1)

But it gives me this error:
ValueError: Input 0 of layer conv_lst_m2d_60 is incompatible with the layer: expected ndim=5, found ndim=4. Full shape received: (None, 7, 7, 512)
Kindly guide me how do I solve it.
Regards,


texts: What happens if I do not encode the lables or classifiers in the data?

texts: I have a data where the three variables are numerical and one variable is a string. I am using the simple decision tree algorithm. Read somewhere that the data in strings must be encoded with one hot encoding. Will this affect the accuracy of my system ?


texts: How to prevent a neural network from outputing too clustered probabilities?

texts:
I have a CNN outputting probabilities using a logistic output. The performances are good on the test set. Yet the probabilities it outputs are very clustered, they are either 0 or 1!

I would like some more granularity in the probabilities outputed by the network. Some features are more obvious than other and the network should be able to learn this. Any idea on how I could solve this problem?


texts: Ensemble technique for combining predictions from classification and regression algorithms

texts:
Given an anomaly detection problem - A, I have divided the problem into two independent subtasks -A1 and A2 and tackled the tasks using classification and regression algorithms respectively. In order to combine the predictions from the classification and regression algorithms, I have derived an average weighing formula. My question is if this approach can be regarded as ensemble learning? Otherwise, are there better ways to combine such predictions into a common score? I am new here :)

UPDATE
Subtasks A1 and A2 are tackled with One Class SVM (unsupervised) and Gaussian  Process Regression (GPR)(supervised) respectively. The scores from both algorithms are added using weighed mean like so:

Final_Score = alpha * OCSVM_Score + (1 - alpha) * GPR_Score

where alpha = 0.5


texts:
The question

How do I predict the rating for a new user in an ALS model trained in  Spark? (New = not seen during training time)

The problem

I'm following the official Spark ALS tutorial here:

http://ampcamp.berkeley.edu/big-data-mini-course/movie-recommendation-with-mllib.html

I'm able to build a good recommender with a decent MSE but I'm struggling with how to input new data to the model. The tutorial changes the first user's ratings prior to training, but this is really a hack. They give the following hint:

  9.2. Augmenting matrix factors:

  In this tutorial, we add your ratings to the training set. A better way to get the recommendations for you
  is training a matrix factorization model first and then augmenting the
  model using your ratings. If this sounds interesting to you, you can
  take a look at the implementation of MatrixFactorizationModel and see
  how to update the model for new users and new movies.

The implementation does not help me at all though. Ideally, I'm looking for something like:

predictions = model.predictAllNew(newinput)

But not such method exists. I could go and modify the original RDD, but I think that would require me to retrain the model, so that wouldn't be an ideal solution either. Surely there must be a more elegant way?

Where I am right now:

I think I need to find the latent representation of the new vector. According to the original paper we can compute this like so:

$X_u = (Y^T C^u Y + \lambda I)^{-1} Y^T C^u p(u)$

But when I calculate using the values in the paper, it doesn't match the values from the model. I fix alpha and the regularization parameter, but I think the MLLIB implentation has a different $C^u$ implementation. It's defined here (see line 1304), but not being adept at Scala, this is very hard to reverse engineer for me ...

My current attempt:

V = model.productFeatures().map(lambda x: (x[1])).collect() #product latent matrix Y

Cui =  alpha * np.abs(newinput)
Cui =  (1. + Cui) / (Cui)
Cui[np.where(newinput == 0)] = 0
Cui = np.diag(Cui)

lambdaI = len(np.where(newinput!=0)) * regularization_parameter * np.eye(np.shape(V)[1]) #
term   = np.dot(np.dot(Vt,Cui),V)+lambdaI
term   = np.dot(np.linalg.inv(term),Vt)
term   = np.dot(term,Cui)
term   = np.dot(term,newinput)
latentinput = term

But this doesn't match.


texts: How to concatenate many .psv files in google collaboratory?

texts:
I have a folder named 'training' in my local drive which has 20000 .psv files. I zipped it and uploaded to google collaboratory, with the upload option in the Files
section. I unzipped it with the following command.

!unzip training

Now I have a folder called training in Files. Each file contains 40 columns which are same for all the files and rows of different lengths.I wish to merge all the files. The resulting file should contain all the rows of all the files with 40 columns(ignore the index columnas to avoid duplicate index). The header should have column names since they are common to all the files.
The merged file should be converted to a single data frame.
Thanks in advance.


texts: Is the decision tree the right choice to classify for this dataset?

texts:
I have a bunch of tuples like this; [SourceIP, DestinationIP, Port, TimeStamp]

If a destination IP recorded 21, 22, 23 and 80 port (set of 5 tuples) then I will decide something, if it has set of 4 tuples I will decide something if less than 4 I will decide another thing...

I have already handled it with SQL rules but I need to use the decision tree algorithm.

Any idea?

An important thing that I forgot to write is a single tuple has no meaning, I should check timestamps and ports that group of tuple became meaningful...

etc: A set of tuples that is in 5 min and includes ports that 21,22,23,25


texts: Linear Discriminant Analysis (LDA) before or after k-fold cross-validation?

texts:
I have features extracted from a small dataset, would like to reduce the dimensions by using LDA. Also want to do a SVM classification with k-fold cross-validation.

My question is: What would be the best practice: to do LDA before the CV, or to do LDA within the CV (i.e. to each train and test fold)?


texts: I need to find anomalies from this. The term anomaly in my case is in which square id and in which time the total activity goes higher, how can I detect anomalies with the help of machine learning algorithms.


texts: In Tableau, how can I create sub-category fields out of an overarching dimension?

texts:
I have a data set where it's more or less a star schema-like format, but in an Excel file. Don't worry about primary keys setup or any of that unless it significantly matters for answering this question. Here's a sample of the kind of issue I'm facing.
How, from a table like this, would I use Tableau to search by "Cat" and "Dog" as their own fields? In the default Tableau view, they all just show up as "pet," and I can't find any way to seperate the two. I want to set up a chart with quantity of "Cat" on the X-Axis, quantity of "Dog" on the Y-Axis, and then each person graphed as a point relative to how many cats and dogs they have.
No matter what I search, all I find is information about grouping sub-categories,  and I can't find anything about how to create the sub-categories in the first place if they aren't given to you. Could anyone give me a hand?

Name
Pet
Quantity

Sam
Cat
1

Sam
Dog
2

Pat
Bird
3

Bert
Dog
1


texts:
I am trying to figure how the embedding layer works for the pretrained BERT-base model. I am using pytorch and trying to dissect the following model:
import torch
model = torch.hub.load('huggingface/pytorch-transformers', 'model', 'bert-base-uncased')
model.embeddings

This BERT model has 199 different named parameters, of which the first 5 belong to the embedding layer (the first layer)
==== Embedding Layer ====

embeddings.word_embeddings.weight                       (30522, 768)
embeddings.position_embeddings.weight                     (512, 768)
embeddings.token_type_embeddings.weight                     (2, 768)
embeddings.LayerNorm.weight                                   (768,)
embeddings.LayerNorm.bias                                     (768,)

As I understand, the model accepts input in the shape of [Batch, Indices] where Batch is of arbitrary size (usually 32, 64 or whatever) and Indices are the corresponding indices for each word in the tokenized input sentence. Indices has a max length of 512. One input sample might look like this:
[[101, 1996, 4248, 2829, 4419, 14523, 2058, 1996, 13971, 3899, 102]]

This contains only 1 batch and is the tokenized form of the sentence "The quick brown fox jumps over the lazy dog".
The first word_embeddings weight will translate each number in Indices to a vector spanned in 768 dimensions (the embedding dimension).
Now, the position_embeddings weight is used to encode the position of each word in the input sentence. Here I am confused about why this parameter is being learnt? Looking at an alternative implementation of the BERT model, the positional embedding is a static transformation. This also seems to be the conventional way of doing the positional encoding in a transformer model. Looking at the alternative implementation it uses the sine and cosine function to encode interleaved pairs in the input. I tried comparing model.embeddings.position_embeddings.weight and pe, but I cannot see any similarity. The in the last sentence under under A.2 Pre-training Procedure (page 13) the paper states

Then, we train the rest
10% of the steps of sequence of 512 to learn the
positional embeddings.

Why is the positional embedding weight being learnt and not predefined?
The next layer after the positional embedding is the token_type_embeddings. Here I am confused about how the segment label is inferred by the model. If I understand this correctly each input sentence is delimited by the [SEP] token. In the example above there is only 1 [SEP] token and the segment label must be 0 for that sentence. But there could be a maximum of 2 segment labels. If so, will the 2 segments be handled separately or are they processed in parallel all the same as one "array"? How does the model handle multiple sentence segments?
finally the output from theese 3 embeddings are added togheter and passed through layernorm which I understand. But, are the weights in these embedding layers adjusted when fine-tuning the model to a downstream task?


texts: How can I plot a bunch of images, given their centre coordinates?

texts: The points here represents centre coordinates of images with unique names. I want to plot all the images on the same 2d space to get a final representation of all the images merged into sort of a large image.Any ideas on how to do it would be appreciated.


texts: Using a K-NN Classification Approach for Time Series Data?

texts:
I have a dataset which contains time-series data of water flow over time.  I have a flow meter connected to a kitchen faucet, and I am trying to cluster or classify specific water usage events.

The data is collected every second, and in each row I am given a value for the amount of gallons which are flowing through my flow meter.

For example, I am trying to classify someone washing their hands, filling a teapot, cleaning dishes, etc...

Is this something that I can use a k-NN Classification Approach to cluster these events?  If a clustering based approach isn't good, what other method of classified would be good for this type of data?

If I run some experiments, I can classify each event and turn it into a supervised learning problem.  But at the moment, none of the water events are classified.

A very abridged version of my dataset looks like the following:

EDIT

water = pd.DataFrame(shower1)
rng = pd.date_range('2016-09-01 00:00:00', '2016-09-30 23:59:58', freq='S')
water = water.reindex(rng,fill_value=0.0)
water = water['shower1']
df = pd.DataFrame({'time_stamp':rng,'water_amount':water})

starts = (df['water_amount']>0)&(df['water_amount'].shift(1)==0) #find all starts of events
n_events = sum(starts) #total number of events
df.loc[starts,'event_number'] = range(1,n_events+1) #numerate starts from 1 to n
df['event_number'] = df['event_number'].fillna(method='pad').fillna(-1) #forward fill all the values
df.loc[df['water_amount']==0,'event_number']=-1 #set all event numbers to -1 where the water amount is 0

df.groupby('event_number').agg({'time_stamp':'first',
                                    'water_amount':'sum'}) #feature matrix


texts: *(CATEGORIES[int(prediction[0][0])])* giving me different result for single image prediction from saved model

texts:
From a saved model, I am trying to predict a single image.
I followed this code - https://www.youtube.com/watch?v=A4K6D_gx2Iw
I am getting different result for two different command-
*model.predict_classes([prepare('tb2.JPG')])*

and
prediction = model.predict([prepare('tb2.JPG')])
*(CATEGORIES[int(prediction[0][0])])*

My saved model have 2 classes, I have used sigmoid activation function for dense layer, and loss function is 'categorical_crossentropy'
The code is -
CATEGORIES = ["ba",  "to"]

def prepare(filepath):
    IMG_SIZE = 150  # 50 in txt-based

    img_array = cv2.imread(filepath, cv2.IMREAD_GRAYSCALE)
    img_array = img_array / 255.0
    new_array = cv2.resize(img_array, (IMG_SIZE, IMG_SIZE))
    return new_array.reshape(-1, IMG_SIZE, IMG_SIZE, 1)

model = tf.keras.models.load_model("test2.h5", compile=False)

prediction = model.predict([prepare('tb2.JPG')])
print(prediction)
print(CATEGORIES[int(prediction[0][0])])


texts: Multiple-input multiple-output CNN with custom loss function

texts:
I have a set of 2D input arrays $(n\times m)$ namely $A,B,C$ and I have to predict two 2D output arrays namely $d,e$ for which I do have the expected values. You can think of the inputs/outputs as grey images if you like.

Because of the spatial information is relevant (there are actually 2D physical domains) I want to use a Convolutional Neural Network to predict $d$ and $e$. My design (not tested yet) looks as follows:

Because I have multiple inputs, I guess I should use multiple columns (or branches) to find different features for each of the inputs (they look fairly different). Each of these columns follows a encoding-decoding architecture used in segmentation (see SegNet): Conv2D block involves a convolution+batch normalisation+ReLU layer. Deconv2D involves a deconvolution+batch normalisation+ReLU.

Then, I can merge the output of each column by either concatenating, averaging or taking the maximum for example. To obtain the original $m\times n$ shape for each of the outputs I have seen I could do this with a $1\times1$ kernel convolution.

I want to predict the two outputs from that single layer. Is that okay from the network structure point of view? Finally my loss function depends on the outputs themselves compared to the target plus another relation I want to impose.

A would like to have some expert opinion on this since this is my first design of a CNN and I am not sure if I it makes sense as it is now and/or if there are better approaches (or network architectures) to this problem.


texts: Understanding python XGBoost model dump output of a very simple tree

texts:
I am trying to understand the model dump output from XGBoost. I would like to step through and see exactly how the model arrived at it's prediction. To simplify I trained a model with 1 tree and 1 max depth, and as expected all records get one of two predictions as it has a single split - the values are {0.5386398434638977, 0.5011891722679138}. However, when I look at the model dump I see the following

booster[0]:
0:[f40<70.5] yes=1,no=2,missing=1
    1:leaf=0.00475667231
    2:leaf=0.154868156

I have no idea how to interpret this in a way that makes sense with the prediction. What am I missing? Thanks!


texts: Using pipelines with a cross validation of several models in scikit-learn

texts: Is there a simple way to cross-validate several models using sklearn pipelines?


texts: Why can't SPPnet update convolutional weights according to the fast R-CNN paper?

texts:
While discussing the drawbacks of SPPnet in the fast R-CNN paper, the authors state
"But unlike R-CNN, the fine-tuning algorithm proposed in [SPPnet] cannot update the convolutional layers that precede the spatial pyramid pooling."
To me, this seems to imply that SPPnet shouldn't be able to train at all?


texts: Why is it okay to set the bias vector up with zeros, and not the weight matrices?

texts:
We do not initialize weight matrices with zeros because the symmetry isn’t broken during the backward pass, and subsequently in the parameter updating process.
But it is safe to set the bias vector up with zeros, and they are updated accordingly.
Why is it safe to do so, and not the opposite?
Why can’t we initialize bias vectors with random numbers and weight matrices with zeros?
My initial thought is that a vector is of rank (n, 1) where $n \in \mathbb{N}$. This is not true for a matrix. And thus symmetry does not really come into play in the case of vectors.
But that does not answer the question that each layer of a deep neural network has its own weight matrix, and there is no need for symmetry across different layers.
So, when we talk about symmetry are we talking about symmetry across different rows of the same matrix?
Column wise symmetry should not matter much as they are for different training examples (for the first hidden layer). Does column-wise symmetry disturb the training process much in the case of hidden layers other than the first one?


texts: Matrix factorization for like/dislike/unknown data

texts:
Most literature focus on either explicit rating data or implicit (like/unknown) data. Are there any good publications to handle like/dislike/unknown data? That is, in the data matrix there are three values, and I'd like to recommend from unknown entries.

And are there any good open source implementations on this?

Thanks.


texts: what is a good performance measure for comparing different neural network architectures in unsupervised clustering task?

texts:
What is a good measure to use when trying to decide between picking unsupervised clustering NN architectures?
There seems to some ideas here, but i am trying to find out feedback/suggestions from people who've used it before and seen the measures to be meaningful.


texts: Keras one_hot encoding - what's the point when unicity not guaranteed?

texts:
In the following example the text:

  'The quick brown fox jumped over the lazy dog.'

gets the encodings [5, 9, 8, 7, 9, 1, 5, 3, 8].

My code:

In this way the words 'brown' and 'dog' get both the encodings of 8 and 'quick' and 'jumped' 9.

What is the point of encoding if unicity is not guaranteed?

https://machinelearningmastery.com/prepare-text-data-deep-learning-keras/
https://keras.io/preprocessing/text/


texts: How can we identify potential customers for a new list of customers?

texts:
I have two data sets:

Customer demographic data;
Transaction data of the customers.

Now, if I have to identify potential customers to develop a marketing strategy, I would make use of clustering to identify the similar groups and develop a marketing strategy accordingly.
How could I proceed to identify potential customers in a new list of customers not previously seem by the clustering algorithm? Is it like we apply the interpretations we observed from the clusters formed to the new customers??
Need some guidance on this.


texts: Can someone explain the Tensorflow code below?

texts:
I have been trying to understand the below code (written in Tensorflow), but unable to do so as I am not very proficient with Tensorflow. I am looking for some help. Below is the code:
    if model == 'mse':
        rec_loss = tf.losses.mean_squared_error(img, img_rec)
        kld_loss = -tf.reduce_mean(0.5 * (1 + z_log_sigma_sq - z_mu ** 2 - tf.exp(z_log_sigma_sq)))
    else:
        if model == 'gaussian':
            log_sigma = tf.Variable(0.0, trainable=False)
        elif model == 'sigma':
            log_sigma = tf.Variable(0.0, trainable=True)
        elif model == 'optimal':
            log_sigma = tf.log(tf.sqrt(tf.reduce_mean((img - img_rec) ** 2, [0, 1, 2, 3], keepdims=True)))

        rec_loss = tf.reduce_sum(gaussian_nll(img_rec, log_sigma, img))

Where the gaussian_nll function is defined below:
def gaussian_nll(mu, log_sigma, x):
    return 0.5 * ((x - mu) / tf.exp(log_sigma)) ** 2 + log_sigma + 0.5 * np.log(2 * np.pi)

I have these questions:

What is the difference between log_sigma under model == 'gaussian' and under model == 'sigma'? (only difference in the definition of the variable is the flag trainable)
What is the difference between the tf.losses.mean_squared_error under the model=='mse' and log_sigma under model == 'optimal'? Aren't they both similar?

They are correct, but I am unable to correctly read it. Any help regarding the code would be greatly appreciated. For those of you who wants to take a look at the full code on github, here is the link github page (the above piece of code appears between lines 34 and 44)
Thanks once again for your help.


texts: how print f1-score with scikit´s accuracy_score or accuracy of confusion_matrix?

texts:
I would like to print the f1-score. I got confused about the wording f1-accuracy score and accuracy score. What is the difference of these 2 scikit-learn metrics and how can I print the f1-score out of this code?
from xgboost import XGBClassifier
from sklearn.metrics import accuracy_score, classification_report
from sklearn.model_selection import train_test_split
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split

df = pd.read...path
X = df.drop('pricing_class', axis=1)
y = df.pricing_class

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)

xgb_classification = XGBClassifier(random_state=42)
xgb_classification.fit(X_train,y_train)

xgb_results=xgb_classification.predict(X_test)
print(classification_report(xgb_results,y_test))
print('accuracy_score',accuracy_score(xgb_results, y_test))

this is a test sample:

Question 1:
There is under the column f1-score the row of accuracy. Is this the f1-score of the scikit-learn classification report? or is this the accuracy score?
I have the feeling, that this value is the rounded value of the accuracy_score the line below (when I use full dataset).
Question 2:
How can I print the f1-score?


texts: Multivariate outlier detection with isolation forest..How to detect most effective features?

texts:
I am trying to detect outliers in my data-set with 5000 observations and 800 features. I have followed the simple steps told in http://scikit-learn.org/stable/auto_examples/ensemble/plot_isolation_forest.html

There are some examples in stackoverflow and other sources as well, however, I could not find a decent explanation about the interpretation of outliers returned by isolationforest. First, what I did is:

from sklearn.ensemble import IsolationForest

X_train = trbb[check_cols]

clf = IsolationForest(n_jobs=6,n_estimators=500, max_samples=256, random_state=23)
clf.fit(X_train)
y_pred_train = clf.predict(X_train)
y_pred_train

This returns array([1, 1, 1, ..., 1, 1, 1]) where -1's are the outliers.

The shape of y_pred_train is 5000, which is identical with X_train[0]. So the index of -1 corresponds to the index of X_train. For exp. one of the outlier indices returned by IF is 532. So this means that the point (in 800 dimensional space) in this index is detected as an outlier. After detecting this, how can I approach and use the results to dig more? For example, can I reach the most important features causing the outliers?

To this end, I applied PCA and used the newly created (50) components as new variables. Then, by plotting component pairs with -1 & 1s returned by IF, I tried to get some insight of possible outliers. Indeed, I observed some points as shown below:

Here, the black dots are -1s (outliers) and the yellow ones are 1s (inliers). I interpret this that if both components have very high values, it might cause an outlier. However, how can I decide which original values are the most effective ones for those outliers? For example, can I use the original variables with the highest weights (let's pick top 10) in the related components? Or could you suggest any other method?

Thanks in advance!


texts: Sequence labeling with partially known labels

texts:
I am working on a sequence labeling task where, based on experience, many of the labels of a given input sequence can be reliably extracted with a simple rule-based approach. For example, considering the following input sequence:

[1, 2, 3, 1, 2, 1, 1] (ground truth labels of input sequence)
[1, X, 3, 1, 2, 1, Y] (labels as extracted by the rule-based approach)

In the above example, the rule-based system was able to extract all but two labels from the input. That is, only X and Y still need to be determined by a machine learning approach.

Are there known ways or algorithms to incorporate the labels extracted by a rule-based system into the machine learning process instead of running a sequence labeling algorithm from scratch? Intuitively, the additional information from the rule-based system should make it "easier" for the machine learning algorithm to fill the remaining slots.


texts: clarification on splitting individual trees in extra trees?

texts:
So I am a beginner in machine learning and just started learning about random trees in this article here. When it talks about tuning the hyperparameter K, I'm a bit confused as to how it works. It says:

  The parameter K denotes the number of random splits screened at each node to develop
  Extra-Trees. It may be chosen in the interval [1, ... , n], where n is the number of attributes.

So K would be the number that determines how many attributes to consider for a random split. Then to split, a random attribute from that set will be chosen? But what I'm wondering is that:

If K > 1, in a given set of attributes [1,2,3,4,...,n], is it always a contiguous subset of size K? Or is it K random attributes chosen from those n attributes? And once you choose a random attribute from that subset, it is replaced or left out?

It also says:

  For a given problem, the smaller K is, the stronger the randomization of the trees

I'm confused as to why this is.


texts: Data Visualisation for Dataframe having 3 columns

texts:
I have the below data, I am looking for a effective visualization or graphical method (in Python) to understand how the err is distributed with respect to min and max.

Data_X_err.head()
Out[137]:
     min    max   err
0  35435  35933  1.40
1  35155  36382  3.43
2  35305  36042  2.07
3  35216  36225  2.82
4  35196  36259  2.98

Thank you.


texts: What are some good papers that discuss Tufte's 'data density index' and 'data-to-ink ratio' in data visualisation?

texts:
I've found a number of resources that mention Tufte's 'data density index' and 'data-to-ink ratio' when considering the analysis of particular visualisations and visualisation techniques, but I am yet to come across any papers that actually critique the measures themselves.

Are there critiques of these principles available?


texts:
I am new to deep learning and tensor flow and I am trying to train a CNN at localizing digits in the Street View House Numbers data set. To this end I have an input set of 32x32 images and, since I want to recognize up to 5 digits, I am using as labels vectors of 20 elements like this

[top_x_digit1,top_y_digit1,widht_digit1,height_digit1,top_x_digit2, etc..]

0,0,0,0 when there is no digit

As far as I understand, after (let me say) 3 layers of convolution and pooling I can add 5 (parallel) fully connected layers aimed at extracting each the box features of a different digits (when present, 0 0 0 0 otherwise).

Is my approach correct?


texts: Dataset map function error : TypeError: Expected list for 'input' argument to 'EagerPyFunc' Op, not Tensor

texts:
I am currently trying to write a script to create a TFRecord file.
Therefore, I am following the instruction on the offical tensorflow website: https://www.tensorflow.org/tutorials/load_data/tfrecord#writing_a_tfrecord_file
However, when applying the map function to each element of the Dataset I get an error that I do not understand.
This is my code (should be copy and pasteable):
import numpy as np
import tensorflow as tf
from tensorflow.data import Dataset

def generate_random_img_data(n_count=10, patch_size=5):
    return np.random.randint(low=0, high=256, size=(n_count, patch_size, patch_size, 3))

def as_int64_feature(value):
    return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))

def serialize_one_image(img):
    features = {}
    for i in range(img.shape[0]):
        for j in range(img.shape[1]):
            for k in range(img.shape[2]):
                features.update({str(i) + "_" + str(j) + "_" + str(k) : as_int64_feature(img[i,j,k]) })
    example_proto = tf.train.Example(features=tf.train.Features(feature=features))
    return example_proto.SerializeToString()

def tf_serialize_one_image(img):
    tf_string = tf.py_function(serialize_one_image, img, tf.string)
    return tf.reshape(tf_string,())

ds = Dataset.from_tensor_slices(generate_random_img_data())
ds_serialized = ds.map(tf_serialize_one_image) # <--- not working

And when running this code I get the error:
TypeError: in user code:

    <ipython-input-116-ec81a7077c70>:25 tf_serialize_one_image  *
        tf_string = tf.py_function(serialize_one_image, img, tf.string)
    /Users/Tom/ML-Projects/vdst/lib/python3.7/site-packages/tensorflow/python/ops/script_ops.py:455 eager_py_func  **
        func=func, inp=inp, Tout=Tout, eager=True, name=name)
    /Users/Tom/ML-Projects/vdst/lib/python3.7/site-packages/tensorflow/python/ops/script_ops.py:341 _internal_py_func
        name=name)
    /Users/Tom/ML-Projects/vdst/lib/python3.7/site-packages/tensorflow/python/ops/gen_script_ops.py:69 eager_py_func
        name=name)
    /Users/Tom/ML-Projects/vdst/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:385 _apply_op_helper
        (input_name, op_type_name, values))

    TypeError: Expected list for 'input' argument to 'EagerPyFunc' Op, not Tensor("args_0:0", shape=(5, 5, 3), dtype=int64).

What exactly am I doing wrong here?


texts: Why is rpart not splitting this data even when there is gain in gini?

texts:
df <- tibble(x1=factor(c("S1", "S1", "S2", "S2")), y=factor(c(1, 1, 0, 1)))
md <- rpart(formula=y~., data=df, method="class", control=rpart.control(minsplit=2, cp=0))
nrow(md$frame) #outputs 1

consider the split
left child node :
"S1", 1
"S1", 1
right child node :
"S2", 0
"S2", 1
Here the gain in gini would be 1/8 = 0.125
why is rpart not doing this split?


texts: Finding a range of values for each feature that contribute to positive classification

texts:
Consider a classification problem(lets say 2 classes, 'good' and 'bad), where all features are continuous.

what I need is a range of values for each feature that contributes to 'good' classification.
What I thought was simply partitioning the feature values based on good or bad label, problem is all values don't equally contribute for good/bad classification.

So what methods can be applied to find such range for each feature ?


texts: Running Jupyter notebook on remote server with local files

texts:
I would like to train my model faster and that's why would like to create a standalone remote machine with GPU support. The solution I am looking for is to have the data locally on my machine and to use the resources from the remote machine. Is this possible?
I am checking my project for example locally on my laptop. When I use some remote Jupiter notebook, then it needs the data also to be on the remote machine. But after the model is trained I would like to commit it in git from my local machine. How to make this?