texts:
In working on exploratory data analysis, and developing algorithms, I find that most of my time is spent in a cycle of visualize, write some code, run on small dataset, repeat.   The data I have tends to be computer vision/sensor fusion type stuff, and algorithms are vision-heavy (for example object detection and tracking, etc), and the off the shelf algorithms don't work in this context.  I find that this takes a lot of iterations (for example, to dial in the type of algorithm or tune the parameters in the algorithm, or to get a visualization right) and also the run times even on a small dataset are quite long, so all together it takes a while.

How can the algorithm development itself be sped up and made more scalable?

Some specific challenges:

How can the number of iterations be reduced?  (Esp. when what kind of algorithm, let alone the specifics of it, does not seem to be easily foreseeable without trying different versions and examining their behavior)

How to run on bigger datasets during development?  (Often going from small to large dataset is when a bunch of new behavior and new issues is seen)

How can algorithm parameters be tuned faster?

How to apply machine learning type tools to algorithm development itself?  (For example, instead of writing the algorithm by hand, write some simple building blocks and combine them in a way learned from the problem, etc)


texts:
I have a dataset with the ID, name, joining date, leaving date as features. I was asked to measure employee retention and health of it. What can I derive from these?

What are some latest trends and examples which I can find relating these?

Thanks.

I know this is a discussion, but given that I couldn't find this on google search, it would be helpful for someone.


texts: How to find anomalies in (almost) constant stream of data?

texts:
I have a process that (simply put),  starts every 5 minutes, collects data, and put that data into the database.
More detailed explanation would be that process starts, collects data (which takes some time) and put it on kafka topic (which takes some time). Finally, data from kafka topic are consumed by database (which also take some time).
Every record in the database has its insertedOn time rounded up to the second.
When I count records (for 4 hours) by insertedOn time the graph look like this:

If I count records by 5 minutes intervals, the graph look like this:

On this graph you can see that all dots are around the same level (just a little bit above than 7000), but the dot marked with red arrow and its neighbor from the left, are bellow 7000.
Min, mean and max for these counts per 5 minutes are:
min     6262
mean    7154
max     7186

As the dot marked with red arrow is around 12% bellow mean or max, we could (probably) consider it as anomaly.
I am aware of several ML outlier/anomaly detection algorithms, but I am not sure how to use them when new data are constantly coming to the database. Whatever I do, I would like to avoid using fixed thresholds (like if count falls 10% bellow an average raise an alarm).
For example, the dot market with red arrow on the graph above happened at 2021-01-06 14:30:00, so few minutes later I should have raised an alarm because of that.
At the moment, this is the procedure that I have come up with regarding this 5 minutes data collection process. The following will be executed every 5 minutes:

take from database last few hours (time window) of counts by 5 minutes (n) periods
drop last period (n) because it is maybe not completed (records still not in the database)
use some ML algorithm (still not sure which one) on time window (up to n-2) in order to see whether count for last period (n-1) is an anomaly
if anomaly, raise an alarm
exclude an anomaly data point from future data collections in step 1.

I am not sure that this is a good approach.
If someone has done something similar before, with stream of data, please share your best practice about it.
In case that someone needs to see a dataset you can find it on https://pastebin.com/UaXeEjq9 as a csv.


texts: Trying to understand Logistic Regression Implementation

texts:
I'm currently using the following code as a starting point to deepen my understanding of regularized logistic regression.  As a first pass I'm just trying to do a binary classification on part of the iris data set.

One problem I think I have encountered is that the negative log-loss (computed with loss and stored in loss_vec) doesn't change much from one iteration to the next.

Another challenge I am facing is trying to figure out how to plot the decision boundary once I have learned the logistic regression coefficients. Using the coefficients to plot the 0.5 decision boundary is way off.  This makes me think I have made a mistake somewhere

http://fa.bianp.net/blog/2013/numerical-optimizers-for-logistic-regression/

import numpy as np
import matplotlib.pyplot as plt
from sklearn import linear_model, datasets

iris = datasets.load_iris()
X = iris.data[:, :2]  # we only take the first two features.
Y = iris.target

X = X[:100,:]
Y = Y[:100]

def phi(t):
    # logistic function, returns 1 / (1 + exp(-t))
    idx = t > 0
    out = np.empty(t.size, dtype=np.float)
    out[idx] = 1. / (1 + np.exp(-t[idx]))
    exp_t = np.exp(t[~idx])
    out[~idx] = exp_t / (1. + exp_t)
    return out

def loss(x0, X, y, alpha):
    # logistic loss function, returns Sum{-log(phi(t))}
    #x0 is the weight vector w are the paramaters, c is the bias term
    w, c = x0[:X.shape[1]], x0[-1]
    z = X.dot(w) + c
    yz = y * z
    idx = yz > 0
    out = np.zeros_like(yz)
    out[idx] = np.log(1 + np.exp(-yz[idx]))
    out[~idx] = (-yz[~idx] + np.log(1 + np.exp(yz[~idx])))
    out = out.sum() / X.shape[0] + .5 * alpha * w.dot(w)
    return out

def gradient(x0, X, y, alpha):
    # gradient of the logistic loss
    w, c = x0[:X.shape[1]], x0[-1]
    z = X.dot(w) + c
    z = phi(y * z)
    z0 = (z - 1) * y
    grad_w = X.T.dot(z0) / X.shape[0] + alpha * w
    grad_c = z0.sum() / X.shape[0]
    return np.concatenate((grad_w, [grad_c]))

def bgd(X, y, alpha, max_iter):
    step_sizes = np.array([100,10,1,.1,.01,.001,.0001,.00001])
    iter_no = 0
    x0 = np.random.random(X.shape[1] + 1) #initialize weight vector

    #placeholder for coefficients to test against the loss function
    next_thetas = np.zeros((step_sizes.shape[0],X.shape[1]+1) )
    J = loss(x0,X,y,alpha)
    running_loss = []
    while iter_no < max_iter:
        grad = gradient(x0,X,y,alpha)
        next_thetas = -(np.outer(step_sizes,grad)-x0)
        loss_vec = []
        for i in range(np.shape(next_thetas)[0]):
            loss_vec.append(loss(next_thetas[i],X,y,alpha))
        ind = np.argmin(loss_vec)
        x0 = next_thetas[ind]
        if iter_no % 500 == 0:
            running_loss.append(loss(x0,X,y,alpha))
        iter_no += 1
    return next_thetas


texts: When should i use fit(x_train) and when should i fit( x_train,y_train)?

texts:
I am a rookie in data science and I'm confused about some part of ML.
My Problems:
If I use a package of linear-model like LiearRegression() or PolynomialFeatures():

Should i to fit my (x_train,y_tarin) or only is enough when i fit my (x_train)?
Should i to transform my (x_train,y_tarin)?
Should i fit my (x_test) too or it will be only transformed? If only transform why?

from sklearn.linear_model import LinearRegression()
LinReg=LinearRegression()
LinReg.fit(x_train) or LinReg.fit(x_train,y_train) ???? ;-|
LinReg.transforn(x_test)


texts: Using the trainbr function for classification in Matlab

texts:
I am training a neural network for classification using Matlab, and I don't understand if I can use the trainbr training function (Bayesian Regularization Backpropagation). It uses the MSE performance measure, but I want to use the crossentropy. If I set crossentropy as the performance function, the algorithm sets it back to MSE.

On the other way, I can't use a validation set with this training and I don't find how to change it.

The code is:

x = A';
t = y';

% Choose a Training Function
% For a list of all training functions type: help nntrain
% 'trainlm' is usually fastest.
% 'trainbr' takes longer but may be better for challenging problems.
% 'trainscg' uses less memory. Suitable in low memory situations.
trainFcn = 'trainbr';  % Scaled conjugate gradient backpropagation.

% Create a Pattern Recognition Network
net = patternnet(hiddenLayerSize,trainFcn);

% Choose Input and Output Pre/Post-Processing Functions
% For a list of all processing functions type: help nnprocess
net.input.processFcns = {'removeconstantrows','mapminmax'};
net.output.processFcns = {'removeconstantrows','mapminmax'};

% Setup Division of Data for Training, Validation, Testing
% For a list of all data division functions type: help nndivide
net.divideFcn = 'dividerand';  % Divide data randomly
net.divideMode = 'sample';  % Divide up every sample
net.divideParam.trainRatio = 60/100;
net.divideParam.valRatio = 20/100;
net.divideParam.testRatio = 20/100;

% Choose a Performance Function
% For a list of all performance functions type: help nnperformance
net.performFcn = 'crossentropy';  % Cross-Entropy
net.trainParam.epochs = 5000;

% Choose Plot Functions
% For a list of all plot functions type: help nnplot
net.plotFcns = {'plotperform','plottrainstate','ploterrhist', ...
    'plotconfusion', 'plotroc'};

% Train the Network
[net,tr] = train(net,x,t);

% Test the Network
y = net(x);
e = gsubtract(t,y);
net.performParam.regularization = 0;
performance = perform(net,t,y);

tind = vec2ind(t);
yind = vec2ind(y);
percentErrors = sum(tind ~= yind)/numel(tind);

% Recalculate Training, Validation and Test Performance
trainTargets = t .* tr.trainMask{1};
valTargets = t .* tr.valMask{1};
testTargets = t .* tr.testMask{1};
trainPerformance = perform(net,trainTargets,y);
valPerformance = perform(net,valTargets,y);
testPerformance = perform(net,testTargets,y);

Thanks


texts:
I created my dummy variables, trained my model and tested it as below:

dummy <- dummyVars(formula = CLASS_INV ~ ., data = campaign_spending_final_imputed)
campaign_spending_final_dummy <- predict(dummy, newdata = campaign_spending_final_imputed) %>% as.data.frame() %>%
  mutate(CLASS_INV = campaign_spending_final$CLASS_INV)

I now want to test my process on an unseen real world data. How can I do it? How can I encode the category class to its respective dummy encoding?


texts:
I tried to generate a dataset with a respect to a correlation matrix R using cholesky decomposition. The problem is that i always get an KMO index that is very low ( smaller than 0.5 ).
Is there any solution to increase the KMO index without reducing the number of columns ?
I tried to compare the dataset with an other dataset which have the same number of columns and i calculated the correlation matrix of this dataset. Is there anyone who know a method to compare the two correlation matrix ?


texts: What scenarios are best to capture the behavior of a single person performing a regular task ? Not yet answered

texts:
I'm doing research on Predicting actions using CCTV feeds to control appliances. For example,
it can be something as simple as turning the light off in your room when you go to bed at night.
For my research, I need CCTV feed data. In the beginning, I aim to monitor the behavior of a single person performing a common task.
I would like to know what would be the best scenario to collect data for this reliably and quickly.
I have these examples in my mind

CCTV in dining room monitoring a single person switching the TV on/off regularly
CCTV in kitchen monitoring a single person turning a specific light on/off
CCTV in a lift monitoring a single person the floors he/she regularly visits.

I want to monitor only a single person at the start of my research. And I want his action to be a simple one that
he/she performs regularly. I also want to have lots of instances performing the same action over a brief period of time(Like two weeks)
Is my approach correct for this and What else scenarios can I use to gather data efficiently for my research?


texts: Model deployment with Flask, Tensorflow Serving and Docker

texts:
I am puzzled over this example: https://towardsdatascience.com/deploying-deep-learning-models-using-tensorflow-serving-with-docker-and-flask-3b9a76ffbbda
In the write-up, only the TF Serving is within the docker container while both model and flask app still reside outside the container.  In this case, how is isolation achieved? For portability, i suppose all components (model+flask) should also be "encapsulated"  within the same container?
I don't quite understand what was actually achieved in this example.


texts:
I'm interested in segmentation models for medical imaging purposes.
When I looked at the state of the art, I fell on a paper on a new architecture, Unet++:

UNet++: A Nested U-Net Architecture for Medical Image Segmentation from Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang
at Arizona State University

Like Unet, it is has an encoder/decoder architecture with skip connections (adding fine-grained feature maps of the encoder to the decoder). However, in Unet++, the skip connections are nested and dense so that a model can improve its ability to capture fine-grained details.
The second difference with Unet is the use of deep supervision.
The papers says that deep supervision enables:

"the model to operate in two modes: 1) accurate mode wherein the outputs from all segmentation branches are averaged; 2) fast mode wherein the ﬁnal segmentation map is selected from only one of the segmentation branches, the choice of which determines the extent of model pruning and speed gain"

I didn't understand how deep supervision works and what are the benefits in the case of Unet++.
Can you explain how it works?
Thank you in advance for your help.


texts: How to group by one column and count frequency from other column for each item in the previous column in python?

texts:
I am trying to group my data by the 'ID' column. Then I want to count the frequency of 'sequence' for each 'ID'. Here is a sample of the data frame:

ID     Sequence
101    1-2
101    3-1
101    1-2
102    4-6
102    7-8
102    4-6
102    4-6
103    1118-69
104    1-2
104    1-2

I am looking for a count same as:

ID   Sequence   Count
101    1-2        2
       3-1        1
102    4-6        3
       7-8        1
103    1118-69    1
104    1-2        2

I tried this code in python which doesn't give me what I want

  df.groupby('ID')
  Blockquote

df.groupby('Sequence').count()


texts: How to test the influence of a feature on conversion?

texts:
I have a user journey where I have data of the format:

  userID, did_interact_with_feature(0/1), did_convert(0/1)

I want to verify the hypothesis that if a user is engaging with the feature, he's more likely to get converted.

Now I can get the % of user who engaged in feature and then got converted. But I seem to have hit a mental block here.

I was leaning toward a z-test, but could not formulate the problem.

Any help really appreciated.


texts:
I am looking for a financial corpus or any form of publicly available financial texts which is replete with technical terms and acronyms.

Any suggestion is appreciated.


texts: Where is the rotated angle actually located in fitEllipse method?

texts:
Main task: I fit the ellipse using the fitEllipse() method and then I'd like to count the rotation angle between the horizontal axis and the major axis of the generated ellipse. I'm going to do this using the 3rd returned argument from the fitEllipse() method - $\theta$ (rotation angle).
Main issue: I can't find exact information about which axes this angle is located between.
Other: If I'm right the length of minor axis and the length of major axis in ellipse it's the same lengths as two sides in a rotated rectangle.
Sources:

From documentation here (section nb. 9) it seems that this angle is between horizontal axis and the first side how it's written in CvBox2D section. So, it means that the angle can be between the horizontal axis and minor or major axis.
But:
In this article (section 3) the first example is good but in the 2nd example the angle should be between horizontal axis and height instead of width
(referring to my aforementioned first point).
In this article it shows that the angle is between vertical axis and one side of rectangle.

So, where is the rotated angle located in fitEllipse() method?
Any hints how it works are welcome.


texts: Regression with a feature which has its own depth

texts:
I'm relatively new to ML/Statistical Analysis, and I'm facing a dataset structured like this
person_id, pay, task, hours
1, 560, A, 3
1, 560, B, 5
2, 650, A, 7
3, 520, C, 6
3, 520, A, 2
...

meaning person 1 is cumulatively paid 560 to perform task A 3 hrs and task B 5 hrs; person 2 paid 650 for task A 7 hrs; person 3 paid 520 for task C 6 hrs and A 2 hrs, etc. I hope it's clear.
I'd like to perform a regression, where my X plane is (task, hours) and Y is the per-person pay, but I haven't figured out yet how to approach such a problem. My tool box would be based on python+scikit-learn, preferably. But a generic discussion would be useful as well.
This is like
person_id, pay, tasks
[1, 560, [[A, 3], [B, 5]]
[2, 650, [[A, 7]]
[3, 520, [[C, 6], [A, 2]]
...

where person_id is a high cardinality feature which can be easily neglected, the Y label is pay, while the tasks (X) feature has its own structure, with fixed shape (2 dimensions here), but not a predetermined "depth", although limited in size (maybe 5-10 possible different tasks). I can't understand how to fit this in a regression schema, with such a structured feature data. Should I "flatten" tasks out, by explicitly having all possible values (A hours, B hours, C hours,... etc) as different columns, or is a more general approach possibile?
Moreover, this is a simplified version of my problem, to make the description simple enough, but it could include even more dimensions in the tasks structure, in which case the number of "flattened" task features would easily explode, to account for all possible combinations.
Any help welcome and appreciated!
Thanks


texts: Correct interpretation of summary_plot shap graph

texts:
While through the various resources online to understand the shap plots, I ended up slightly confused. Find below my interpretation of the overall plot given in examples -

Shap value 0 for a feature corresponds to the average prediction using all the other possible combinations of features except for the feature of interest. For e.g, shap value 0 for LSTAT corresponds to the average prediction of models having different combinations of features (excluding LSTAT)
Shap value of 4 with respect to 0 means that there is a relative increase in prediction being 1. How do we explain shap values 4, 2, etc. with respect to 0 in layman's terms?
Slight confusion in interpreting the overall graph vs the single point graph ( (

) -
In a single data point example, the base value/expected value is the average prediction in the entire dataset -> but in the overall plot (referenced at the top) we are saying it is the average prediction without that feature. So things don't add up?
Thank you for your help! :)


texts: Balance two crossentropy losses with different number of neurons

texts:
I have a model with a few outputs, each output with shape:

Shape: (batch_size, labels_1) -> softmax -> categorical_crossentropy -> loss_value_1
Shape: (batch_size, labels_2) -> softmax -> categorical_crossentropy -> loss_value_2
Shape: (batch_size, labels_3) -> softmax -> categorical_crossentropy -> loss_value_3
....

Now this model is used for a classification problem, so I'm applying a softmax activation to each and using the categorical_crossentropy (or just "crossentropy") loss.

Basically, everything is working reasonably well, but I noticed the losses are unbalanced. (Not because of class unbalance, but because of a different number of neurons)

An output with more neurons tend to have greater losses because their initial predictions for the correct label (the only label considered by this loss) is closer to zero.

I could try to balance them with a simple loss / number_of_neurons, but I'm not confident that this will indeed balance the losses (since softmax is not linear), it sounds like unbalancing in a different way.

Is there a mathematically correct way to balance these losses to compensate for their number of neurons?


texts: Tensorflow oscillating Test and Train Accuracy?

texts:
I have implemented a CNN with images as input and 101 classes as output.
I have applied mean subtraction and normalization to the input before giving it as input to the network.
I have also experimented with multiple optimizers (ADAM & SGD) and also changed the learning rate (0.1,0.01,0,001) and also batch sizes (19,32,64,128,200).
But with all of these cases I had observed a similar pattern.
I dont think the network is learning much and more importantly the accuracy oscillating between a few values.
I had initially thought it was because of the batch size but I was wrong.
I have built and trained the exact same network with same inputs with caffe.  And achieved a 55% accuracy(check for working of  network model).
My guess is the implementation went wrong somewhere but not sure where.
Mycode
Output
This particular output is with lr=0.001 batch_size=200 and ADAM optimizer
Iter 2000, Minibatch Loss= 12560.197266, Training Accuracy= 0.01000
Testing Accuracy: 0.025
Iter 4000, Minibatch Loss= 4587.021484, Training Accuracy= 0.00500
Testing Accuracy: 0.005
Iter 6000, Minibatch Loss= 2196.316406, Training Accuracy= 0.02000
Testing Accuracy: 0.02
Iter 8000, Minibatch Loss= 918.784119, Training Accuracy= 0.01500
Testing Accuracy: 0.0
Iter 10000, Minibatch Loss= 315.327911, Training Accuracy= 0.01000
Testing Accuracy: 0.0
Iter 12000, Minibatch Loss= 102.744026, Training Accuracy= 0.01000
Testing Accuracy: 0.01
Iter 14000, Minibatch Loss= 59.417763, Training Accuracy= 0.00500
Testing Accuracy: 0.005
Iter 16000, Minibatch Loss= 33.026432, Training Accuracy= 0.00500
Testing Accuracy: 0.02
Iter 18000, Minibatch Loss= 30.361868, Training Accuracy= 0.02000
Testing Accuracy: 0.005
Iter 20000, Minibatch Loss= 33.944790, Training Accuracy= 0.01000
Testing Accuracy: 0.02
Iter 22000, Minibatch Loss= 15.029477, Training Accuracy= 0.02000
Testing Accuracy: 0.02
Iter 24000, Minibatch Loss= 13.751340, Training Accuracy= 0.02500
Testing Accuracy: 0.02
Iter 26000, Minibatch Loss= 9.085027, Training Accuracy= 0.01500
Testing Accuracy: 0.015
Iter 28000, Minibatch Loss= 14.407433, Training Accuracy= 0.02000
Testing Accuracy: 0.02
Iter 30000, Minibatch Loss= 11.090684, Training Accuracy= 0.02500
Testing Accuracy: 0.02
Iter 32000, Minibatch Loss= 9.385777, Training Accuracy= 0.02000
Testing Accuracy: 0.03
Iter 34000, Minibatch Loss= 12.902111, Training Accuracy= 0.01500
Testing Accuracy: 0.02
Iter 36000, Minibatch Loss= 5.580408, Training Accuracy= 0.02000
Testing Accuracy: 0.005
Iter 38000, Minibatch Loss= 11.641799, Training Accuracy= 0.02000
Testing Accuracy: 0.03
Iter 40000, Minibatch Loss= 6.941562, Training Accuracy= 0.01500
Testing Accuracy: 0.02
Iter 42000, Minibatch Loss= 6.781301, Training Accuracy= 0.01000
Testing Accuracy: 0.0
Iter 44000, Minibatch Loss= 6.840708, Training Accuracy= 0.01000
Testing Accuracy: 0.01
Iter 46000, Minibatch Loss= 5.501908, Training Accuracy= 0.01000
Testing Accuracy: 0.03
Iter 48000, Minibatch Loss= 4.667253, Training Accuracy= 0.03000
Testing Accuracy: 0.015
Iter 50000, Minibatch Loss= 4.613351, Training Accuracy= 0.01000
Testing Accuracy: 0.01
Iter 52000, Minibatch Loss= 7.234328, Training Accuracy= 0.02000
Testing Accuracy: 0.01
Iter 54000, Minibatch Loss= 8.120478, Training Accuracy= 0.02500
Testing Accuracy: 0.01
Iter 56000, Minibatch Loss= 6.804164, Training Accuracy= 0.01000
Testing Accuracy: 0.01
Iter 58000, Minibatch Loss= 5.060484, Training Accuracy= 0.03000
Testing Accuracy: 0.02
Iter 60000, Minibatch Loss= 4.937673, Training Accuracy= 0.02000
Testing Accuracy: 0.015
Iter 62000, Minibatch Loss= 8.758166, Training Accuracy= 0.01000
Testing Accuracy: 0.02
Iter 64000, Minibatch Loss= 4.616434, Training Accuracy= 0.01500
Testing Accuracy: 0.025


texts:
How to transform raw data to fixed-frequency time series?

For example I have the following raw data in DataFrame

                       A    B
2017-01-01 00:01:01    0    100
2017-01-01 00:01:10    1    200
2017-01-01 00:01:16    2    300
2017-01-01 00:02:35    3    100
2017-01-01 00:02:40    4    100

I'd like to transform it into a time series:

1 minute frequency
column A should have sum of values in time interval
column B should have mean of values in time interval
possibly other functions over other columns

Note: Raw data is not periodic.

Transformed data should be:

                       A    B
2017-01-01 00:01:00    3    200
2017-01-01 00:02:00    7    100


texts: What level of math is required for machine learning research

texts:
There are several levels of math understanding:

Know the math
Know the intuitions behind math concepts
Know the intuitions and proofs of math concepts
Know the intuitions, proofs of math concepts and be able to apply them to deduce new results

My question is what level of understanding is required for machine learning research? Especially for publishing papers in conferences like CVPR or NeurIPS.


texts: Why does categorical_crossentropy work in when input is not 1-hot encoded?

texts:
I'm going through lessons on the REINFORCE algorithm to solve Cartpole/Pong/etc (using AIGym) and every one uses categorical_crossentropy as the loss function.
What's confusing me is that categorical_crossentropy expects a 1-hot encoded setup as its true parameter. The states being passed in are not 1-hot encoded, they're just the numerical representation of the state of the universe at that moment. Why does this work?


texts: Understanding predict_proba from MultiOutputClassifier

texts:
I'm following this example on the scikit-learn website to perform a multioutput classification with a Random Forest model.
from sklearn.datasets import make_classification
from sklearn.multioutput import MultiOutputClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.utils import shuffle
import numpy as np

X, y1 = make_classification(n_samples=5, n_features=5, n_informative=2, n_classes=2, random_state=1)
y2 = shuffle(y1, random_state=1)
Y = np.vstack((y1, y2)).T

forest = RandomForestClassifier(n_estimators=10, random_state=1)
multi_target_forest = MultiOutputClassifier(forest, n_jobs=-1)
multi_target_forest.fit(X, Y).predict(X)

print(multi_target_forest.predict_proba(X))

From this predict_proba I get a 2 5x2 arrays:
[array([[ 0.8,  0.2],
       [ 0.4,  0.6],
       [ 0.8,  0.2],
       [ 0.9,  0.1],
       [ 0.4,  0.6]]), array([[ 0.6,  0.4],
       [ 0.1,  0.9],
       [ 0.2,  0.8],
       [ 0.9,  0.1],
       [ 0.9,  0.1]])]

I was really expecting a n_sample by n_classes matrix. I'm struggling to understand how this relates to the probability of the classes present.
The docs for predict_proba states:

array of shape = [n_samples, n_classes], or a list of n_outputs such arrays if n_outputs > 1.
The class probabilities of the input samples. The order of the classes corresponds to that in the attribute classes_.

I'm guessing I have the latter in the description, but I'm still struggling to understand how this relates to my class probabilities.
Furthermore, when I attempt to access the classes_ attribute for the forest model I get an AttributeError and this attribute does not exist on the MultiOutputClassifier. How can I relate the classes to the output?
print(forest.classes_)

AttributeError: 'RandomForestClassifier' object has no attribute 'classes_'


texts: Spark Mllib - FPG-Growth - Machine Learning

texts:
Is the FPG-Growth an Machine Learning algorithm? Because I'm looking at this code:

import org.apache.spark.mllib.fpm.FPGrowth
import org.apache.spark.rdd.RDD

val data = sc.textFile("data/mllib/sample_fpgrowth.txt")

val transactions: RDD[Array[String]] = data.map(s => s.trim.split(' '))

val fpg = new FPGrowth()
  .setMinSupport(0.2)
  .setNumPartitions(10)
val model = fpg.run(transactions)

model.freqItemsets.collect().foreach { itemset =>
  println(itemset.items.mkString("[", ",", "]") + ", " + itemset.freq)
}

val minConfidence = 0.8
model.generateAssociationRules(minConfidence).collect().foreach { rule =>
  println(
    rule.antecedent.mkString("[", ",", "]")
      + " => " + rule.consequent .mkString("[", ",", "]")
      + ", " + rule.confidence)
}

And I'm not seeing how the algoritm can learn because it don't have an train, test or validation set...

Thanks


texts: I'm working on neural networks. I have one very important question. For instance, there are two layers with 2 neurons in each. They are all fully connected. Which neurons do the weights belong to? Do they belong to neurons in next layer or the layer preceding it? Moreover, when weights are saved in Keras, are they saved for each neuron completely or they are saved as linknumber(first weight of every neuron)? I hope I'm able to clarify my question


texts:
Are there methods to tune and train an xgboost model in an optimized time - when I tune paramaters and train the model it takes around 12 hours to execute?

I would like to run the solution 100 times with 100 seeds; my machine has 8 GB RAM and I can't buy a cloud solution.


texts: How do you do 1-vs-rest classifiers in XGBoost Library (Not Sklearn)?

texts: I am working with a very large dataset that would benefit from using training continuation with the xgb_model parameter in xgb.train(). The label (Y) of dataset itself has 4 classes and is highly imbalanced, so I would like to generate per-label PR curves for it to evaluate its performance, and would thus need to treat each class as it's own binary problem using a one-vs-rest classifier. After a lot of reading I haven't found an equivalent to sklearn's OneVsRestClassifier in the xgboost library. Could anyone provide some guidance on how to implement continuously training one-vs-rest classifiers using the XGBoost library? Thank you in in advance.


texts:
I am a computer science student working on a small information retrieval project. I have a dictionary with a domain as a key and it's ranking as value.

Based on that ranking, I need to score every domains. I was thinking to do 1/ranking but the disparity is too high. For example the first domain will have a score of 1 (1/1) and the domain ranked 10th will have a score of 0.1 which does not make sense for this.

I have 1000 domains in total and the last one should be close to 0.10 and the first one close to 1


texts: how does gradient descent update weigths in neural network

texts:
Im currently trying to learn about back propagation, and it's going forward, but theres one thing that keeps me scratching my head, and doesnt really seems to be answered in any of the videos or articles im looking at.
I understand now, that based on my loss, the weigths of my network is updated. But what i dont understand is how this happens. lets say i have this exercise network with the following weigths:

W_1 = 1.2 - w_2 = 0.4 - W_3 = 1.0

Now i do some training, and lets say i have the loss o.8.
Now when i use my loss to update my weights, what happens specifically to the weights? are something being added, subtracted maybe multiplied?

Thanks a lot


texts: How can I get semantic word embneddings for compound terms?

texts:
I need to build semantic word embeddings representation of compound terms like "electronic engineer" or "microsoft excel". One approach would be to use a standard pretrained model an average the words but, since I have a corpus  of my domain, is there a possible better approach?

To be more precise:

The data I have is a corpus of millions of documents. Each document is ~ half a page and contains these compound terms. However there may be compound terms not included in the corpus.

Thanks


texts: Explaining instances of a machine learning regressor

texts:
I trained an XGBoost regressor, and I'm looking for a way to interpret new instances that the model will receive in production.
There's a nice solution for classification models (Lime), but it doesn't offer explanations to regression problems.

Does anyone know of an existing "explainer" for regressions?


texts: I have an R function for (i in 1:10) that reads in data and performs some cleaning. Is it possible to write a flexible statement to execute certain parts of the code conditionally, for example have different steps for i = 2 and 8?


texts:
I am trying to generate similar sentences, called paragraph generation. For example, what is the name of the eldest brother of ram? - For these paragraphs can be - who is the oldest brother of ram? , Who is the oldest blood brother of ram? , I want to know the eldest sibling of ram.  etc etc

BERT and OpenAI GPT are one of the most powerful NLP systems, as per my knowledge, at least for now. However, I heard that BERT is based on MLM so it cant be used for sentence generation, but can be used for word prediction.

Can OpenAI GPT be used for this purpose? If so, please give some pointer.
If any other name is in mind which is the State of Art please suggest.


texts: Combining binary features with textual ones

texts: I have a binary feature that i want to use it with textual features i.e. unigrams. I use logistic regression and TF/IDF for representing text. So i simply add a unique feature, say ss or oo, to text of each instances. But in practice, i see adding more number of these features to instances, say two oo or ss or more get me a better results. What is the reason? How these weights improve the classification results? Should not logistic regression can get more weights to this features instead of weighting them by hand?


texts: How to find correlation between two junctions of a road

texts:
I am working on traffic data analytics, where I need to predict the number of vehicles passing through each traffic junctions.

Shape of my dataset is something like this :

Road A-Junction B-Junction   Date       Hour  CarCount
X         a          b      2000-06-09   7       10
Y         b          c      2000-06-09   7       20

Road – this is the road name (for instance M25 or A3).
A-Junction – The road name of the start junction of the link
B-Junction – The road name of the end junction of the link
Hour - time in 24 hour format
CarCount - actual car count

I want to find out the correlation between A-Junction and B-Junction.
Or in simple words , let's say in the given example for road X and Y , can we correlate between a , b and c roads ?


texts: How to clusterize points lying on the sphere

texts:
I want to cluster protein conformations by dihedrals angles.
My point is an n-dimensional vector, where is n - number of dihedral angles.
I think I can't use Euclidean distance for distance metric because of the distance between +179 degrees and -179 degrees is 2 degree, not 358.

Can you suggest some clustering algorithms and distance metrics for such tasks?
Maybe some algorithms from geo points clusterization.


texts: Does feature selections matter to Decision Tree algorithms?

texts:
Background: Currently I'm working with my thesis project, which is to build a Tree-based ensemble methods for classification on a large data set. Before I started with modelling, I've spent a large amount of time on feature selection using correlation-based criteria to select a subset of features, so that those chosen features have high correlation with the response variable and low correlation with each other.

However, one of my supervisors questioned why I spent so much time in feature selection, as he mentioned, the decision tree algorithm can naturally select which features are most important. Later I checked book Introduction to Data Mining by Tan (2014) and it says clearly that "Feature selection occurs naturally as part of the data mining algorithm, ..., such as decision tree classifiers" .
I felt a bit frustrated by spending time on feature selection. When I eventually run the models with and without feature selection, it does NOT show any significant differences between the results.

My question is: Does feature selections matter at all to Decision Tree algorithms?


texts: sklearn & Meanshift for NLP only returns 1 cluster

texts:
I am using sklearn.clustering to work with some text data and the MeanShift algorithm. I have:

Done all standard NLP data prep like lemmatizing, removing stop words, etc.
Used the TfidfVectorizer to create my word vectors on 80k-plus records
The vectorizer gives me a sparse array so I converted it using a standard .toarray() command
I made a call to sklearn Meanshift and then accepted all of the default parameters. The call looks like meanshift = MeanShift().fit(fitted_vector_data.toarray()) and results in the following output when I call the model: MeanShift(bandwidth=None, bin_seeding=False, cluster_all=True, min_bin_freq=1, n_jobs=1, seeds=None)

The problem is that no matter what data I pass in (whether it's 10 records or 10k records, it always just gives me 1 cluster when I should be getting hundreds of clusters.

This is my first time using MeanShift, so I'm guessing there is a problem with how I'm setting up my data and/or parameters? I should also point out, I have used other models like k-means and affinity propogation - with the same data prep - and those models gave multiple clusters.


texts: How to validate regex based Resume parser efficiently

texts:
I am using rule based logic to extract features from resume. Basically I am trying to find if the candidate switched the company in less than 1 year. So I have the code in place to find it using python. However if I want to validate it, I am currently doing it manually for few resumes, meaning I open the resume and find if the candidate switched the company in less than 1 year and compare it with what my regex logic gives. But this is time consuming process to validate.

Is there any better way to validate this piece of python code. ?


texts: How to predict multiple images from folder in python

texts:
Here is the code for the Prediction of multiple images from the folder. But getting the same label(class) for all the images.I'm not able to find out why every image shows the same label.
# import the necessary packages
from tensorflow.keras.models import load_model
import argparse
import pickle
import cv2
from tensorflow.keras.preprocessing.image import img_to_array
from tensorflow.keras.applications.imagenet_utils import decode_predictions
import numpy as np
import logging, os
import sys
from keras.preprocessing import image
import tensorflow as tf
import math
import operator
from pathlib import Path

# disable the warnings
logging.disable(logging.WARNING)
os.environ["TF_CPP_MIN_LOG_LEVEL"] = "3"

image_path = "test_image_folder"

images = []

# load all images into a list
for img in os.listdir(image_path):
        img = os.path.join(image_path, img)
        img = image.load_img(img, target_size=(64,64))
        img = image.img_to_array(img)
        img = np.expand_dims(img, axis=0)
        # normalize the image
        processed_image = np.array(img, dtype="float") / 255.0
        images.append(processed_image)

images = np.vstack(images)

# relative paths to the model and labels
model_path = os.path.join("Output", 'VGG_model.h5')
label_file_path = os.path.join("Output", 'labels')

# load the model and the label encoder
model = load_model(model_path)
lb = pickle.loads(open(label_file_path, "rb").read())

# make a prediction on the image
images_data = []
filenames = []
for filename in os.listdir(image_path):
    pred_result = model.predict(images)
    images_data.append(pred_result)
    filenames.append(filename)

#sorts attributes according to confidence score (how probable attribute exists)
top_k = []
pred = []
for i in range(len(images_data)):
            rank = images_data[i][0].argsort()[-len(images_data[i][0]):][::-1]
            top_k.append(rank)
            top = top_k[i][:15]
            print(filenames[i])
            for node_id in top:
                human_string = label_file_path[node_id]
                score = images_data[i][0][node_id]
                print('%s (score = %.5f)' % (human_string, score))


texts: I need help in PCA results using WEKA Tool

texts:
I'm working on an experiment using KDD'99 cupset I have 42 features. the paper I 'm comparing with concludes that 3 features with precision  ..% ok are the best subset to identify the attack X.
In my experiment,  I applied 4 different classifier through the PCA.  How to compare between them in order to conclude the number of features used in my experiment ? how to explain my  features in order to say that n features gives higher preciosion.


texts: Kernel approximation of a function known only point-wise?

texts:
Assume that I have a set of $N$ points $x_i, i=1,...,N,$ in some space $\mathbb{R}^D$, and corresponding point-wise (scalar) function evaluations $f(x_i)$. It is my goal to approximate the unknown function $f(x)$ with RBF kernels:
$$\tilde{f}(x)=\sum_{i=1}^N w_i k(x,x_i)$$
where $k(x,x_i)$ is a RBF kernel centred on $x_i$. It may seem intuitive to set $w_i=f(x_i)$, but then I will usually not reclaim $f(x_i)$ due to the influence of other basis functions. Towards this end, I have a number of questions, and would appreciate it if you could answer some of them:

Is there a procedure you can recommend for finding the weights $w_i$ so that $\tilde{f}(x_i)=f(x_i)$?
Can this procedure work with higher-dimensional output (i.e., $f:\mathbb{R}^D \rightarrow \mathbb{R}^E,E>1$)?
Are there any references or lectures you can recommend on the topic?


texts: Any tool to load large csv files quickly on windows 7

texts:
I am working with relatively large CSV files (1.5 MB+) and while testing the code, I have to frequently open up the CSV files to read a particular row, validate the data are there and then close the CSV file and move to another one.

Files which are larger than 1.5 MB are taking about 40 seconds per file open on my computer. Is there any tool that will allow me to open large CSV files quickly (I am only looking to read the CSV file and therefore a CSV reader that opens up files quickly will be useful)

Alternatively, if there is any tool that will perhaps when the first 1000 lines
or the last 1000 lines will be good.
I know I can do this in UNIX by using commandline utilities but then they dump the data in different files and again I will have to delete those files. Moreover, I'm looking for a solution on Windows.

I hope somebody can help resolve this. Thank you for any inputs!


texts:
Neural networks are trained to minimize some error function over the weights of the neural connections. In some applications, these weights could be considered intellectual property. Is there a way to encrypt these weights and still have an operational neural network?
Some context: I'm trying to scale a neural network algorithm, but right now we're doing all the computations on a centralized server and it's getting bogged down. We can shift the computation to the client side, but we don't want someone to unpack the executable and obtain the weights of the network. Is there some way to distribute an "encrypted neural network" such that our IP is protected?
To clarify: I am not talking about "oblivious" neural networks that operate on encrypted data, I am talking about the weights of the neural network themselves.
I'm fine with alternative obfuscation techniques.
Edit: I found this paper, but it states

Great attention
is paid to avoid any unnecessary disclosure of information,
so that at the end of the protocol the user only knows
the final NN output, whereas all internal computations are
kept secret. In this way, the possibility for a malevolent user
to provide a set of fake inputs properly selected to disclose
the network secrets is prevented. A solution is also sketched
that permits to obfuscate the network topology, however, a
deeper investigation in this direction is left for future research.

indicating that this paper is on a related concept, but I'm looking for a resource where the topology is obfuscated.


texts: I just wanted to know how to cluster n-grams based on their semantics. Like clustering together n-grams that are semantically similar by leveraging the distributional hypothesis suggesting that similar words appear in similar contexts.


texts: Expectation Maximization Algorithm (EM) for Gaussian Mixture Models (GMMs)

texts:
I'm trying to apply the Expectation Maximization algorithm (EM) to a Gaussian Mixture Model (GMM) using Python and NumPy. The PDF document I am basing my implementation on can be found here.
Below are the equations:
$\mathrm{E}-\text{step:}$
$$w_{ik} = \frac{\pi_k \cdot p_k(x_i|z_k, \mu_k, \Sigma_k)}{\sum_{m=1}^{K} \pi_m \cdot p_m(x_i|z_m, \mu_m, \Sigma_m)}, \; [1]$$
$\text{where:}$
$${\displaystyle (2\pi )^{-{1}}|{\Sigma_k}|^{-{\frac {1}{2}}}\,\mathrm e^{-{\frac {1}{2}}(x_i -{\mu_k})^{\!{\mathsf {T}}}{{\Sigma_k }}^{-1}(x_i -{\mu_k})}.}
$$
$\mathrm{M}-\text{step:}$
$$\pi_k^{\text{new}} = \frac{N_k}{N}, \; [2]$$
$\text{where:}$
$$N_k = \sum_{i=1}^{N} w_{ik}.$$
$$\mu_k^{\text{new}} = \frac{1}{N_k} \sum_{i=1}^{N} w_{ik} \cdot x_i, \; [3]$$
$$\Sigma_k^{\text{new}} = \frac{1}{N_k} \sum_{i=1}^N w_{ik} (x_i - \mu_k)(x_i - \mu_k)^\mathsf {T}. \; [4]$$
When applying the algorithm I get the mean of the first and second cluster equal to:
array([[2.50832195],
       [2.51546208]])

When the actual vector means for the first and second cluster are, respectively:
array([[0],
       [0]])

and:
array([[5],
       [5]])

The same thing happens when getting the values of the covariance matrices I get:
array([[7.05168736, 6.17098629],
       [6.17098629, 7.23009494]])

When it should be:
array([[1, 0],
       [0, 1]])

for both clusters.
Here is the code:
np.random.seed(1)

# first cluster
X_11 = np.random.normal(0, 1, 1000)
X_21 = np.random.normal(0, 1, 1000)

# second cluster
X_12 = np.random.normal(5, 1, 1000)
X_22 = np.random.normal(5, 1, 1000)

X_1 = np.concatenate((X_11,X_12), axis=None)
X_2 = np.concatenate((X_21,X_22), axis=None)

# data matrix of k x n dimensions (2 x 2000 dimensions)
X = np.concatenate((np.array([X_1]),np.array([X_2])), axis=0)

# multivariate normal distribution function gives n x 1 vector (2000 x 1 vector)
def normal_distribution(x, mu, sigma):
  mvnd = []
  for i in range(np.shape(x)[1]):
    gd = (2*np.pi)**(-2/2) * np.linalg.det(sigma)**(-1/2) * np.exp((-1/2) * np.dot(np.dot((x[:,i:i+1]-mu).T, np.linalg.inv(sigma)), (x[:,i:i+1]-mu)))
    mvnd.append(gd)
  return np.reshape(np.array(mvnd), (np.shape(x)[1], 1))

# Initialized parameters
sigma_1 = np.array([[10, 0],
                    [0, 10]])
sigma_2 = np.array([[10, 0],
                    [0, 10]])
mu_1 = np.array([[10],
                 [10]])
mu_2 = np.array([[10],
                 [10]])
pi_1 = 0.5
pi_2 = 0.5

Sigma_1 = np.empty([2000, 2, 2])
Sigma_2 = np.empty([2000, 2, 2])

for i in range(10):
  # E-step:
  w_i1 = (pi_1*normal_distribution(X, mu_1, sigma_1))/(pi_1*normal_distribution(X, mu_1, sigma_1) + pi_2*normal_distribution(X, mu_2, sigma_2))
  w_i2 = (pi_2*normal_distribution(X, mu_2, sigma_2))/(pi_1*normal_distribution(X, mu_1, sigma_1) + pi_2*normal_distribution(X, mu_2, sigma_2))
  # M-step:
  pi_1 = np.sum(w_i1)/2000
  pi_2 = np.sum(w_i2)/2000
  mu_1 = np.array([(1/(np.sum(w_i1)))*np.sum(w_i1.T*X, axis=1)]).T
  mu_2 = np.array([(1/(np.sum(w_i2)))*np.sum(w_i2.T*X, axis=1)]).T
  for i in range(2000):
    Sigma_1[i:i+1, :, :] = w_i1[i:i+1,:]*np.dot((X[:,i:i+1]-mu_1), (X[:,i:i+1]-mu_1).T)
    Sigma_2[i:i+1, :, :] = w_i2[i:i+1,:]*np.dot((X[:,i:i+1]-mu_2), (X[:,i:i+1]-mu_2).T)
    sigma_1 = (1/(np.sum(w_i1)))*np.sum(Sigma_1, axis=0)
    sigma_2 = (1/(np.sum(w_i2)))*np.sum(Sigma_2, axis=0)

Would really appreciate if someone could point out the mistake in my code or in my misunderstanding of the algorithm.


texts: Filter row depending on specific object value and delete those instances

texts:
I have some categorical data which also contains '?' as data in some rows. I need to filter those rows depending on '?', that which row contain that instances will be deleted.

I tried to drop those rows by applying these command but I failed.

train = train.drop[~train.str.contains('\?')]
train = train.drop[train['?']]

How could I identify those rows which contain '?' instance and drop those row ?


texts: What is the most appropriate machine learning approach for this scenario?

texts:
The scenario is pretty simple, and I'm sure it's been done a million times. The problem is i don't know the terminology to find the correct resources on the web.

Scenario: I have an environment that can be described in terms of 5 parameters, including and input value A and an output  variable B. There is a dataset containing 100 rows and values for each parameter.

The output B depends on A as well as the remaining environmental variables.

The goal is to find the best value for the input A such that the output B is minimised.

What does the solution for this problem look like? Is it  Machine learning, neural networks, a mathematical optimisation problem? How is this best approached?

Extension: if I didn't have a dataset in practice,  how would I train a system to suggest different values for A until a minimum is reached? Can neural networks be applied here? Or are we talking about a looping procedure that does known maths in each operation until the output doesn't change much anymore.

I though the generalisation would make it more difficult to answer.
What I am describing is a number of temperature/humidity measure measurements for both inside my house and outside. The input that I can control is the fan speed setting on my evaporative aircon and the output is the loubge temperature which I want to minimise.

During sample gathering, I don't care much about the output. The set of 100 values was arbitrary and more (and diverse) samples can be obtained.

The range of A is a fan speed with discrete values 1-6.
Humidity is a percentage and temperature is 20-45 degrees.


texts:
Following Andrew Ng's machine learning course, he explains how we can modify logistic regression to obtain SVM algorithm. First he replaces (sort of approximating) cross entropy loss with hinge loss as shown in the image below:

Then he removes the $\frac{1}{m}$ coefficient and divides the whole cost function by the regularization parameter $\lambda$, which leads to parameter $C$ behind the sigma ($C=\frac{1}{\lambda}$). Resulting cost function is shown in image below which is the minimization objective in SVM:

Later he shows that by choosing a very large value for parameter $C$, SVM will be a large margin classifier. Large value for parameter $C$ is the same as choosing a small value for parameter $\lambda$ in logistic regression. So, is logistic regression with a small value for $\lambda$ also a large margin classifier?


texts:
Let's assume I have a number of different documents. I used doc2vec to generate a vector per document. So after plotting it with t-Sne, I can visualize where the documents are in a $2$ or $3$ dimensional space. So I have realistic distances between the documents.

Now there comes the (for me) tricky part:
I want consumers to answer a questionnaire, where they rate the documents on a $5$-point-Lickert scale ($1$ do not like - $5$ like very much). In the end, I want to see where consumer positions himself in the plot I made before.

Which means, that I somehow need to weight the vectors.
document $1$ - with vector $1234$ is rated as $5$ (so this document has a high (I call it) "gravitation" )
document $2$ - with vector $1756$ is rated with $2$ (so with lower gravity...)

Do you have any suggestion on how to solve this problem? I am still not sure on how to "calculate" with (let's say a $100$ dimensional) vector

It would be awesome if you had a hint for me.


texts:
say I have 2 datasets and need to get an average but with a weight based on the number of students.

school 1  = 98.1 . (50 students)
School 2 = 95.4 .   (169 students)

How would i get the average of both with a weighted average reflected the average of the student totals i think it would be around   = 96.ish

thanks for any insight!


texts: How to explain a relationship between Accuracy and F1 Score / F-Measure?

texts:
I am building a CNN model for pitch estimation using a song recording. Pitch estimation is done by inputting spectrogram to CNN model and make the CNN predict pitch sequence (250 pitch values per recording) from that spectrogram. For the evaluation metrics, I am using Accuracy and F1 Score. Sample of overall test result are given below using mean measurement.

Some notes:

Val-Acc is the validation accuracy. I am using this to see how well the model analyze new data that is not given during training.
Delta acc is the difference value between accuracy and val-acc.

Right now, I am wondering how can I explain the relationship between Accuracy and F1 Score. My supervisor said to me that accuracy is measured to get how accurate the model performs, and F1 is how well the model performs. Is the relationship really like that? May I get some insight on how to explain the relationship between them?


texts: What is the difference between parameters & cooficients in Machine learning?

texts:
are both terms interchangeable?
I'm kinda new to machine learning field and very confused about these terms in machine learning perspective.


texts: Question about sklearn's StratifiedShuffleSplit

texts:
I'm reading through the book Hands-On Machine Learning with Scikit-Learn and Tensorflow by Aurélien Géron. In a regression project on California Housing Prices, he goes over the concept of stratified sampling.

I think I understand the concept as his explanation "the population is divided into homogeneous subgroups called strata, and the right number of instances is sampled from each stratum to guarantee that the test set is representative of the overall population."

So in my own words, simply splitting the dataset with sklearn's train_test_split leaves the train and test set vulnerable to misrepresenting the ratios of categorical variables (ie population has 40% one category, 60% another, but the train/test set are totally different ratios of these categories), so stratifying ensures the sample is 'random', but still maintaining proper ratios within test and train splits. Please correct me if I'm wrong.

Here's the code to his stratified sampling based on income categories (housing is the main dataframe):

split = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
for train_index, test_index in split.split(housing, housing['income_cat']):
    strat_train_set = housing.loc[train_index]
    strat_test_set = housing.loc[test_index]

I'm pretty confused with this code:

1) What does the variable 'split' represent? Does it comprise both the train and test split...?

2) In the 2nd line of code, what does split.split mean? I guess I'm confused with most of how StratifiedShuffleSplit divides the train and test set and why he needed to create this 'for' loop in order to create strat_train_set and strat_test_set.

Thanks,

Greg


texts: I am doing a practice problem predicting a binary outcome. I have plotted an ROC curve and found the optimal threshold percentage to call future predicted observations a 1. I see that this threshold always matches the percentage of observations equal to 1 in my original data. Is there any conceptual explanation for this?


texts: How to normalize results of Singular Value Decomposition (SVD) between 0 and 1?

texts:
I'm building a recommender system and using SVD as one of the preprocessing techniques.
However, I want to normalize all my preprocessed data between 0 and 1 because all of my similarity measures (cosine, pearson, euclidean) depend on that assumption.

After I take the SVD (A = USV^T), is there a standard way to normalize the matrix 'A' between 0 and 1? Thanks!

Edit: I want all of my similarity measurements to give results between 0 and 1 and my normalized euclidean distance in particular fails if the input matrix does not have values between 0 and 1.


texts: How to resolve the instability of average reward per episode in training of DQN (Deep Q-Network)?

texts:
what is shown when  average reward per episode in training is unstable?
If there is big difference between average reward per episode and final reward by test section, what we can say?
For instance in paper of Atari by deep mind, what is determine by it in figure 2. left? What is difference between figure 2 left and right?

Figure 2: The two plots on the left show average reward per episode on Breakout and Seaquest respectively during training. The statistics were computed by running an -greedy policy with  = 0.05 for 10000 steps. The two plots on the right show the average maximum predicted action-value of a held out set of states on Breakout and Seaquest respectively. One epoch corresponds to 50000 minibatch weight updates or roughly 30 minutes of training time.


texts: How to update bias and bias's weight using backpropagation algorithm

texts:
I'm writing my own training algorithm, but I don't know how to set the bias weight.

Have I to set bias in any layer?

Must the bias weight, be updated in every layer?


texts:
pretty simple question here but just can't seem to find the answer in the normally great documentation for sklearn.

I am working with binary classifiers, but we can just assume i am using LogisticRegression, and I was wondering if there is a general way to have the classifier select, say only 10 best (most sure) data points?

For example, say I train a set with 500K data points, and my test set has 10K lines, and out of the 10K, I just want to choose 10 that have the highest chance of being true positives.  Does this make sense?

I have read about, and have been playing with the class_weights attribute, which works well for giving more/less weight to each of the binary outcome classes, but its not quite working for what I want in that it always give different number of position predictions, and I can't really tell how sure the classifier is about each one of those.


texts:
There are several classic datasets for machine learning classification/regression tasks. The most popular are:

Iris Flower Data Set;
Titanic Data Set;
Motor Trend Cars;
etc.

But does anyone know similar datasets for networks analysis / graph theory? More concrete - I'm looking for Gold standard datasets for comparing/evaluating/learning:

centrality measures;
network clustering algorithms.

I don't need a huge list of publicly available networks/graphs, but a couple of actually must-know datasets.

EDIT:

It's quite difficult to provide exact features for "gold standard dataset", but here are some thoughts. I think, real classic dataset should satisfy these criteria:

Multiple references in articles and textbooks;
Inclusion in well-known network analysis software packages;
Sufficient time of existence;
Usage in a number of courses on graph analysis.

Concerning my field of interest, I also need labeled classes for vertices and/or precomputed (or predefined) "authority scores" (i.e. centrality estimates). After asking this question I continued searching, and here are some suitable examples:

Zachary's Karate Club: introduced in 1977, cited more than 1.5k times (according to Google Scholar), vertexes have attribute Faction (which can be used for clustering).
Erdos Collaboration Network: unfortunately, I haven't find this network in form of data-file, but it's rather famous, and if someone will enrich network with  mathematicians' specialisations data, it also could be used for testing clustering algorithms.


texts:
I evaluate results of an algorithm from analysis of acoustical signal depending on SNR. Algorithm finds right frequencies in spectrum.

The kind of finding of TPs, FPs and FNs (for every time slot)

if the right frequency was detected it's a true positive
if wrong frequency was detected it's a false positive
if no frequency was detected it's a false negative

The variable value is an addition to main signal of SNR-signal from -60 dB to +60 dB.
My problem/question is about my Precision and recall results.
Precision: TP/(TP+FP)
Recall: TP/(TP+FN)
If I'm looking for precision and recall diagrams in several papers/ scripts - I always see a similar look of diagram (one falling curve from upper left edge to lower right edge).
In my case, I get a growing precision and recall (both) from left to right.
Actually, is that I waiting for:  with growing SNR > main-acoustic will be less disturbed > algorithm can find a right frequency > more TPs, less FPs and FNs
I attach some diagrams of TP/FP/FN values, precision & recall to SNR diagram,  precision & recall curve and F-Measure curve. Maybe you can help me to explain what is wrong or maybe is not wrong?


texts: Multiple Input (Binary) and Single Ouput: How to calculate correlations?

texts:
I have a dataset with multiple criteria which are either "passed" (green) or "not passed" (red) and an output (passed or not passed) -> see table.
Also the criteria have a chronological order so first criteria 1 then criteria 2 and so on..
My question is: Which algorithms/approaches could be used to gain insights into dependencies/correlations between the inputs and the output? I have a lot of input criteria and just a single output.


texts: I am doing classification using SVM for two classes , is this result good and what does these values refer to?

texts:
precision    recall  f1-score   support

           0       0.92      1.00      0.96        12
           1       1.00      0.98      0.99        52

    accuracy                           0.98        64
   macro avg       0.96      0.99      0.98        64
weighted avg       0.99      0.98      0.98        64

what does these values refer to?


texts: XGBoost Feature Importance, Permutation Importance, and Model Evaluation Criteria

texts:
I have built an XGBoost classification model in Python on an imbalanced dataset (~1 million positive values and ~12 million negative values), where the features are binary user interaction with web page elements (e.g. did the user scroll to reviews or not) and the target is a binary retail action. My ultimate goal was not so much to achieve a model with an optimal decision rule performance as to understand which user actions/features are important in determining the positive retail action.

Now, I have read quite a bit in forums and literature about evaluating/optimizing an XGBoost model and subsequent decision rule, which I assume is required before achieving my ultimate goal. It seems that there are a lot of different ways to evaluate the decision rule part (e.g.  Area Under the Precision Recall Curve, AUROC, etc) and the model (e.g. log-loss). I believe that both AUC and log-loss evaluation methods are insensitive to class balance, so I don't believe that is a concern. However, I am not quite sure which evaluation method is most appropriate in achieving my ultimate goal, and I would appreciate some guidance from someone with more experience in these matters.

Edit: I did also try permutation importance on my XGBoost model as suggested in an answer. I saw pretty similar results to XGBoost's native feature importance. Should I now trust the permutation importance, or should I try to optimize the model by some evaluation criteria and then use XGBoost's native feature importance or permutation importance? In other words, do I need to have a reasonable model by some evaluation criteria before trusting feature importance or permutation importance?


texts: How to interpretate multiple histograms corresponding to each feature in multiple linear regression for relationship?

texts:
Used matplotlib to plot the histograms for each feature in  Boston dataset available in scikitlearn library.
How to interpretate the histograms to determine the correlation or significance of that feature in model training?


texts: Statistical comparison of 2 small data sets for 2X increase in the population mean

texts:
I am trying to determine whether or not we are 90% confident that the mean of a proposed population is at least 2 times that of the mean of the incumbant population based on samples from each population which is all the data I have right now. Here are the data.

incumbantvalues = (7.3, 8.4, 8.4, 8.5, 8.7, 9.1, 9.8, 11.0, 11.1, 11.9)

proposedvalues =  (17.3, 17.9, 19.2, 20.3, 20.5, 20.6, 21.1, 21.2, 21.3, 21.7)

I have no idea if either population is or will be normal.

The ratio of the sample means does exceed 2.0 but how does that translate to confidence that the proposed population mean will be at least twice that of the mean of the incumbant population with 90% confidence ?

Can re-sampling (bootstrapping with replacement) help answer this question ?


texts: Text Classification : Classifying N classes vs rest of the classes

texts:
Apologies if this is naive, I am fairly new to the domain.
I have a requirement where I am trying to classify 2 types of text data, i.e, I have got 2 classes to classify my data upon. I am able to get acceptable results for them using word vectors, dimensionality reduction and then putting the data through to LinearSVC for classification. But my model is biased towards sentences with longer lengths. I know tfidf can help in this, but is there a way to apply that with word vectors, I dont want to lose its ability to predict on unseen similar data?
A follow up problem to this is, when there is data from some other class apart from the 2 classes that I have trained my data upon. I would like my model to be able to predict that the data doesn't belong to any of the 2 classes. Currently, it just predicts 1 of the 2 classes randomly.
I was thinking if I should first put my data through topic modelling, which would give an idea on what topic the text data is. Based on keywords from topic modelling, we can detect if my text data belongs to one of the 2 classes or not and then put it to my classification model for final prediction. But this doesn't seems very clean and I can see it failing as there's too much dependency on keywords generated by topic modelling. Is there any other finer and better way to do this?


texts: find monthly climatology for a particular month

texts:
I have a 10-year sst monthly data. I want to plot the interannual monthly climatology for the month of October. By using the code I have for python with me I am able to plot the interannual monthly climatology for all the 12 months. If anyone can help me by going over my code and say where I have to put index so that I can have monthly climatology only for the month of October it will be much appreciated. Below is my code:

import xarray as xr
import numpy as np
import matplotlib.pyplot as plt
#time_slice = slice(9,10)
lat_slice = slice(30, 0)
lon_slice = slice(40, 80)
nc = xr.open_dataset('E:/X49.37.14.247.143.9.43.4.nc')
np.disp(nc)
ds = nc.groupby('time.month', squeeze = False).mean('time')
dt = ds.sst.sel(lat = lat_slice,lon =lon_slice)
np.disp(dt)
dt.plot.pcolormesh(x = 'lon', y = 'lat', col = 'month', col_wrap =3)


texts: Calculating accuracy for each category in a multi-lable classification problem

texts:
I need to calculate the accuracy for each category (NOT the overall accuracy) in a multi-lable classification problem. It is easy to find the precision, recall and F-score for each category using classification_report from sckkit-learn library. There are 13 categories distributed as follows:
                   precision    recall  f1-score   support

     Category 1       0.58      0.48      0.53       244
     Category 2       0.91      0.85      0.88       728
     Category 3       0.90      0.92      0.91      1319
     Category 4       0.70      0.55      0.62       533
     Category 5       1.00      0.10      0.18        20
     Category 6       0.94      0.84      0.89      2038
     Category 7       0.83      0.78      0.80      1930
     Category 8       0.85      0.44      0.58       113
     Category 9       0.88      0.87      0.87      1329
     Category 10      0.79      0.54      0.64        61
     Category 11      0.81      0.77      0.79       562
     Category 12      0.71      0.62      0.66       416
     Category 13      0.76      0.60      0.67       500

      micro avg       0.86      0.78      0.82      9793
      macro avg       0.82      0.64      0.69      9793
   weighted avg       0.85      0.78      0.81      9793
    samples avg       0.86      0.82      0.83      9793

I know that accuracy can be found as follows:     Accuracy=(TP+TN)/(TP+TN+FP+FN) but finding TP and TN for this multi-lable classifcation problem was an issue for me.
There is a similar question to this one on stackoverflow https://stackoverflow.com/questions/42041078/calculating-accuracy-from-precision-recall-f1-score-scikit-learn
but for binary-classification problems only.
Any Ideas? Thanks


texts: Best network structure for unsupervised learning

texts: Currently using networkX to build a probability based network for unsupervised learning, it is a basic reader understanding relational connectivity through probabilities, network slows significantly after about 40 pages (8000 words) any suggestions for a better network platform would be much appreciated.


texts: Difference between Blocking and Clustering?

texts:
What is the difference between blocking and clustering?

As far as I know clustering originates from the Machine Learning branch and refers to methods of grouping data points together in a "We and them" way: Points in the same cluster are related, points of different clusters are not.

Blocking, however, seems to come from the statistics and picked up later in data analysis. I encountered Blocking/Indexing etc. primarily in Record Linkage, where it is used to partition data into blocks before they are analyzed.

One could even have both blocking and clustering methods used in Record Linkage. However, I find, I don't understand their difference too well yet and thus would like to have a different perspective on that matter.

My interpretation thus far: The goal of blocking and clustering is different:

Blocking is used to throw potential unnecessary data out - distinguishing between important and non-important data.

Clustering however does make that distinction, all data points are important. is used for classification purposed - identifying all data.


texts: is there an adequate number of levels of categorical variables?

texts:
I have a project that I'm working on. The dataset contains many categorical variables and some of them have too many levels (+100). My question is : is there any advice to know the "adequate" number of levels of a variable ? is it based on the number of levels of other variables ? (for example most variables have between 10 and 30 levels and one or two variables have 80 100 levels).
For the variables that contain too many levels, I want to take 80% of most frequent levels and put the 20% into a new level "others" but I don't know at which number of levels I should stop (for example :
var 1 : 70 levels, var 2 : 100 level, var 3 : 13, var 4 : 30, var 5 : 60, should I apply the 80-20 method starting from 60? 70? 100?)
I don't know if I'm being clear but I hope you understand


texts: Dealing with issues in "test" predictons for single "items" (null values, standardization in place, etc)

texts:
I know this is kind of a broad question but I have tried to scour both this forum and the internet in general to no avail for this particular situation. So imagine I have a model trained for which, though the data initially might not be complete and clean, I took steps to make the data complaint with the model requirements (no outliers where appropriate, de-skewed if necessary, normalized if necessary, null values imputed appropriately). This is done in a cross validation framework. All this stuff works and is absolutely fine when tuning the model but I run into problems when I try to make a single prediction out of it (meaning I have a single "test" record - think web service with some fields that can be null). In fact, null values generally need a dataset to refer to for filling, as well as for the normalization/outlier procedures.

Initially I thought about linking such "test" record to a portion of the "train" dataset so that I would not run into this problem (such problems would be resolved) but at that point other issues would arise: how would I choose such dataset? if I used the most recent data, would I bias it somehow? and using the whole dataset is impractical as well as potentially unfeasible when dealing with "big" data.

do you happen to know whether there are some best practices on the topic or could you refer me the themes/keywords that deal with these issues?

p.s.: for the relevance of the problem, the null values most likely will remain there (I have no way of forcing them beforehand in the web application in order to have a smoother user experience)


texts:
I am trying to predict a time serie from another one. My approach is based on a moving windows. I predict the output value of the serie from the following features: the previous value and the 6 past values of the source serie.

Is it usefull to add the previous value of the time serie ?

I feel like I don't use all the information contained in the curve to predict futures values. But I don't see how it would be possible to use all previous data to predict a value (first, the number of features would be growing trough time...).

What are the caveats of a 6 month time-window approach ?

Is there any paper about differents method of feature selection for time-series ?


texts:
I'm using CNN, RNN and OpenCV to identify people and cars within images, once I identify several images I'm cropping them and dividing them in cars and people.

I would like to group all same-looking cars and all same-looking people. What would be the best approach to do so?

I'm recollecting these images from an IP camera, so I cannot base my code in colors, because the light is not the same when is day and night. Plus the cars aren't always looking at the same direction. I guess SVM would work nice, but I would need classify some data by hand first. I'm looking for something to make clusters of similar images without a supervising method.

Would something like k-means work?


texts: Pytorch: Reduce forward prediction dimensions of GRU network / Improving Network Architecture

texts:
I am currently working on a GRU network to predict a time series, please note that I am completely new to machine learning and pytorch. Also I have never had a formal education in programming. This should however not play a role in my problem.
Essentially I have N training data points in my training set. Each consists of n_l time stamps from a time series. Each of these time stamps contains n_feat data points (e.g. for n_feat = 4: temperature, pressure, humidity and wind speed). Therefore my input tensor to the network is of shape torch.Size([N, n_l, n_feat]). To the best of my knowledge (cf. documentation) a GRU net will now map this onto a tensor of shape torch.Size([N,n_l, output_dim]). In case my output_dim is not equal to n_feat, I can then use a linear network to map this tensor onto the desired shape torch.Size([N, n_l, n_feat]).
This is what my code (a single layered GRU) is successfully reproducing. However, I don't want my GRU network to predict the next n_l time stamps, but only the following n_f, i.e. I want a network which maps a tensor of shape torch.Size([N, n_l, n_feat]) to a tensor of shape torch.Size([N, n_f, output_dim]).
My questions are as follows:

Is this fundamentally possible with a single layer GRU network or do GRU networks not have this
functionality? I have tried a plethora of ways without success and from my understanding this should indeed not work.

What would be the best practice to achieve this, in case this behaviour is not
supported? I thought about layering multiple nets to achieve this
(Conv1D -> GRU -> Linear (optional) or GRU -> Conv1D -> Linear (mandatory), prefering the latter variant.

Any help would be greatly appreciated and thanks in advance for your time !


texts: Tool for clustering and cleansing data set

texts:
I have a large-ish data set (400K records) composed of two fields (both strings). I am looking for a tool that will enable me to cluster the data e.g. around the first column, either using exact matches or some kind of string proximity function like Levenshtein distance. I would also like to be able to find all duplicate records and merge them into one.

OpenRefine looks ideal for my purposes but it is so slow when clustering my data or creating a text facet that it is unusable. Apparently this is a known issue.

I looked around but couldn't find another tool that would enable me to explore a data set of this size, cluster, eliminate dupes, look for anomalies, etc. Can anyone recommend something that might fit the bill?


texts: Why gradient boosting uses sampling without replacement?

texts:
In Random Forest each tree is built selecting a sample with replacement (bootstrap). And I assumed that Gradient Boosting's trees were selected with the same sampling technique. (@BenReiniger corrected me).  Here there are the sampling techniques implemented for Catboost

My questions:

Why is Gradient Boosting sampling done without replacement?
Why would it be worst to sample with replacement?
Are there any sampling techniques used in GB that are with replacement?

I quote a paper for SGB:

  Stochastic Gradient Boosting is a randomized version of standard Gradient Boosting algorithm... adding randomness into the tree building procedure by using a subsampling of the full dataset. For each
  iteration of the boosting process, the sampling algorithm of SGB selects random s·N objects without replacement and uniformly


texts: I ran some text classification on text data. When I tried to save the SGD classifier object, I was surprised to find it is more than 4GBs (even though my input is ~600 000 rows, 700MB). What's in that object that makes it so huge? Any thoughts?


texts: How to extract and classify data from a column in excel?

texts:
I have a column in an Excel sheet that contains a lot of data separated by || delimiters. The data can be classified to some classes like Entity, IFSC codes, transaction reference id, etc.

A single cell looks like this:

EFT INCOMING||0141201||NHFI0141201||UTR||SBIN118121948660 M S||some-name ||some-purpose||TRN REF NO:a1b2c3d4e5

Not every cell has the same number of classes or even the same type of classes.
Another example:

COMM/CHARGES/FEES||CHECK/REF.6546644473||BILPAY CCTY BEARING C||00.00||00012||18031358||BLPY||TRN REF NO:a1b2c3d4e5

I tried extracting this information using regular expressions and am able to get a list of ref-ids or IFSC codes extracted as a single list. But I need to break a cell to multiple cells with individual information. If some cell does not has that class data, it shall remain blank.

I also tried using named entity recognition but the same problem arises, I get the list of entities as output, not the breakdown.

Please help me in identifying what kind of problem this is? A text classification? And what would be the approach to solve it?


texts:
I was trying to install jupyter package for anaconda in my current environment but constantly getting the following error.

  Preparing transaction: done Verifying transaction: failed

  CondaVerificationError: The package for ipython located at
  D:\Anaconda\pkgs\ipython-7.1.1-py36h39e3cac_0 appears to be corrupted.
  The path 'Lib/site-packages/IPython/lib/tests/test.wav' specified in
  the package manifest cannot be found.

  CondaVerificationError: The package for notebook located at
  D:\Anaconda\pkgs\notebook-5.7.2-py36_0 appears to be corrupted. The
  path
  'Lib/site-packages/notebook/static/components/MathJax/extensions/a11y/invalid_keypress.mp3'
  specified in the package manifest cannot be found.

I have Googled many Q&A for this problem, mostly I found:

conda clean --packages --tarballs
conda clean --all

After trying both I tried the installation again, but none worked...

Any solution for this?


texts:
I want to do the following project and I think the best way is using tensorflow/keras: I have photos as images from a list with prices of products and I need classify these text objects from the images such that i identify the product name and the price.
So I think it is different from normal Image classification since my objects are text objects?
First, I thought it is like object detection in images but I think i need more to take these texts to another neural network?


texts: how to learn from unlabeled samples but labeled group of samples?

texts:
I'm trying to perform anomaly detection on the open data from citibike.
They are giving bikeshare trips for the past 30+ months, as well as monthly reports. In those reports they say how many bikes have been repaired each month.

The samples I am building are a sample by day and by bike. I actually don't have label for those samples, since I don't know which bike has been repaired which day. But I know that by classifying each sample normal or anormal, I can sum the number of bikes that have been classified anormal during a month and compare that number to the monthly report.

I want to know how one usually deal with it, or how this is called so I can read research paper on the subject.

Exemple of samples :

bikeid, day,         feature1, feature2...
1,      2016-01-01,  0.6,      -0.2
2,      2016-01-01,  0.5,      -0.8
1,      2016-01-02,  0.7,      -0.1
2,      2016-01-02,  0.9,       1
   ...
1,      2016-01-31, -0.32,     -0.45
2,      2016-01-31, -0.5,      -0.8

Example of label: 3456 bicycle repairs in January.

But the shape of the data is irrelevant, what is important is that the labels are not about one sample but a group of samples.


texts: Will the performance of my NER model improve?

texts:
I am training a spacy model from scratch by creating a dataset of my own with format spacy needs it to be in, the model is an NER model and the entity i am trying to recognize is Food items. I have created a dataset with 263 rows and after training the spacy model from scratch on this dataset, my model is performing great (I am getting around 80% accuracy) and this accuracy may not look a lot but it is better and I am being able to do my task a lot better now.

Now I want to improve my model even further by increasing train data. For increasing the train data I am thinking of using the rows I am sending as test and checking manually if spacy recognised each entity correctly or not and if all entities in a sentence are recognised correctly use this sentence in my training set.

My question is will this method improve my model in any way ?
Or in general if we add data which is correctly classified by our model in training set, will the performance of model improve to some extent?


texts: How to scale exponential data for a regression problem?

texts: I understand that I should be scaling features between (0, 1) before feeding them into a neural network. However, what happens if future data could be larger than my current training data? For instance, if I am training a RNN on time-series data to perform demand forecasting, the product I am forecasting may be in a growth stage that will yield higher demand numbers in the future months than the past months. Is there a way to normalize data but still allow for larger values in the future?


texts: Neural network is not giving the expected output after training in Python

texts:
My neural network is not giving the expected output after training in Python. Is there any error in the code? Is there any way to reduce the mean squared error (MSE)?

I tried to train (Run the program) the network repeatedly but it is not learning, instead it is giving the same MSE and output.

Here is the Data I used:

https://drive.google.com/open?id=1GLm87-5E_6YhUIPZ_CtQLV9F9wcGaTj2

Here is my code:

#load and evaluate a saved model
from numpy import loadtxt
from tensorflow.keras.models import load_model

# load model
model = load_model('ANNnew.h5')
# summarize model.
model.summary()
#Model starts
import numpy as np
import pandas as pd
from tensorflow.keras.layers import Dense, Activation
from tensorflow.keras.models import Sequential
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
# Importing the dataset
X = pd.read_excel(r"C:\filelocation\Data.xlsx","Sheet1").values
y = pd.read_excel(r"C:\filelocation\Data.xlsx","Sheet2").values

# Splitting the dataset into the Training set and Test set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.08, random_state = 0)

# Feature Scaling
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)

# Initialising the ANN
model = Sequential()

# Adding the input layer and the first hidden layer
model.add(Dense(32, activation = 'tanh', input_dim = 4))

# Adding the second hidden layer
model.add(Dense(units = 18, activation = 'tanh'))

# Adding the third hidden layer
model.add(Dense(units = 32, activation = 'tanh'))

#model.add(Dense(1))
model.add(Dense(units = 1))

# Compiling the ANN
model.compile(optimizer = 'adam', loss = 'mean_squared_error')

# Fitting the ANN to the Training set
model.fit(X_train, y_train, batch_size = 100, epochs = 1000)

y_pred = model.predict(X_test)
for i in range(5):
    print('%s => %d (expected %s)' % (X[i].tolist(), y_pred[i], y[i].tolist()))

plt.plot(y_test, color = 'red', label = 'Test data')
plt.plot(y_pred, color = 'blue', label = 'Predicted data')
plt.title('Prediction')
plt.legend()
plt.show()

# save model and architecture to single file
model.save("ANNnew.h5")
print("Saved model to disk")


texts: How to create dictionary with multiple keys from dataframe in python?

texts:
I have a pandas dataframe as follows, I want to convert it to a dictionary format with 2 keys as shown:

    id              name                        energy             fibre
0   11005   4-Grain Flakes                          1404            11.5
1   35146   4-Grain Flakes, Gluten Free             1569             6.1
2   32570   4-Grain Flakes, Riihikosken Vehnämylly  1443            11.2

I am expecting the result to be of

 nutritionValues = {
  ('4-Grain Flakes', 'id'): 11005,
  ('4-Grain Flakes', 'energy'): 1404,
  ('4-Grain Flakes', 'fibre'):  11.5,
  ('4-Grain Flakes, Gluten Free', 'id'): 11005,
  ('4-Grain Flakes, Gluten Free', 'energy'): 1569,
  ('4-Grain Flakes, Gluten Free', 'fibre'):  6.1,
  ('4-Grain Flakes, Riihikosken Vehnämylly', 'id'): 32570,
  ('4-Grain Flakes, Riihikosken Vehnämylly', 'energy'): 1443,
  ('4-Grain Flakes, Riihikosken Vehnämylly', 'fibre'):  11.2}

foods, fiber = multidict({
  '4-Grain Flakes': 11.5,
  '4-Grain Flakes, Gluten Free':   6.1,
  '4-Grain Flakes, Riihikosken Vehnämylly':   11.2})

How can I achieve this?


texts: I am not getting classification output by predict_generator()

texts:
I am trying to classify pre-downloaded images from my dataset to "Rock" , "Paper" , "Scissor" classifications but I am getting outputs as numbers.

I have already divided the dataset to Train folder where 3 folders with the names of the categories exist.

Is it because I am not using a convolutional neural network?

I don't understand what to do here .

Code:

from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Dense, Flatten
import numpy as np

train_directory = 'D:\D_data\Rock_Paper_Scissors\Train'
training_datgagen = ImageDataGenerator(rescale = 1./255)
training_generator = training_datgagen.flow_from_directory(
    train_directory,
    target_size = (28,28),
    class_mode = 'categorical', classes = ["Rock", "Paper" , "Scissor"])

validation_directory = 'D:\D_data\Rock_Paper_Scissors\Train'
validation_datagen = ImageDataGenerator(rescale= 1./255)
validation_generator = validation_datagen.flow_from_directory(
    validation_directory,
    target_size = (28,28),
    class_mode = 'categorical',
    classes = ["Rock", "Paper" , "Scissor"]
    )

model = Sequential()
model.add(Flatten(input_shape = (28,28,3)))
model.add(Dense(128,activation = 'relu'))
model.add(Dense(64, activation = 'relu'))
model.add(Dense(16, activation = 'relu'))
model.add(Dense(3, activation = 'softmax'))

model.compile(optimizer = 'adam', loss = 'categorical_crossentropy',metrics = ['accuracy'],)
filenames = validation_generator.filenames
nb_samples = len(filenames)
desired_batch_size = 1

model.fit_generator(training_generator,epochs=20,validation_data = validation_generator)
predict = model.predict_generator(validation_generator, steps = np.ceil(nb_samples/desired_batch_size))

print(predict)

Output:

[[2.2792105e-01 7.7206087e-01 1.8017196e-05]
 [2.8221733e-03 9.9715865e-01 1.9176909e-05]
 [5.1788869e-04 9.9948126e-01 8.8551565e-07]
 ...
 [9.9283433e-01 7.1645100e-03 1.1474091e-06]
 [9.9999678e-01 3.2666808e-06 4.5816462e-09]
 [9.9640071e-01 3.5960779e-03 3.1497514e-06]]


texts: Decision tree to get difference in rates in two groups?

texts:
I have two sample groups of customers, each customer has 100s of features. For a single sample, i would use Decision Trees to find sub-groups that have a high churn rate. Thats easy.

However, my requirement is: between two samples (below), find segment(s) such that in one sample its churn rate is high and in the other, it is low. In other words, find a sub-group which has the highest difference in churn rate.

What is an appropriate algorithm to solve this?

Thanks.


texts:
I am trying to figure out, how I could build a model which takes an input image of varying illuminations and outputs an image which has adjusted illuminations by a given label-image.
I found out that regression might be the right keyword to search for, unfortunately I haven't found anything useful yet (neural style transfer, regression). The closest thing would be this here.
However, according to this git it will takes the image coordinates to "regress" towards the label pixels.
What I have so far is something like this:
input_shape = np.shape(img) # array of(300, 300, 3)

model = tf.keras.models.Sequential([
    # Note the input shape is the desired size of the image 300x300 with 3 bytes color
    tf.keras.layers.Conv2D(64, (3,3), activation='relu', input_shape=input_shape),
    tf.keras.layers.MaxPooling2D(2, 2),

    tf.keras.layers.Conv2D(128, (3,3), activation='relu'),
    tf.keras.layers.MaxPooling2D(2,2),

    # Flatten the results to feed into a DNN
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dropout(0.5),

    tf.keras.layers.Dense(512, activation='relu'),
    tf.keras.layers.Dense(num_img_pixels, activation='relu')
])

model.compile(loss='mean_squared_error', optimizer='adam')
# x are training images, a tensor with 4 dims [batch_size, 300, 300, 3]
# y are training labels a tensor with [batch_size, 300*300*3]
model.fit(x, y, ...)

result = model.predict([test_img])
result = np.reshape(result[0], input_shape)

Due to the lack of my knowledge I would like to know if there is a similar problem out there and how I could solve that.
Thanks in advance.


texts: Relationships between groups of features against independent variables

texts:
I have several groups of features that I'd like to test against independent variables. The idea is to find which groups tend to be associated with a specific value of an independent variable.
Let's take the following example where s are samples, f are features, i are independent variables associated with each s.
        s1    s2    s3     s4 ....
    f1  0.3   0.9   0.7    0.8
    f2  ...
    f3  ...
    f4  ...
    f5  ...
    i1  low   low   med    high
    i2  0.9   1.6   2.3    10.5

Features f1, f2, f3 belong to group1 and f4,f5 belong to group2. If I wanted to find whether a given feature tended to be associated with a given independent variable, I could regress each feature vs i2 or an encoded i1 and test whether there is an association between feature and independent variable.
But now I'm wondering, is it possible to test whether a group of features tends to be associated with an independent variable? I'm not sure how to approach this problem.
One idea is to test each independent variable against all features in each group using multilinear regression. The model to regress would contain only features in each group separately, so in this case we would have $2*2$ models in total (for group1 and group2, and for 2 independent variables).
I have a feeling that this could also be formulated as a classification problem, but not really sure how.


texts: How mAP is unfair evaluation metric for Object Detection?

texts:
The following figure is from the last page in YOLOv3 paper highlighting how mAP is unfair metric for evaluating Object Detectors:

The figure shows two hypothetical Object Detector results which the author say they give the same perfect mAP, while visually the first detector is clearly more accurate than the other.

According to my understanding, the two detectors do not give the same mAP. This is how I calculate it for each detector:

Detector 1, 'Dog' class AP table:
 ______________________________________
| Object  | True? | Precision | Recall |
|_________|_______|___________|________|
| Dog_99% | Yes   |     1     |    1   |
|_________|_______|___________|________|
Hence, AP_dog = 1

Detector 1, 'Person' class AP table:
 ________________________________________
| Object    | True? | Precision | Recall |
|___________|_______|___________|________|
|Person_99% | Yes   |     1     |    1   |
|___________|_______|___________|________|
Hence, AP_person = 1
And by continuing doing so for the other 7 classes in the dataset, mAP=1.

Detector 2, 'Dog' class AP table:
 ______________________________________
| Object  | True? | Precision | Recall |
|_________|_______|___________|________|
| Dog_48% | Yes   |     1     |    1   |
|_________|_______|___________|________|
Hence, AP_dog = 1

Detector 2, 'Bird' class AP table:
 _______________________________________
| Object   | True? | Precision | Recall |
|__________|_______|___________|________|
| Bird_90% | Yes   |     1     |    1   |
| Bird_89% | No    |     0.5   |    1   |
|__________|_______|___________|________|
Hence, AP_bird = 0.75
And by continuing doing so for the other 7 classes in the dataset, mAP is less than 1 because AP for at least one class is less than one (AP_bird).

Hence, according to my understanding mAP for the first detector is 1, and for the second detector is less than 1. What is the mistake I'm doing in the calculation? Or is there some assumption in the paper that I'm not considering?


texts: Why there is very large difference between cross validation scores?

texts:
I have a very simple regression model and I am doing the cross validation. When cv=10 the highest score i got is 60.3 and lowest is -9.7 which is useless.
Average will be 30.

No of row data set= 658


texts: Why the Silhouette Score and optimal number of Cluster changes when using 2D and 3D data?

texts:
I am experimenting with Kmeans clustering. My data (vectors) was in 300 dimensions which I am converting into 2D and 3D using PCA. Now, to find the optimal number of clusters, I used the Silhouette score. However, for 2D the best Silhouette score is showing for 3 clusters (silhouette score = 0.45), and for 3D it is showing 9 clusters (silhouette score = 0.3861).
I want to know whether it is normal? If yes, what is the reason for this? What should I choose 2D or 3D?
Also, the reason for experimenting with 2D and 3D is because I wanted to plot the 3D graph using seaborn.


texts: How to put the variable names of pandas data frame on Diagonal of Seaborn PairGrid?

texts:
The given below code is not working now(earlier it was working) to put the column names on the diagonal of Seaborn PairGrid.

import matplotlib.pyplot as plt
import seaborn as sns
iris = sns.load_dataset('iris')

def diagfunc(x, **kws):
  ax = plt.gca()
  ax.annotate(x.name, xy=(0.05, 0.9), xycoords=ax.transAxes)

sns.PairGrid(iris).map_diag(diagfunc)

At present it showing this error!

<ipython-input-137-f4d9b71087cb> in diagfunc(x, **kws)
      5 def diagfunc(x, **kws):
      6     ax = plt.gca()
----> 7     ax.annotate(x.name, xy=(0.05, 0.9), xycoords=ax.transAxes)
      8
      9 sns.PairGrid(iris).map_diag(diagfunc)

AttributeError: 'numpy.ndarray' object has no attribute 'name'

Can anyone help me how to put the column name in diagonal?


texts: How to perform one hot encoding on multiple categorical columns

texts:
I am trying to perform one-hot encoding on some categorical columns. From the tutorial I am following, I am supposed to do LabelEncoding before One hot encoding. I have successfully performed the labelencoding as shown below

#categorical data
categorical_cols = ['a', 'b', 'c', 'd']
from sklearn.preprocessing import LabelEncoder
# instantiate labelencoder object
le = LabelEncoder()
# apply le on categorical feature columns
data[categorical_cols] = data[categorical_cols].apply(lambda col: le.fit_transform(col))

Now I am stuck with how to perform one hot encoding and then join the encoded columns to the dataframe (data).

Please how do I do this?


texts: Are neural networks able to deal with non-normalised inputs?

texts:
All the techniques/models that I have learnt so far for deep learning start with some sort of normalization to the features, for example gaussian method, minmax scaling, robust scaling, batch normalization, instance normalization.

Are there any techniques to run neural networks without normalization so that the network can see (in absolute values) the magnitude of the value and respond according to that instead of normalized values? Will there be exploding/vanishing gradient issues if I don't normalize my data?

For example, if I am training a custom LSTM network for multivariate time series data, the input dimension for a feature vector $x$ is all the values from $t-n$ to $t$, where $n$ is the number of time steps and the output vector is the value at $t+1$. Is there any need for normalization in this case?


texts:
A batch of semantic objects in the image (lesions in CT scans) are represented in feature space, $X_{B \times C}$. I want to represent the whole batch in a single vector, $1 \times C$, in order to predict the image class from this vector. I want to find an affinity for each feature. This affinity matrix is $A_{C_2\times C}$, and I take a matrix-matrix product:  $X_2 =A^{T}X$. Then, I take matrix-vector product of $X_2$ and affinity vector $a_{C_2 \times 1}$ to get $x_3= X_2 a$ which has $C$ dimensions and hence the representation I want.
My question is: is this a legit method of obtaining an affinity among objects? I plotted $X_2$ after 1 and 100 epochs, it looks good. Overall accuracy is also good, around $90%$, but I have doubts about the method-never did anything like this before.
In the first step, for each feature $\mathbf{x}_j$ I learn $C_2$ affinities, so I map each feature into the affinity space. In the next step, I scale these affinities with vector $a$ and map the features back into the feature space. Is this OK?
$X_2$ matrix" />