texts: Assessing performance of an agent based on commission rate, market share and revenue

texts:
I have a set of data for agents selling properties (apartments) for a company in different states. The company would like to assess the performance of the different agents given the following:

Number of apartments sold
Market share (%)
Total Revenue
Commission Rate (%)

In each state, there are a number of agents working for the company. Of course we would like to keep the agents that are bringing more revenue, selling more apartments, having greater market share and lower commission rate. However I am not sure how to tackle this four dimensional problem in the simplest possible way. I would like to give a score of 10 for each of the agents based on his performance. After that I will like to discard some of the under-performing agents in each state. Any clue from where to start?


texts:
How exactly does k-means clustering for categorical data work?
I have a dataset which has several categorical features that can have 2,3,4,..,n values. I could one hot encode them, but I'm not sure if it even makes sense because k-means uses a distance metric.
I looked into k-prototype clustering, but tried it on my dataset, but plotting it does not make sense when the y-axis and y-axis are just integer values that are small.


texts:
I'm trying to build a toy recommendation engine to wrap my mind around Singular Value Decomposition (SVD). I've read enough content to understand the motivations and intuition behind the actual decomposition of the matrix A (a user x movie matrix).

I need to know more about what goes on after that.

from numpy.linalg import svd
import numpy as np

A = np.matrix([
  [0, 0, 0, 4, 5],
  [0, 4, 3, 0, 0],
  ...
])
U, S, V = svd(A)

k = 5 #dimension reduction
A_k = U[:, :k] * np.diag(S[:k]) * V[:k, :]

Three Questions:

Do the values of matrix A_k represent the the predicted/approximate ratings?
What role/ what steps does cosine similarity play in the recommendation?
And finally I'm using Mean Absolute Error (MAE) to calculate my error. But what I'm values am I comparing? Something like MAE(A, A_k) or something else?


texts: Automatically assess training data quality for land cover classification system

texts:
I am working on a Land cover classification system, wherein, Sentinel-hub imagery is being used to categorize the land cover by using a time series of multispectral imagery. Training data is being captured manually by drawing polygons of a particular Land cover type (using QGIS).

Again, since the polygons are being captured manually, there is a slight chance that a polygon has been labeled incorrectly or it comprises of more than 2 land cover classes. I need a way to filter out the polygons which have a high probability of being mislabelled so that the prediction-stage model receives only the most reliable samples for training. So in a way, it does an Auto Quality Check. Are there any metrics or approaches (preferably unsupervised) that can be used for this use-case?


texts: Why are parameter updates downscaled by uncentered variance (instead of centered variance) in Adam optimizer?

texts:
In Adam optimizer algorithm, parameter updates are computed as follows:

$\theta_t \leftarrow \theta_{t-1} - \alpha \frac{\hat{m}_t}{\sqrt{\hat{v}_t}+\epsilon}$

Where $\hat{m}_t$ is a bias-corrected moving average of gradient means:

$\hat{m}_t \propto \beta_1 m_{t-1} + (1 - \beta_1) g_t$

And $\hat{v}_t$ is a bias-corrected moving average of uncentered variance:

$\hat{v}_t \propto \beta_2 m_{t-1} + (1 - \beta_2) g_t^2$

I understand the idea that, if the mean is large compared to the variance, gradients can be trusted more and you should take larger steps. But it this case, wouldn't it be more logical to use centered variance instead of uncentered variance in the above formula?

I looked for the answer and found this thread but I still wasn't convinced by the answer. Essentially the explanation was that uncentered variance is roughly equal to centered variance because the mean gradient is approximately zero. But if the mean gradient is zero, it means that you are equally likely to move in any direction, so you aren't optimizing anything...

So, why is Adam optimizer using uncentered variance? Why not centered variance which would seem a lot more logical?


texts: How to evaluate RMSE with standard deviation

texts:
I have regression model, where target is between 0 to 1. standard deviation of target is 0.817 and RMSE of model on hold out is 0.52. I am wondering if this good model or not.

Any feedback will be useful

Thanks


texts:
Working on a school project where we have to match some users based on common interests. Assuming I have a list of inputs like this:

Name Interest1 Interest2 Interest3 Interest4 Interest5
Name Interest4 Interest6 Interest7 Interest8 Interest9
Name Interest1 Interest2 Interest4 Interest3 Interest5
Name Interest7 Interest8 Interest9 Interest11 Interest12

And another user comes

Name Interest4 Interest6 Interest13 Interest12 Interest7

Closest match for him would be user 2. If I want to take a ML approach in solving this what algorithm would fit this kind of problem?


texts: gpu pytorch code way slower than cpu code?

texts:
I have the following pytorch code in a jupyter notebook:

import torch
t_cpu = torch.rand(500,500,500)
%timeit t_cpu @ t_cpu

Which outputs:

422 ms ± 3.93 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

And the following code, which took like 100 times as long:

import torch
t_gpu = torch.rand(500,500,500).cuda()
%timeit t_gpu @ t_gpu

which outputs:

31.8 ms ± 550 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Clearly, the reason the gpu version took longer is because it did 1000 loops each.

Why did it do 1000 loops each, given that I didn't specify this at all?

ps. side question: what does the "@ t_cpu" do in "%timeit t_cpu @ t_cpu"?


texts: k-Nearest Neighbours with time series data - how to obtain whole-time-period estimators

texts:
I have a large dataset for the activities performed by multiple staff in a factory over a long period of time - 01/01/2017 - present.  The activities performed by different staff are recorded at each point in time (since they interact with software).  I have tabulated these to record the number of each activities performed by each operator for each day.

My table looks something like this:

Date         Name    Activity  UnitsProcessed   Shift      Team
01/10/2017   MMouse  Soldering   1000           Shift A   Team A
01/10/2017   MMouse  Welding     3000           Shift A   Team A
01/10/2017   DDuck   Soldering   4000           Shift A   Team B
01/10/2017   DDuck   Welding     1000           Shift A   Team B
...
02/11/2017   MMouse  Soldering   1100           Shift A   Team A
02/11/2017   MMouse  Welding     2900           Shift A   Team A
02/11/2017   DDuck   Soldering   4010           Shift A   Team B
02/11/2017   DDuck   Welding     900            Shift A   Team B

Each line represents the activities performed by each operator each day they worked, and the number of units processed during each activity, along with a classifier for the shift and team that the operator was a member of (for simplicity, assume that this does not change over time).

For brevity, I have omitted values from Shift B, but essentially, the team members of Shift B will (usually) work different days from Shift A, but the proportion of tasks performed will usually be similar to the corresponding Team on Shift A - Team A or Team B.

The problem I want to solve is:  I have the date, activity and units processed - as well as various other variables - for another operator, "GOofy".  But I don't know which shift/team Mr GOofy works.  So I am using the data for Mr Mouse and Mr Duck (obviously, in reality, scores of people) to see if I can try to train kNN to correctly guess the Shift and Team pattern for each Name, given the Date, Activity and Units Processed fields, among others.

My thinking is:

a) There is no particular value in keeping Team and Shift separate, so I should create a joined column which combines both

B) If I create separate columns for "Soldering", "Welding" and the other activities, I can have the number of units processed for each day for each activity by date.  Given that the kind of activity performed is (I hypothesise) a strong predictor of the shift/team combination, I should be able then to train to recognise shift/team based on both the date of that line (perhaps converted to a numeric date) and the relative values for each line of soldering, welding, etc.

The problem is that, when extended to the test set, this will only give me predictors for each individual day.  But I suspect that the days worked is also a significant predictor, so I would like to include this as well.  But I'm not sure how to approach this.

The two approaches I can think of are:

Option 1:  To use kNN to generate estimators for shift/team for each operator/day combination, with the x variables being the date, units soldered and units welded per day, and then calculate, for each operator, the proportion of times "Shift A, Team A", "Shift A, Team B", etc were assigned to that operator - so if DDuck is assigned "Shift A, Team A", on 90% of days, it seems most likely that DDuck is indeed "Shift A, Team A".

Option 2:  To somehow create a variable which represents all of the days worked for each operator - although I'm not sure how to do this - and thus create a  set which contains one row for each operator.

Are either of these approaches valid?  Is there a widely-used alternative approach for this?

Note that this is not a duplicate of previous questions relating to kNN and time-series data, since those were concerned with forecasting, whereas I am trying to analyse retrospective data.


texts:
I have clusters as shown in the picture below.

The data is 2d : the two parameters are error and time.
I tried using the following clustering algorithms:
1) kmeans:clusters are spherical.
This algo clustered as follows:.
Kmeans did form clusters as needed.
2)dbscan: problem- I have clusters of varying density. It assigned every data point to noisy cluster I.e. it assigned a label of -1 to every point.
3) GMM: this also did not cluster the data points as needed. It performed clustering similar to kmeans.
4)affinity propagation is throwing a memory error.

Can someone please suggest a way of assigning different labels to each of the visually separable cluster.


texts: Is there a way to recognize which of these scatter plots is "better"?

texts:
I doubt better is the correct adjective, apologies for that. What I mean is this:
I have a set of files (1200~) each paired with a scatterplot image. I need to find a way to classify which data files will have scatter plots that a person would classify as "clearly separated data"(again, not the right words), and which ones as "not clearly separated".
For example:

The first and second scatter plots are examples of data that would end classified as "good" or "clear", while the third one would be classified as "unclear". Are there metrics or distances that could be applied on the data to predict wheter the plots generated by that file would be of either the first kind or the second?

Thanks for taking the time to read this. I'm new to this, and english is not my first language.


texts: Multi-Collinearity in Classification Problems

texts:
I have a "Small data matrix" of scraped data from multiple websites trying to account for sentiment towards certain cellphone models. The matrix includes independent variables such as "count of times X model of cellphone is mentioned", "count of number of times IOS X is mentioned", "count of positive/neutral/and negative expressions found for camera, IOS, performance, etc" (total independent variables are around 50). The dependent variable is a sentiment from 0 to 5 (really dislike to really like) that was manually filled from reading the actual content of each website. I am trying to train this dataset and apply the best model to a bigger matrix. This being a classification problem, how important is to account for collinearity? There are no highly correlated features with my dependent variable, but there are many independent variables highly correlated (>.8 R2) to each other. How important is to filter for these correlated independent variables, and if necessary, how to choose which one to get rid of? I will be using C5, SVM, KNN, and Random Forests to build my model.
Thanks!


texts: What are non-machine-learning methods called?

texts:
For my masters, I'm comparing different approaches to solve a problem (like image segmentation).

I'm comparing machine learning and deep learning approaches to «classical» algorithms (like simple or adaptative threshold).

How would you call these «classical» or «standard » approaches? Like algorithms that don't rely on data but are «hand-designed».
I can't seem to find a correct term.


texts: Detecting outlier with combining two vectors

texts:
I want to combine the following vectors in a way that just the red point (number 7) becomes inconsistent with other points( become an outlier and become distant from other points) and other points become consistent with each other. Please note that I have tested mahalanobis distance and Kullback-Leibler divergence between two vectors but they were not so good and detects.

a=[1.3269   1.3354  1.3318  1.3282  1.34666 1.3460  1.36084 1.3526  1.3539  1.3510  1.3480  1.3479  1.34893]

b=[0.0352,0.0992,0.1570,0.1431,0.1634,0.1629,0.1046,0.1655,0.1635,0.1642,0.1658,0.1666,0.15735]

Thanks in advance.


texts: How to determine sample rate of a time series dataset?


texts: Dataset management: What are some strategies/solutions for efficiently storing datasets with their versions?

texts:
The problem: I've N classification models (independent), for each of these N models, I've different versions (eg: V0, V1, ..., Vfinal_production,Vexperimental). I'm looking for a way to store my datasets efficiently on the cloud (for redudency).

Note: We're not talking about BigData here.

Current Solution: Created a private GitHub repo. Made N directories and inside, pushed different dataset versions as different files.

Are there better solutions for this (because I feel VCS is an overkill for this problem)?


texts: Are there any techniques to explore the inter - target correlations and the input - output correlations in the Multi Traget Regression Problem

texts:
I am working on the Multi target regression problem in which there are 4 target variables and 17 input features.
I tried to use hard parameter sharing method to predict the target variables.
I am getting the good predictions for the first target variable but the model is unable to predict the predictions, well, for the remaining 3 target variables.
I think the model is unable to learn the nonlinear relationships between the input and output variables and it is also unable to learn the inter target correlations.
So I would like to know the techniques by which I can explore the inter target correlations as well as input - output correlations.
Also if there is any any other model I should try, I would like to know the recommendations.


texts: how to augument speech sentiment dataset?

texts:
I am building an LSTM to recognize if the person is sad, happy, angry or neutral. This is done by feeding-in his the wave of his voice into the network, as a sequence of bytes (each byte is 0-to-255).

The problem is, my dataset is not large enough, are there efficient ways I could augument my dataset? I am training on short 1.5 second clips and I have 800 of those, which is not enough.

My current augumentation is:

to add variations in volume
to add a bit of white noise, which makes it worse :(

Reversing the sequences doesn't seem to be applicable, after all, my network will be predicting non-reversed speech when it's fully trained.


texts:
i am using XGboost to predict a 2 classes target variable on insurance claims. I have a model ( training with cross validation, hyper parameters tuning etc...) i run on another dataset.

My question is :

is there a way to know why a given claim has been affected to one class i.e. the features that explain the choice made by the model?

The purpose is to be able to justify the choice made by the machine to third party human.

Thanks for your answer.


texts:
I am new to R and trying to install the gRain package. However, I am getting the error:

 dependency ‘gRbase’ is not available

Then, when trying to install gRbase I get:

package ‘gRbase’ is not available (for R version 3.5.2)


texts: removing special character from CSV file

texts:
I read my csv file as pandas dataframe. Originally it's a dict with multiple entries per keys. Its looks like this after reading as pandas dataframe:

aad,"[1,4,77,4,0,0,0,0,3]"
bchfg,"[4,1,7,8,0,0,0,1,0]"
cad,"[1,2,7,6,0,0,0,0,3,]"
mcfg,"[0,1,0,0,0,5,0,1,1]"

so I want to firstly remove the double quotes" symbol from the file and then want to create a new csv file from previous one with consecutive three entries in each row.

aad,1,4,77
aad,4,0,0
aad,0,0,3
bchfg,4,1,7
bchfg,8,0,0
bchfg,0,1,1
cad,1,2,7
cad,6,0,0
cad,0,0,3
mcfg,0,1,0
mcfg,0,0,5
mcfg,0,1,1


texts: How many Hidden Layers and Neurons should I use in an RNN?

texts:
I am very new to neural networks and machine learning and I have been making a Bitcoin price predictor to learn it. I was wondering about the number of hidden layers I'd need in a recurrent neural net using LSTM cells.

I have 60 inputs for 30 previous days' close prices in 12-hour intervals and require 1 output for the future 12 hours.

I am doing this with Keras in python 3.6.
Any help would be awesome!


texts:
When NN is construsted, batch size is not defined and place holder is used and its summary(tensorfow) shows the batch size as None.
This is useful because you can change batch size later.
In case of a simple model with 10 input features, 1 hidden layers with 10 neurons and output layer, the shape of the hidden layer would be (None, 10), which means if the batch size is 20, the hidden layer would have the shape of (20, 10).
When the model is used to predict for a single output, with shape(10, 1), how does the math work?


texts: Generate text using user-supplied keywords

texts:
I've got a use case where I need to generate sentences based on a set of user supplied keywords. Here is an example of what I need:

User input:

End-User: Data Scientists

Region: Middle East

Country: UAE

Solution: BigPanda

Application: machine learning

Benefits: lower costs and runtime

Output (Curly-Brackets are just there to highlight):

Learn how {data scientists} in the {Middle East} such as in the {UAE} are using {BigPanda} to streamline their {machine learning} processes to {lower costs and runtime}.

So the model needs to use the keywords given by the user and generate similar sentences. I also have a dataset of about 2000 of such sentences, which may come in handy.

One way I thought this could be possible is by using the GPT-2 model and perhaps finetuning it with the dataset, but I haven't been able to figure out how I would actually go about using it for something like this.


texts: How to detect time for the future events in time series data?

texts:
I am dealing with IOT data from a mechanical machine. On the input I have ~100 features that are measured every minute. On the output, I have labels of zeros and ones, where zero indicates the absence of the event and 1 indicate the presence of an event. The event represent the failure for the machine in place. Therefore, the goal is to predict at every time step the remaining "minutes" for a failure to occur. I would like to know how to tackle this problem, and if possible for some material to read.
Is there a way to know which features in the past leads to a failure in the future if I'm using an LSTM?


texts: Are Deep Neural Networks limited to grayscale images depending on whether you use Seq. or Func. API?

texts:
When I say DNN, I mean the simple usage of densely connected neurons (not CNN).
Say we are using Keras (with Tensorflow backend), the input_dim, using the Sequential API, can only take an integer or None value which restricts it to taking grayscale images right?
However since the functional API can accept dimensions that aren't just restricted to integers I would assume that it can take RGB images as opposed to only grayscale images?
If anyone can confirm or support, that would be great?


texts: Regression - Unbalanced Categorical Features

texts:
I have a data set that has some unbalanced categorical features. I would like to build a regression model to predict a label using machine learning (ML).

How do I handle data imbalances in categorical variables to avoid any bias in my ML regression model?


texts: How to calculate inverse of square matrix for streaming or online data as all data are not available at once?

texts:
Suppose initial data is $D$ and need to calculate the inverse of covariance of matrix $D$ i.e.
$C = cov(D,D)$, where $cov$ represents covariance.
$B = inv(C)$
Now, new data $N$ appears. So matrix D and C both will updated as follows:
$D^{new} = \begin{bmatrix}
D\\
N
\end{bmatrix}$
$C^{new} = \begin{bmatrix}
cov(D,D) & cov(D,D^{new})\\
cov(D^{new},D) & cov(D^{new},D^{new})
\end{bmatrix} = \begin{bmatrix}
C & cov(D,D^{new})\\
cov(D,D^{new})^T & cov(D^{new},D^{new})
\end{bmatrix}$
Similarly, data will be updated continuously. Now, inverse of $C$ (i.e. $B$) is given and need to calculate inverse of $B^{new}$. However, I don't want to use direct or general inverse as it will be very expensive if matrix size will be large and also want to utilize the inverse calculated for old data i.e. $B$.
P.S: I have tried inverse of block or partitioned matrix but it doesn't yield similar result as general inverse in all cases (for more detail). If someone needs then I can provide the MATLAB code to verify it.


texts: Logistic Regression implementation does not converge

texts:
I am currently trying to implement logistic regression with iteratively reweightes LS, according to "Pattern Recognition and Machine Learning" by C. Bishop. In a first approach I tried to implement it in C#, where I used Gauss' algorithm to solve eq. 4.99. For a single feature it gave very promising (nearly exact) results, but whenever I tried to run it with more than one feature my system matrix became singular, and the weights did not converge. I first thought that it was my implementation, but when I implemented it in SciLab the results sustained. The SciLab (more concise due to matrix operators) code I used is

phi = [1; 0; 1; 1];
t = [1; 0; 0; 0];
w= [1];

w' * phi(1,:)'

for in=1:100
    y = [];
    R = zeros(size(phi,1));
    R_inv = zeros(size(phi,1));

    for i=1:size(phi,1)
        y(i) = 1/(1+ exp(-(w' * phi(i,:)')));
        R(i,i) = y(i)*(1 - y(i));
        R_inv(i,i) = 1/R(i,i);
    end

    z = phi * w - R_inv*(y - t)
    w = inv(phi'*R*phi)*phi'*R*z
end

With the values for phi (input/features) and t (output/classes), it yields a weight of  -0.6931472, which is pretty much 1/3, which seems fine to me, for there is 1/3 probability of beeing assigned to class 1, if feature 1 is present (please forgive me, if my terms do not comply with ML-language completely, for I am an software developer). If I now added an intercept feature, which would accord to

phi = [1, 1; 1, 0; 1, 1; 1, 1];
w = [1; 1];

my R-matrix becomes singular and the last weights value is

w  =
  - 5.8151677
  1.290D+30

which - to my reading - would mean, that the probability of belonging to class 1 would be close to 1 if feature 1 is present about 3% for the rest. There has got to be any error I made, but I do not get which one. For both implementations yield the same results I suspect that there is some point I've been missing or gotten wrong, but I do not understand which one.


texts: How to write a code on Link prediction on dynamic network in python?

texts:
I have read plenty of papers on link prediction on dynamic networks, but I am getting confusion on how to do it. I know link prediction on dynamic networks is different from static network.In dynamic network your given a series of snapshots of the graphs(G=(G1, G2....Gn)) and trying to predict link in time T+1

For example, I have a file with a bunch of edge lists (link here) each edge list is associated with the time in which the edge was formed, here is the example on how I read the datasets

G=nx.read_edgelist('enron.txt',create_using=nx.Graph(), nodetype=int)

but the final graph when you draw it is just one graph, how can I have those series of graphs?
for example if I want to find the adjacency matrix for each snapshot, how to do it?


texts: Setting class-weights in a batch in where a certain class is not present

texts:
I'm handling a high imbalanced dataset, thus, I'm weighing the loss function in order to penalize the misclassification of the minority classes, I set the weights in each batch as follows:

w[i] = num_total_instances / num_instances_of_class[i]

Where 'i' goes from 0 to N-1 (for N classes), 'num_total_instances' is the total number of instances in the batch, and 'num_instances_of_class[i]' is the number of instances of class 'i' in the batch.

The problem is, as I'm doing it on each batch, It may occur that, in a certain batch there is no instance of class 'j', thus w[j] = inf. ¿What weigh should I set for class 'j'? At this moment, I'm setting this w[j] to 1E6

I know that there are other approaches to fight against imbalanced datasets and I also know that I can pre-compute the weights for the whole dataset and use them (fixed) cross all batches, but, I'd like the specific answer for this question, whereas it makes sense. For sure, it will be well received other suggestions for the calculation of the weights.


texts: I want to create an additional feature(column) based on some manipulation of values from existing features

texts:
Consider my data-frame to be like this ('x','y','z' are features):

I want to create a python function which will take an expression as a string (something like this: 'x+y-2z') and create a new feature by evaluating the expression. Output should be like:

I want to generalize this function so that it will work for different data-frames with different column(feature) names in the expression.

Edit- I have a prototype of the desired function(named 'parser'):

def parser(exp):

df['new_col'] = df.apply(lambda row: row.x+row.y-2*row.z, axis=1)

However, I want to generalize this part - row.x+row.y-2*row.z so that it will adjust itself according to the string(i.e. expression) provided as its argument.


texts: Interpreting Gradients and Partial Derivatives when training Neural Networks

texts:
I am trying to understand of purpose of partial differentiation in NN training by knowing how to interpret gradients and their partial derivatives. Below is my way of interpreting them so I would like to know if I am correct, and if not, could someone please point me in the right direction.

If we are working with functions that depend on a single variable, then derivative of that function with respect to that particular variable is a slope (i.e. constant) which tells us how the changes in the dependent variable will effect the changes in the function value.

If we are working with functions that depend on several (N) variables, then derivative of that function with respect to all of these dependent variables is a gradient (i.e. vector of partial derivatives) which points to the direction of function extreme. Each partial derivative corresponds to a specific dimension in the N dimensional space that we are trying to optimize (e.g. quadratic cost function C(W,b)).

My question is, when we calculate partial derivative with respect to one parameter (e.g. weight between input x1 and 1st hidden layer neuron) then we are treating all other weights and biases as constants and we are evaluating how will cost function change if we were to take a step in the direction that is represented by that particular weight. Is this correct? If not, please correct my understanding of partial differentiation in NN training procedure.

Also, what is the role of Jacobian matrix in NN training?

Thank you so much!


texts: Neural network for MNIST: very low accuracy

texts:
I am working on solving the handwritten digit recognition problem by implementing a neural network. But the accuracy of the network is coming out to be very low, around 11% for the train dataset. I am not sure what is wrong with my program. I tried changing the learning rate and the number of hidden units, but no luck. Could anyone please take a look and help me out with what I am missing? I am pasting my Julia code below:

# install
Pkg.add("MNIST");
using MNIST

# training data
X,y = traindata();
m = size(X, 2);
inputLayerSize = size(X,1);
hiddenLayerSize = 300;
outputLayerSize = 10;

# representing each output as an array of size of the output layer
eyeY = eye(outputLayerSize);
intY = [convert(Int64,i)+1 for i in y];
Y = zeros(outputLayerSize, m);
for i = 1:m
    Y[:,i] = eyeY[:,intY[i],];
end

# weights with bias
Theta1 = randn(inputLayerSize+1, hiddenLayerSize);
Theta2 = randn(hiddenLayerSize+1, outputLayerSize);

function sigmoid(z)
    g = 1.0 ./ (1.0 + exp(-z));
    return g;
end

function sigmoidGradient(z)
  return sigmoid(z).*(1-sigmoid(z));
end

# learning rate
alpha = 0.01;
# number of iterations
epoch = 20;
# cost per epoch
J = zeros(epoch,1);
# backpropagation algorithm
for i = 1:epoch
    for j = 1:m # for each input
        # Feedforward
        # input layer
        # add one bias element
        x1 = [1, X[:,j]];

        # hidden layer
        z2 = Theta1'*x1;
        x2 = sigmoid(z2);
        # add one bias element
        x2 = [1, x2];

        # output layer
        z3 = Theta2'*x2;
        x3 = sigmoid(z3);

        # Backpropagation process
        # delta for output layer
        delta3 = x3 - Y[:,j];
        delta2 = (Theta2[2:end,:]*delta3).*sigmoidGradient(z2) ;

        # update weights
        Theta1 = Theta1 - alpha* x1*delta2';
        Theta2 = Theta2 - alpha* x2*delta3';
    end
end

function predict(Theta1, Theta2, X)
    m = size(X, 2);
    p = zeros(m, 1);
    h1 = sigmoid(Theta1'*[ones(1,size(X,2)), X]);
    h2 = sigmoid(Theta2'*[ones(1,size(h1,2)), h1]);
    # 1 index is for 0, 2 for 1 ...so forth
    for i=1:m
        p[i,:] = indmax(h2[:,i])-1;
    end
    return p;
end

function accuracy(truth, prediction)
    m = length(truth);
    sum =0;
    for i=1:m
        if truth[i,:] == pred[i,:]
            sum = sum +1;
        end
    end
  return (sum/m)*100;
end

pred = predict(Theta1, Theta2, X);
println("train accuracy: ", accuracy(y, pred));


texts:
I am using ludwig to train and test on the dataset. There is one independent variable that has a text data type. I used 'Parallel_CNN' as a text encoder for the independent variable. My dependent variable has the category data type and it has a binary value (i.e. 0 or 1). After the model finished I plotted the predicted and actual output in the form of confusion matrix which is shown as below:

I was expecting TP,TN,FP, and FN in the confusion matrix. However,  which is unknown also found in the confusion matrix. I tried to google it to see the meaning and impact of unknown on model learning but could not find any useful information. Could anyone help me why does 'Unk' comes into model? Does that mean model is not efficient in learning?


texts: How does backpropogation behave when there is a dropout in FC layer?

texts: I read about dropout and how it helps in catering overfitting. In simple layman terms, it randomly drops some of the neurons in forward propagation. My question is that since these neurons will be 0 in forward propagation, are we supposed to update them while backpropagating?


texts: Training LSTM for time series prediction with nan labels

texts:
I have a time series of features $x_1,x_2,x_3,...,x_n$. I want to make a prediction $y_1,y_2,y_3,...,y_n$ for each timestep. However, in my training data some of the $y$ can be nan. I'd like the fit to just ignore these (i.e. the cost for this pair measured $y$ and predicted $y$ is zero).
I'm currently using tensorflow through Keras.

Is there an analogue of the masking layer for the label? I'm currently using tensorflow through Keras. Alternatively, it might be possible to change the loss function, but I don't know how, expecially while retaining numerical efficiency.


texts: Will cross validation performance be an accurate indication for predicting the true performance on an independent data set?

texts:
I feel that this question is related to the theory behind cross-validation. I present my empirical finding here and wrote a question related to the theory of cross-validation at there.

I have two models M1 and M2, I use the same data set to train them and perform cross validation using that same data set to find the optimal parameters for each model. Say eventually I found that M1 under its optimal parameter, performs better than M2 under its optimal parameter in terms of the 10-fold cross validation score. Now if I have another independent test data set with both predictors and labels and this test data set is generated from the same distribution of my training data set, then before I apply these 2 well-tuned model on that new test data set, can I claim or should I expect to see that M1 will still perform better than M2 over that new test data set?

I was playing Kaggle Titanic example. I have 2 xgboost model, M1 is well-tuned and M2 is less well-tuned in the sense that M1 has a better 10 fold cross validation performs on the training data set. But then when I submit both, I found that the less well-tuned model actually has a better scores on the test data set. How could that be? And if it is true, then what should we looking for when we fit the data to different models and tune the model parameters?

Here are my specific submission results: I did a random grid search

params_fixed = {'silent': 1,'base_score': 0.5,'reg_lambda': 1,
'max_delta_step': 0,'scale_pos_weight':1,'nthread': 4,
'objective': 'binary:logistic'}
params_grid = {'max_depth': list(np.arange(1,10)),
'gamma': [0,0.05,0.1,0.3, 0.5,0.7,0.9],
'n_estimators':[1,2,5,7,10,15,19,25,30,50],
'learning_rate': [0.01,0.03,0.05,0.1,0.3,0.5,0.7,0.9,1],
'subsample': [0.5,0.7,0.9], 'colsample_bytree': [0.5,0.7,0.9],
'min_child_weight': [1,2,3,5], 'reg_alpha': [1e-5, 1e-2, 0.1, 0.5,1,10]
}
rs_grid = RandomizedSearchCV(
          estimator=XGBClassifier(**params_fixed, seed=seed),
          param_distributions=params_grid,
          n_iter=5000,
          cv=10,
          scoring='accuracy',
          random_state=seed
)

Each time I change the variable n_iter. First, I set n_iter=10, it gives me a set of values of those hyper parameters, let's call this vector $\alpha_1$ and the cv score (accuracy rate) is 0.83389, then I use $\alpha_1$ to train my model and generate prediction on the independent test data set, and when I submit to Kaggle it generates true accuracy on the test data set 0.79426

Second, I set n_iter=100, it gives me $\alpha_2$ and the cv score is 0.83614, i.e., higher than the first one, makes sense, but when I submit to Kaggle, 0.78469, lower than the first one.

Third, I set n_iter = 1000, it gives me $\alpha_3$ and the cv score is 0.83951, i.e., higher than the second one, makes sense, but when I submit to Kaggle, 0.77990, lower than the second one.

Fourth, I set n_iter = 5000, it gives me $\alpha_4$ and the cv score is 0.84512, i.e., higher than the third one, makes sense, but when I submit to Kaggle, 0.72249, lower than the third one.

This is really frustrated. The model is getting better and better on the cross-validation score but when performed on an actual independent data set, its performance is getting worse and worse. Did I interpret the CV scores in the exactly opposite way? I see some paper mentioned that the CV score can be too optimistic for inferring the true test score. However, even if that is true, then I think the CV scores for all of my 4 models should be all optimistic about their own true test score, i.e., the order should preserve. But when applying on the real test data set, the order reversed.

The only reason I can imagine would be, that test data set has a different distribution than the training data set. However, if it is indeed the case, then I believe there is no method under then sun that can cure this problem.


texts: Neural Network not learning when more than 1 training data is given

texts:
I am very new to neural networks and data science in general and wanted to try getting my hand in making a simple neural network in python.

I tried to make a neural network from scratch hoping to start off with one that can learn to draw a regression line.

Here is my code, its pretty messy.

import numpy as np
import matplotlib.pyplot as plt
import random

class Layer: #format is input multiplied weights
    def __init__(self,n_inputs,n_neurons):
        self.n_inputs = n_inputs
        self.n_neurons = n_neurons
        self.weights = np.random.randn(n_inputs,n_neurons)
        self.biases = np.zeros((1,n_neurons))

    def forward(self, inputs):
        self.prevX = np.array(inputs)
        self.output = np.dot(inputs,self.weights)+self.biases
        self.ReLU = np.maximum(0,self.output)

    def Correction(self,target,Lrate,ReLUTrue = True):
        if ReLUTrue:
            output = self.ReLU.copy()
        else:
            output = self.output.copy()
        self.biases += 2*(target-output)*Lrate
        dWeights = np.dot(self.prevX.T,2*(target-output))
        dX = np.zeros((1,self.n_inputs))
        for j in range(self.n_neurons):
            dX += 2*(target[0][j]-output[0][j])*self.weights.T[j]
        self.targetprevX = self.prevX + dX*Lrate/self.n_neurons
        self.weights += dWeights*Lrate

class Network:
    def __init__(self):
        self.layers=[]

    def appendLayer(self,layerObject):
        self.layers.append(layerObject)

    def forward(self,inputs):
        for i in range(0,len(self.layers)-1):
            self.layers[i].forward(inputs)
            inputs = self.layers[i].ReLU.copy()
        self.layers[-1].forward(inputs)

    def backProp(self,target,Lrate):
        for i in range(-1,-len(self.layers)-1,-1):
            self.layers[i].Correction(target,Lrate)
            target = self.layers[i].targetprevX.copy()

    def UpdateLayers(self):
        for layer in self.layers:
            layer.Update()

    def Output(self):
        return self.layers[-1].output

neunetwork = Network()
neunetwork.appendLayer(Layer(1,8))
neunetwork.appendLayer(Layer(8,8))
neunetwork.appendLayer(Layer(8,1))

for iter in range(0,100):
    X = random.uniform(-5,5)
    Y = X**2
    neunetwork.forward([[X]])
    print("error:",neunetwork.Output()-Y)
    neunetwork.backProp([[Y]],0.001)

plotXs=[]
plotYs=[]
for points in range(0,100):
    X=random.uniform(-5,5)
    neunetwork.forward([[X]])
    Y=neunetwork.Output()[0][0]
    plotXs.append(X)
    plotYs.append(Y)
plt.plot(plotXs,plotYs,'ok')
plt.show()

The programme converges to the correct value when a single constant value is parsed but never converges when multiple different values are parsed. My understanding of neural networks is very limited and am just doing this as a little project for my own learning so pardon me for any incorrect use of terms.

Thank you for your time.


texts: Unseen classes - 'Other' class or classifying based on the existing classes

texts:
Let's suppose that I have a classifier which detects whether a script of code is written in Python or C++ or Java.

Therefore, my classifier has 3 outputs which give the probabilities that a certain code script is one of these 3 programming languages above.

However, unfortunately, there is the chance that the test data my contain some code scripts written in other programming languages such as Javascript, Ruby, PHP etc.

If I leave my classifier as it is then it going to classify these code scripts as being in Python or C++ or Java and this will be certainly pretty wrong and misleading.

Does it make sense to add another class called 'Other' which would contain code scripts of various other programming languages (Javascript, Ruby, PHP etc) or even irrelevant text to cope with that?

Is there any better idea?


texts:
I understand the mechanics of Encoder-Decoder architecture used in the Attention Is All You Need paper. My question is more high level about the role of the decoder. Say we have a sentence translation task: Je suis ètudiant -> I am a student
The encoder receives Je suis ètudiant as the input and generates encoder output which ideally should embed the context/meaning of the sentence.
The decoder receives this encoder output and an input query (I, am, a, student) as its inputs and outputs the next word (am, a, student, EOS). This is done step by step for every word.
Now, do I understand this correctly that the decoder is doing two things?

Figuring out relationship between the input query and encoder embedding i.e how is the query related to the input sentence Je suis ètudiant
Figuring out how is the current query related to previous queries through the masked attention mechanism. So when the query is student, the decoder would attend to relevant words which have already occurred (I am a).

If this is not the right way to think about it, can someone give a better explanation?
Also, if I have a task of classification or regression for a time series, do I need the decoder? I would think just the encoder would suffice as there is no context in the output of the model.


texts:
I recently came across an article about RNNs here. Which describes different types of RNNs like:

The first figure makes sense. A regular feedforward network.
The second is a big question for me. Is it one timestep cloned three times to make it recurrent with 3 timesteps?
The third is returning just the last timestep's output.
The fourth is an even bigger question. Is it getting the last timestep's output, clone it three times, and put another RNN layer on it? Or it's two RNN layers, where the output of the first layer is the input of the second (returning all timestep's outputs)?
The fifth makes sense as well, it returns the output of all timesteps.
So am I missing something, or the second, and fourth cases can only be made by cloning inputs/outputs as specified above?


texts: PCA formulation - Deep Learning book by Ian Goodfellow

texts:
I am reading this deep learning book by Ian goodfellow. In the PCA formulation in the first chapter i.e Linear Algebra, he mentions the following:

  we need to choose the encoding matrix D. To do so, we revisit the idea of minimizing the L2 distance between inputs and reconstructions. Since we will use the same matrix D to decode all the points, we can no longer consider the points in isolation. Instead, we must minimize the Frobenius norm of the matrix of errors computed over all dimensions and all points.

I can't understand the explanation for why can we no longer see the points in isolation. Would really appreciate if someone could help me out here.


texts: Beautifulsoup iterating through scraped data

texts:
I have this html code that is repeating multiple times:
     <div class="Company_line-logo image-loader-target" data-image-loader-height="47" data-image-loader-height-mobile="47" data-image-loader-src="/var/fiftyPartners/storage/images/startups/woleet/3261-1-fre-FR/Woleet_company_line_logo.png" data-image-loader-src-mobile="/var/fiftyPartners/storage/images/startups/woleet/3261-1-fre-FR/Woleet_company_line_logo_mobile.png" data-image-loader-width="189" data-image-loader-width-mobile="189" style='background-image:url("http://en.50partners.fr/var/fiftyPartners/storage/images/startups/woleet/3261-1-fre-FR/Woleet_company_line_logo.png");'>

Woleet

By using:
    for blocks in soup:
        block = soup.find('a', class_='Company_line logo-contains-name').find('h5').get_text()

I can get what I want that is "Woleet" in between the h5 tags.
I tried to iterate this to get all of them:
    block = soup.find('a', class_='Company_line logo-contains-name')
    for name in block:
        names = block.find_all('h5')

it returns only 1
h5 class="Company_line-title">Woleet</h5
whereas I should get 50 names like "Woleet".
How can I iterate this and get the text from all h5 tags?
Thanks!


texts: Extract keywords from text misses some entities

texts:
I have tried a few of the keyword extraction techniques and they work fine. But when I uploaded documents from scientific domains such as medical, genetics, healthcare, many medical or rather technical terms were missed.

What would I have to do to have the algorithm include those as well, am I missing a dictionary of such terms etc?


texts: App or Service to QA 80,000 classified images?

texts:
I've been reviewing services and apps for the past week and my head is exploding.  I just haven't found the right one yet, but I know it has to be out there, so I'm turning to the collective wisdom for help.

We have about 80,000 images that have objects from about 60 classes already tagged.  Most are in VOC format, some are in YOLO.  These were automatically tagged in most cases and we'd like to go through each one to verify the tag, delete images that are irrelevant, change tags when necessary, etc.

I've tried RectLabel on OSX, Dataturks, LabelBox, imglab, and some others.  None really seem to nail it, most are for initial labeling rather than QA/QC.  I'd also be happy to pay a service to do it, but I'm coming up empty on QA services - everyone wants to do the initial labeling.

Any suggestions would be tremendously appreciated.


texts:
I am writing my data science master thesis on "Decision making with sensor fusion and machine learning in unmanned systems"

What are the sub topics similar to "Collusion avoidance", "Navigation" and "Object Detection"? Are there publications about using big data in robotic systems, apart from these three?


texts: What Non-linearities are best in Denoising RNN Autoencoders and where should the go?

texts:
I’m employing a denoising RNN autoencoder for a project relating to motion capture data. This is my first time using auto encoder architectures and I was just wondering what non-linearities should be placed in these models and where they should go. This is my model as it stands:

class EncoderRNN(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers):
        super(EncoderRNN, self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.num_layers = num_layers

        self.rnn_enc = nn.RNN(input_size=input_size, hidden_size=hidden_size, num_layers=num_layers, batch_first=True)
        self.relu_enc = nn.ReLU()

    def forward(self, x):
        pred, hidden = self.rnn_enc(x, None)
        pred = self.relu_enc(pred)
        return pred

class DecoderRNN(nn.Module):
    def __init__(self, hidden_size, output_size, num_layers):
        super(DecoderRNN, self).__init__()
        self.hidden_size = hidden_size
        self.output_size = output_size
        self.num_layers = num_layers

        self.rnn_dec = nn.RNN(input_size=hidden_size, hidden_size=output_size, num_layers=num_layers, batch_first=True)
        self.relu_dec = nn.ReLU()

    def forward(self, x):
        pred, hidden = self.rnn_dec(x, None)
        pred = self.relu_dec(pred)
        return pred

class RNNAE(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers):
        super(RNNAE, self).__init__()
        self.encoder = EncoderRNN(input_size, hidden_size, num_layers)
        self.decoder = DecoderRNN(hidden_size, input_size, num_layers)

    def forward(self, x):
        encoded_input = self.encoder(x)
        decoded_output = self.decoder(encoded_input)
        return decoded_output

As you can see I have a ReLU non-linearity in each of the encoder and decoder networks but am not sure whether this is the correct implementation for these architectures.
The model learns on the data OK but the MSE loss doesn’t really improve after the first few epochs and I have a suspicion it’s because of these ReLU functions.

Any advice on how to improve this/ whether this is basically correct would be very helpful.

Cheers!


texts: Error in Convolutional Neural network for input shape

texts:
I have 1000, 28*28 resolution images. I converted those 1000 images into numpy array and formed a new array with size of (1000,28,28). So, while
 creating convolution layer using Keras, input shape(X value) is specified as (1000,28,28) and output shape(Y value) as (1000,10). Because I have 1000 examples as inputs and 10 categories of output.

model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),activation='relu',kernel_initializer='he_normal',input_shape=(1000,28,28)))
.
.
.
model.fit(train_x,train_y,batch_size=32,epochs=10,verbose=1)

So, while using fit function, it shows ValueError: Error when checking input: expected conv2d_1_input to have 4 dimensions, but got array with shape (1000, 28, 28) as an error. How can I provide proper input and output dimension for CNN.

Code:

 model = Sequential()
 model.add(Conv2D(32, kernel_size=(3, 3),activation='relu',kernel_initializer='he_normal',input_shape=(4132,28,28)))
 model.add(MaxPooling2D((2, 2)))
 model.add(Dropout(0.25))

 model.add(Conv2D(64, (3, 3), activation='relu'))
 model.add(MaxPooling2D(pool_size=(2, 2)))
 model.add(Dropout(0.25))

 model.add(Conv2D(128, (3, 3), activation='relu'))
 model.add(Dropout(0.4))

 model.add(Flatten())
 model.add(Dense(128, activation='relu'))
 model.add(Dropout(0.3))
 model.add(Dense(10, activation='softmax'))

 model.compile(loss=keras.losses.categorical_crossentropy,optimizer=keras.optimizers.Adam(),metrics=['accuracy'])
 model.summary()

 train_x  = numpy.array([train_x])

 model.fit(train_x,train_y,batch_size=32,epochs=10,verbose=1)


texts: Are there any rules regarding correlation between the variables used in a CNN?

texts:
I was wondering if there are any rules regarding correlation between the predictor variables or between the predictor and outcomes variables of a CNN. What should be the value of correlation? Does it matter when working with CNN?
I have the case of using two categorical predictor variables and


texts: How to get spike values from a value sequence?

texts:
I have pile of vectors where the values could be plotted like this:

Now I want to extract the "spike values" (over a certain threshold say 15,000). In this case there is fifteen. How could this be done with Python? (There is no predefined number of spikes but the threshold is a reliable filter value.)


texts: Linear Regression. Interpret weights as an effect on a prediction?

texts:
In a basic linear regression, can I use the weights of each explanatory variable to describe their relative effects on the predicted value? If parameter A has a weight of 100, and parameter B has a weight of 10, can I say that parameter A has 10 times more effect on the outcome than parameter B?

An example: From a weather data set, I used two parameters, humidity and pressure, to make a prediction on temperature.

When I plot humidity vs. temperature, there's a very clear inverse relationship between the two. That is, the scatter plot tends down and to the right.

When I plot pressure vs. temperature, there's no relationship at all. The pressure in this report has very little variance, and the scatter plot is nearly vertical.

Based on those two plots, I intuit that changes in humidity have an effect on the high temperature, while changes in pressure have almost no effect.

I built a linear regression model (gradient descent, by hand, in Octave). Humidity had a weight of -.32386. Pressure had a weight of -.02219. Humidity's weight was 14.6 times larger than pressure's weight.

Based on that, can I state that parameter Humidity has nearly 15 times more effect on Temperature than parameter Pressure?

Forgive what must seem like a very amateur question. If you can give a relatively simple answer and point me toward resources where I can learn more, I'd really appreciate it. Thanks in advance.


texts: LightGBM results differently depending on the order of the data

texts:
I have two datasets A and B which are exactly the same in terms of the number of columns, name of columns and the values. The only difference is the order of those columns. I then train the LightGBM model on each of the two datasets with the following steps

Divide each dataset into training and testing (use the same random seed and ratio for both A and B)
Leave the hyperparameters as pretty much default
Set a random state as a fixed number (for reproduction)
Tune the learning_rate using a Grid Search
Train a LightGBM model on the training set and test it on the
testing set
Learning rate with the best performance on the testing set will be
chosen

The output models on the two datasets are very different, which makes me thinks that the order of columns does affect the performance of the model training using LightGBM.

Do you know why this is the case?


texts: Adding context in a sequence to sequence problem

texts:
The encoder of a seq2seq model is meant to generate a conditioning context for the decoder, as mentioned here

  A RNN layer (or stack thereof) acts as "encoder": it processes the input sequence and returns its own internal state. Note that we discard the outputs of the encoder RNN, only recovering the state. This state will serve as the "context", or "conditioning", of the decoder in the next step.

Following the usual terminology and the implementation in the provided link, the encoder context is denoted as [c,h], where,

c is the internal state of the encoder in the last time step
h is the hidden state (output) of the encoder in the last time step

If one would like to add some external context e, what would be a reasonable way to do so?

Generating new context using some NN architecture with inputs [c,h,e]?
Concatenate the encoder context and the external context [(c,e),(h,e)]?
Any other suggestion?

To clarify the problem, let's take one would like to predict the future trajectory of an object based on past positions and using the location of obstacles as external context


texts: XGBOOST/lLightgbm over-fitting despite no indication in cross-validation test scores?

texts:
We currently work on a project where we aim to identify a set of predictors that may influence the risk of a relatively rare outcome. We are using a semi-large clinical dataset, with data on nearly 200,000 patients. The outcome of interest is binary (i.e. yes/no), and quite rare (~ 5% of the patients). We have a large set of nearly 1,200 mostly dichotomized possible predictors. Of note, our objective is not to create a prediction model, but rather to use the boosted trees algorithm as a tool for variable selection and for examining high-order interactions (i.e. to identify which variables, or combinations of variables, may have some influence on the outcome), so we can target these predictors more specifically in subsequent studies. Given the paucity of etiological information on the outcome, it is somewhat possible that none of the possible predictors we are considering actually have any influence the risk of developing the condition, so if we were aiming to develop a prediction model, it would have likely been a rather bad one. For this work, we use the R implementation of XGBoost/lightgbm.

We have been having some difficulties tuning the models. Specifically, when running cross validation to choose the optimal number of iterations (nrounds), the CV test score continues to improve even at very high values (for example, see figure below for nrounds=600,000 from xgboost). This is observed even when increasing the learning rate (eta), or when adding some regularization parameters (e.g. max_delta_step, lamda, alpha, gamma, even at high values for these). As expected, the CV test score is always lower than the train score, but continuous to improve without ever showing a clear sign of overfitting. This is true regardless of the evaluation matrics that is used (example below is for logloss, but the same is observed for auc/aucpr/error rate, etc.).  Relatedly, the same phenomenon is also observed when using a grid search to find the optimal value of tree depth (max_depth). CV Test scores continue to improve regardless of the number of iterations, even at depth values exceeding 100, without showing any sign of overfitting.

Note that owing to the rare outcome, we use a stratified CV approach. Moreover, the same is observed when a train/test split is used instead of CV.

Are there situations in which overfitting happens despite continuous improvements in the CV-test (or test split) scores? If so, why is that and how would one choose the optimal values for the hyperparameters?

Relatedly, again, the idea here is not to create a prediction model (since it would be a rather bad one, owing that we don’t know much about the outcome), but to look for a signal in the data that may help identify a set of predictors for further exploration. If boosted trees is not the optimal method for this, are there others to come to mind? Again, part of the reason we chose to use boosted trees was to enable the identification of higher -i.e. more than 2- order interactions, which cannot be easily assessed using more conventional methods (including lasso/elastic net, etc.).
Thank you!


texts: What is a better approach for cross-validation with time-related predictors

texts:
I was a given a data set with different predictors about a store and the idea is to forecast the number of daily shoppers. The predictors are the weekday, time of the day (morning, afternoon, evening), week number, month, weather (humidity, dew point, temperature), holidays. The outcome variable is the number of visitors.

I want to build a regression model to predict the number of visitors using traditional machine learning algorithms such as random forests, SVM, and the like.

My main concern is how to validate this model using CV since some of the predictors are time-related. Plain vanilla CV cannot be performed here. In this question, they suggest a way to perform this but my problem is that I only have data from June 2015 to present.

My initial idea was the following:

train with data from June 2015-December 2015. Test with January
train with June 2015-January 2016. Test with February 2016.

Each time one month of data is added to the training data after having asses the error for that month. Then compute average performance.

My questions:

Is this approach reasonable or not?
If so, should I get rid of the month variable? Note that in a., for instance, I am testing with some data that belongs to different months that the one used for training. I mean, for the training I used data from June to December 2015, but I am testing for January 2016. Seasonality can be something I am missing.
How to validate such models in general?


texts: Clustering Tweet Data using DBSCAN Algorithm

texts:
I am doing a tweet clustering using DBSCAN algorithm. I use all the preprocessing steps and convert sentences to vector format before applying the algorithm. However, It always puts a lot of tweets in to the '0' class. The following is the plot showing eps with number of clusters.

The following are the parameters that I pass.
dbscan = DBSCAN(eps=0.15, min_samples=2, metric='cosine').fit(x)

The following are the resulting clusters.

label
-1     1221
 0     1349
 1        2
 2        2
 3        4
       ...
 67       3
 68       3
 69       2
 70       2
 71       2

What is the reason that class 0 getting a high number of tweets than any other classes?


texts:
I am using jupyter notebook to host my machine learning codes and notebooks.

Jupyter Notebook server is sending some HTTP headers like Server = TornadoServer/5.1.1
I want to remove this header. Also want to add some HTTP headers like Cache-Control etc.
Can someone please tell me :

What kind of Http server jupyter notebook runs
Which configuration file controls the HTTP headers sent by Jupyter notebook
Any link which explains how to change/add/remove the HTTP headers

Current jupyter notebook is running on Ubuntu server on https.


texts: Best algorithm for real-time instance segmentation in videos?

texts:
I want to do object detection in real-time (meaning localization and classification) on videos (25 FPS), but with the added constraint that my training data is labelled using binary masks, rather than bounding boxes. I think it might be worthwhile to also output contours rather than bounding boxes. Since I'm fairly new to the field of computer vision I'm figuring out what's currently the best way to do this. I found that YOLO is more or less the go-to tool for real-time object detection, but their tutorials don't show anything regarding segmentation. Google results on this topic are also sparse and lack any hands-on examples.
Maybe YOLO is simply not the right tool for this job, hence my question.


texts: Understanding Support Verctor Regression (SVR)

texts:
This question also asked on another StackExchange with  Bounty. Question here.

I'm working with SVR, and using this resource. Erverything is super clear, with epsilon intensive loss function (from figure). Prediction comes with tube, to cover most training sample, and generalize bounds, using support vectors.

Then we have this explanation.  This can be described by introducing (non-negative) slack variables , to measure the deviation of training samples outside -insensitive zone. I understand this error, outside tube, but don't know, how we can use this in optimization. Could somebody explain this?

In local source. I'm trying to achieve very simple optimization solution, without libraries. This what I have for loss function.

import numpy as np

# Kernel func, linear by default
def hypothesis(x, weight, k=None):
    k = k if k else lambda z : z
    k_x = np.vectorize(k)(x)
    return np.dot(k_x, np.transpose(weight))

.......

import math

def boundary_loss(x, y, weight, epsilon):
    prediction = hypothesis(x, weight)

    scatter = np.absolute(
        np.transpose(y) - prediction)
    bound = lambda z: z \
        if z >= epsilon else 0

    return np.sum(np.vectorize(bound)(scatter))


texts: How to display the entire output and not shortened versions

texts:
I'm trying to display all the columns in my dataset that have NaN values:

print(train.isnull().sum())

The output that I get is this:

Id                0
MSSubClass        0
MSZoning          0
LotFrontage       0
LotArea           0
Street            0
Alley             0
LotShape          0
LandContour       0
Utilities         0
LotConfig         0
LandSlope         0
Neighborhood      0
Condition1        0
Condition2        0
BldgType          0
HouseStyle        0
OverallQual       0
OverallCond       0
YearBuilt         0
YearRemodAdd      0
RoofStyle         0
RoofMatl          0
Exterior1st       0
Exterior2nd       0
MasVnrType        8
MasVnrArea        0
ExterQual         0
ExterCond         0
Foundation        0
                 ..
BedroomAbvGr      0
KitchenAbvGr      0
KitchenQual       0
TotRmsAbvGrd      0
Functional        0
Fireplaces        0
FireplaceQu       0
GarageType        0
GarageYrBlt      81
GarageFinish      0
GarageCars        0
GarageArea        0
GarageQual        0
GarageCond        0
PavedDrive        0
WoodDeckSF        0
OpenPorchSF       0
EnclosedPorch     0
3SsnPorch         0
ScreenPorch       0
PoolArea          0
PoolQC            0
Fence             0
MiscFeature       0
MiscVal           0
MoSold            0
YrSold            0
SaleType          0
SaleCondition     0
SalePrice         0
Length: 81, dtype: int64

I actually want to see the entire output and not a shortened version with ..


texts:
I'm working with the Convex Nonnegative Matrix Factorization Algorithm described in Ding, Li, Jordan 2008 ("Convex and Semi-Nonnegative Matrix
Factorizations").

Good initialization strategies make all the difference and using the described k-means clustering to get started works very well.

But there is a paper describing NNDSVD (Boutsidis, Gallopoulos, 2007) to initialize "traditional" NMF Algos. I wanted to test this, to see if it improves my results.

The nonnegativity constraints for Convex-NMF are relaxed. X can have mixed sign data, where X ~ XWG', with factors W and G having only positive data.
I've implemented NNDSVD just like in the paper (in C++ w/ OpenCV), but since X has mixed sign data, the resulting W contains negative values as well.

Has anyone tried to adapt this initialization strategy? Are there any other recommended initialization strategies for Convex-NMF, aside from the ones mentioned in the Ding et al Paper (and random init)?


texts:
I have 3 data files (no missing values, structured nicely such that each line is a record and each column is a feature/variable).
Each line describes a Person with a unique ID number (and each person has only one line in each file); e.g. (displaying two persons, IDs 217 & 456):

------  ID        Age  Flag
File#1: 217 … … … 26    0
        456 … … … 12    1
        321 … … … 44    0
        789 … … … 67    NA
            … … …
File#2: 217 … … … 34    1
        456 … … … 23    0
        321 … … … 45    0
        789 … … … 76    NA
            … … …
File#3: 217 … … … 31    0
        456 … … … 56    0
        321 … … … 39    1
        789 … … … 77    NA
            … … …

Where the first field is the ID, the next to last is the Age of the person, and the last one is Flag  (= 1 if the Age is correct; 0 if not) => [edit:] NOTE: only about 10% of the records has this flag set [someone verified the correct age], the other 90% the Flag are missing (NA). The other features are all different (each file has its own distinct features not shared with the other, apart from the above 3 fields).
The variable if interest is Age; I need to devise an algorithm to decide which of the "Age" fields is the correct one (assume one is correct, and I know it from the files). This could be a probabilistic algorithm which assigns a probability to each of the age "candidates" and I'll pick the highest as winner.
Can some one please point me in the right direction? What type of problem is this? (and maybe suggest which algorithm(s) can be used for this type of problem?).


texts:
Context

Imagine that I have a dataset about sending messages. Each row as user_id, a batch_id, a is_open field (boolean) and a is_clicked field (boolean).

So one row means that one message was sent. It might have been open (is_open is true) or not (is_open is false). Same for clicked.

For this question, all corner use cases (what if a message is clicked without being opened?) are not relevant.

I want to graph open rate vs. click rate.

Question

How can I group these rows in a valid way, without discarding most of them?

Long version

The crux of my problem is that every single message has an open (and click) rate of exactly 0 or 100%.

I could first group messages per user, but then I would have to discard users having received less than at least 5 or 10 messages, to not have a peak a 0/20/40/60/80/100 %. This is a lot of data to drop, which is perfectly valid (and furthermore, I would like to compute things like median time to open, which does not lend itself well to multi-step calculation). It would take as well a while to get have enough historical data.

I could group by batch. But I could have for instance one batch per month, of 500k users. After a year, I would only have 12 points on my graph, whereas I already sent 6M messages.

My naive idea would be to just take rows by bunches of eg. 1000, and compute the open and click rate for this random bunch. It does not seem intellectually correct to me.

The actual language/implementation does not matter. I want to understand how to do this, actually doing it will come later.


texts: PyTorch is too heavy for deployment, is there a light weight pytorch for running neural networks?

texts:
We deploy PyTorch models in docker container, which massively increased the size of the docker container by more than 1G.

But when we deploy the model the training has already been done, so technically we don't need to include the machinery involved in training.

We don't even need backpropagation, we just need to run the neural network to get the outputs.

Is it possible to include only part of PyTorch (or another totally different product) that executes a neural network and nothing else? So deployment is light-weight.


texts: Find parameters to maximise output score

texts:
Not sure this is the right place to ask. Lets say there is a function f() where its implementation is unknown but it returns a score. I would like to get the highest possible score by modifying the input parameters. I also try to be better than brute force (finding all possible combination of input parameters, if that is even possible)
I do know that

f() runs an algorithm against a known dataset. The algorithm is tweaked by the input parameters.
f() needs 6 parameters
I know the type of each parameter (int, float, boolean)
I know the range of each parameter i.e [-1,5](int), [0,1](float, percentage, i.e 0.5 = 50%)
Some parameters have an open range i.e >1 (int)
Some are dependent of each other. i.e min and max parameter. That is min < max.

Any good pointers to algorithms that could help me solve that would be highly appreciated.


texts:
I am implementing the paper Perceptual GAN for small object detection. The design is described by the picture given below. I need to design my own keras layer. I have described my code below:

The user defined keras layer should consist of the following steps:

 model=VGG16() #loading pretrained VGG16 model
 intermediate_layer_model = Model(inputs=model.inputs, output=[model.layers[1].output, model.layers[5].output])
 y=layers.Conv2D(9,1,padding='SAME')(intermediate_layer_model.output[0])
 y=layers.Conv2D(1,1,padding='SAME')(y)
 output=residualblock(y)
 output1=roipooling(intermediate_layer.output[1],rois)
 add(output,output1)

I donot know how to code my own keras layer performing the above steps. Can anyone help me with the coding?


texts: If I understand the Databricks philosophy correctly, Spark will soon be heavily moving toward dataframes, i.e. away from the usual map/reduce on RDDs. I was wondering if there are any good suggestions for online courses or books that introduce Spark from the dataframe point of view? I know Databricks has a good number of resources on dataframes but I would really like to see some more detailed courses.


texts: Using SMAPE as a loss function for an LSTM

texts:
I am currently working on a time series forecasting problem and am looking into using an LSTM.

My final accuracy metric that I use to determine whether or not the forecast is good or not is defined as follows:

$$\text{SMAPE-3} = \frac{\sum^n_{t=1}|y_t-\hat{y}_t|}{\sum^n_{t=1}\left(y_t+\hat{y}_t\right)},$$

where $y_t$ is the actual value and $\hat{y}_t$ is the forecast value.  So this formula will produce a score between $[0, 1]$.  This score can then be subtracted from 1 to produce a percentage accuracy of the forecast over the specified forecast horizon

$$\text{Accuracy} = 1-\frac{\sum^n_{t=1}|y_t-\hat{y}_t|}{\sum^n_{t=1}\left(y_t+\hat{y}_t\right)}.$$

I am using keras and have seen that I can specify a custom loss function and metric.  As my forecast accuracy will be measured using sMAPE 3 (as defined above) it made sense to me to use this as my loss function.  This is because my networks success will be measured by how well it scores using this.  A score of 0 indicates 100% accuracy and a score of 1 indicates 0% accuracy.

So this raises two questions, can a metric such as smape be used as loss function at all, if so why / why not?  Are the standard loss functions within keras considered better?  Perhaps because they allow quicker convergence?  Does my choice of using SMAPE3 as a loss function have an impact on what kind of optimiser I should use?

I ask because I haven't seen many people using loss functions other than RMSE, MSE and MAE and there must be a reason for this.


texts:
I'm working my way through the book Reinforcement Learning by Richar S. Sutton and Andrew G. Barto and I am stuck on the following question.

The value of a state depends on the the values of the actions possible in that state and on how likely each action is to be taken under the current policy.
We can think of this in terms of a small backup diagram rooted at the state and considering each possible action:

Give the equation corresponding to this intuition and diagram for the value at the root node

, in terms of the value at the expected leaf node,

, given

. This expectation depends on the policy,

. Then give a second equation in which the expected value is written out explicitly in terms of

such that no expected value notation appears in the equation.

I should mention that...

...

Where...

 = Probability of taking action a from state s

 = Given any state s and a, the probability of each next state s'

 = Expected reward given any state s, next state s, and action a

How can I re-evaluate this value function in the way that is asked?


texts: How does Gradient Descent and Backpropagation work together?

texts:
Please forgive me as I am new to this. I have attached a diagram trying to model my understanding of neural network and Back-propagation? From videos on Coursera and resources online I formed the following understanding of how neural network works:

Input is given, which gets weight assigned to it using a probability distribution.
The activation functions use the weights to provide the predicted value.
The cost or loss functions calculate the error of the prediction between the actual class and the predicted value.
The optimization functions such as gradient descent use the results of the cost function to minimize the error.

If the above is correct then I am struggling to understand the connection between Gradient Descent and Backpropagation?

Here is an image of my understanding so far:


texts: How to train an LSTM model with data that has multiple input rows per day but only one row of label/ground-truth (output) data per day

texts:
I am doing a sleep data science experiment and I need a model that outputs multiple columns sleep quality measurement values (that are decimal numbers) for each input.
For training, I collected data using a smartphone (for input data) and a smartwatch (for label or output data). The smartphone collects MULTIPLE rows of sensor data such as accelerometer and gyroscope for ONE night. The smartwatch generates a SINGLE row of sleep quality indicator values (such as TWAK, NWAK, WASO, etc...) for ONE night. The structure of data looks as follows:

The above is a representation of my data. The left side is the phone feature data (with an undefined number of records per day based on how long the person was sleeping on that day) and the right side is the sleep quality measure data which is only one value per day. I need to train a model with this data and online sources seem to suggest LSTM is the way to go. Can you please give me a suggestion on how to do this for my type of dataset structure?
Thank you in advance! I can explain it however many times you need so feel free to ask me any question.


texts: how to evaluate feature quality for decision tree model

texts:
Most of the tutorials assume that the features are known before generating the model and give no way to select 'good' feature and to discard 'bad' ones.

The naive method is to test the model with new features and see how the new results change compared to the previous model but it can be complex to interpret when the tree is complex.

Is there an academic way to select good features and to discard bad ones?

(ressources appreciated)


texts: What algorithm could be used to fuzzy merge multiple datasets?

texts:
Problem Description

I have several tables that are related but do not share any unique key. I've come across this problem several times with customer data in separate source systems that needs to be compared together.

Lets say my data is multiple tables, Table A through Z.

There may be columns where I'm 100% certain on a match. For example table A and B have the column tax ID which is a certain match joining A to B. Both A and B alone cannot match to C, but using columns from A and B can make a certain match to C.

There are columns that are likely matches like matching first name, last name, birthday, etc "John Smith", but may result in false positives. Additional fuzzy matches may increase likelihood of match at different magnitudes.

I would like to feed an ML engine a set of tables that have been verified to be related to the same person, and see what rules it can come up with for matching the tables. I may get only a subset of tables at a time, and I want to be able to match what is possible and await more data. What algorithm could be used?

Thoughts

I know decision trees are great for this type of problem but probably many of the fuzzy checks I could make could better be expressed by some certainty magnitude. I've also never used a decision tree across a set of datasets where I could only be dealing with a subset.
I know Naive Bayes is often used for fuzzy matching but I also want to combine logic for 100% matches
Since this is a case where there can be limited full information via manually tracking down data, evolutionary algorithms could potentially work


texts: How to choose a classifier after cross-validation?

texts: When we do k-fold cross validation, should we just use the classifier that has the highest test accuracy? What is generally the best approach in getting a classifier from cross validation?


texts:
I know this question is rather broad but hopefully on topic. Are there useful references on classifying encrypted images. For example classifying cat/no cat on encrypted images.
Is there efficient and accurate software for this task?
Thank you!

EDIT for clarification: What I mean is the following: We have an encryption method E. A classifier is trained on encrypted images with their labels. Then at run time an image is sent in encrypted form from the client to the server and classified. This is similar to:

ML Confidential:
Machine Learning on Encrypted Data
https://eprint.iacr.org/2012/323.pdf


texts:
Let's say I want to measure a medicine’s impact on height. So I randomly broke down my user base into two groups (control and experiment, obviously). I calculated the average height for both groups before experiment and found out there’s a 2 cm difference. Then, after the medicine is applied to the experiment group, I found a 4 cm difference.

If I recall correctly, the t-test doesn’t handle selection bias, is there a test I can do to put the initial 2 cm difference between group into consideration?


texts: Develop a marketing look-alike profile for prospective customers

texts:
I work for a business-to-business company that has a large database of existing clients (small businesses) with various columns of data (e.g., industry, credit-worthiness, financial data, etc) describing each client. There are several marketing data vendors I know of that maintain small-business databases with columns of data similar to what my company has.

Question: Using the data my company already has, how I can I identify which customers in the vendors' databases have the highest potential of becoming a customer if marketed to? Should I build a predictive model? Should I do some sort of clustering?


texts: What is the difference between outlier detection and anomaly detection?

texts:
I would like to know the difference in terms of applications (e.g. which one is credit card fraud detection?) and in terms of used  techniques.

Example papers which define the task would be welcome.


texts: How do anchor boxes in object detection work?

texts:
I wanted to get more into detail of anchor boxes. However from looking through associated code and papers, I could not grasp the concept in its full detail. I had a look at a lot of quora questions, blog posts and papers as well trying to explain the concept, but they never went into the full detail (for dummies). I hope someone here is kind enough to take some time.
My current understanding is this:

We take an input image and create feature maps of this image until we come up with a feature map of dimension width x height x channels. Where these dimension are dinstinct and smaller than the original input images dimension.
We apply the regression and classification head for the bounding boxes and here the anchors come into play (not sure exactly how). The final loss function then regresses the class of the anchor box through cross-entropy loss and the offset of the coordinates to the anchor box e.g. through an L1 loss. For the loss calculation itself only a few anchor boxes are selected, normally those which possess a high IoU with ground truth boxes and random background boxes.
Additional factors such as focal loss can be applied to increase the training performance. Also the anchor boxes can be applied to features in different depth and thereby scaling of the network.

So far for the high level concept. My question:

How exactly is the loss evaluated for those anchor boxes? does the loss focus only on the inner values of the box or at the whole image?


texts: Which observation to use when doing k-fold validation or boostrap?

texts:
I have to perform predictive model over the dataset $D$ (with 1000 obs).
From $D$, I extract 700 obs for training $(T)$ and 300 obs for validation $( V )$.

I need to perform bootstrap or 10-fold cross validation sampling.

The question is which of these sets should I use?

Divide $D$ in 10 subsets and alternate training and validation between them ?
Divide $T$ (the training subset) in 10 subsets and perform training/validation on those subsets? $V$ is used only for final validation.


texts: Can I apply feature selection before splitting by requiring selection occurs > 90% of time

texts:
I want to move the feature selection step to before splitting to save time and allow bigger input dataset. If, in repeated subsamples, a feature is selected in over X percentage of cases I will keep it. Alternatively use very low X to remove features that will clearly never be selected. I have read warnings against doing this including on this forum because of information leakage. Feature selection: Information leaking if done before CV-split?  But if the feature would have been selected in almost all post split cases then where is the problem?
Edit: it does involve the target features.


texts:
This book "Deep Learning and Convolutional Neural Networks for Medical Image Computing" mentioned a term spatial feature encoding

  On the other hand, CNN models have been proved to have much higher
  modeling capacity, compared to the previous image recognition
  mainstream pipelines, e.g., HAAR, SIFT, HOG image features followed by
  spatial feature encoding, then random forest or support vector
  classifiers. Given millions of parameters to fit during model training
  (much more than previous pipelines), CNN representation empowers and
  enables computerized image recognition models, with a good possibility
  to be able to handle more challenging imaging problems. The primary
  risk is overfitting since model capacity is generally high in deep
  learning but often very limited datasets are available (that are with
  good quality of labels to facilitate supervised training). The core
  topics of this book are represented by examples on how to address this
  task-critical overfitting issue with deep learning model selection,
  dataset resampling and balancing, and the proper quantitative
  evaluation protocols or setups.

I googled a lot and didn't find explanation or definition of "spatial feature encoding".

What is "spatial feature encoding"? Can anyone give a concrete example?


texts: Triplet loss - what threshold to use to detect similarity between two embeddings?

texts:
I have trained my triplet loss model using FaceNet's architecture. I used 11k hands dataset. Now I want to see how well my model performed, so I feed it 2 images of the same class and get back their embeddings. I want to compare the distance between these embeddings and if that distance is not larger than some threshold I can say that the model correctly classifies these 2 images as of the same class.

How do I select the threshold value?

Is the threshold value the same as the margin hyperparameter used in triplet loss function? (alpha = 0.2)

After examining FaceNet's code and OpenFace's code, I saw that they test 400 thresholds raging from 0.0 to 4.0 ([0.00, 0.01, ..., 3.99, 4.00]) to find the one that gives the most accurate results and then treats that as the overall accuracy.

You can see it implemented in FaceNet here and in OpenFace here.

Why does it select 0.01 as the step from 0.0 to 4.0? Why not 0.001 or 0.0001?

Moreover, FaceNet uses LFW dataset. LFW dataset defines its own specific protocol for evaluation which is implemented in FaceNet and OpenFace. So isn't it fair to say, that the way FaceNet and OpenFace evaluate their model is exclusive to only LFW dataset?

In that case, how should we evaluate our model on different datasets that LFW?


texts: Correlation between Time Series Indicators ( Stock Prices )

texts:
I am new to time series analysis and I am currently tackling a stock market prediction problem.

I have a set of market indicators (such as Bollinger Bands, ADX etc) which are derived from the time dependent Open, High, Low and Close prices ( in Dollars ).

I need to find correlations of this indicators with the Open, Low, High and Close prices over time and drop the features which are not correlated enough ( less than 0.70 )

I am working on python. Using Pandas I have tried the pandas.dataframe.corr() method as well, but I want to know if Pearson and Spearman correlation fucntions in pandas serve my purpose ? Is it the right way or is there another way of finding correct correlations ?

Thanks.


texts: Can anyone recommend me a very good pre-trained model for face or head detection?

texts:
I really need to know the best pre-trained models to detect faces and/or peoples' head. Not a face recognition model, but only to classify whether an object is a person's head/face or not.
I'm implementing one on top of Resnet50 from tensorflow.keras.applications.resnet50, but I'm not sure if it's a good approach.
Other models like centernet and efficientnet in tensorflow are pre-trained to detect several objects, but I'm not sure  if I can also use it for this purpose.
I need something better than those provided by OpenCV (e.g. cv2.HOGDescriptor, cv2.CascadeClassifier, etc.).


texts: Can you learn an algorithm from a trained model?

texts:
Are there any papers where an algorithm was entirely based on the results of a trained model? Let me explain.

Suppose you want to come up with an algorithm that sorts three numbers $a,b,c$. I can generate several training data points $x_i = (a_i, b_i, c_i), a_i, b_i, c_i \in \mathbb{R}$ with their label $\hat y_i = (\min(x_i), mid(x_i), \max(x_i))$. That way, I can generate a lot of data points and train a model to predict them in their order.

My question is, are there any papers which were able to translate the trained model back to an algorithm that is understandable by humans (instead of just the values of the parameters ?). I'd be very interested even if the algorithms were very simple.


texts: Is there a documentation where it is explained why scikit-learn does not provide p-values?

texts: Is there a documentation, paper etc. where it is explained why scikit-learn does not provide p-values/confidence levels (1, 2, 3, 4)? Note: I'm not asking about opinions, but about documentation. For example the R package lme4 does not provide p-values either, but has a FAQ point  about this. On the official scikit-learn website I found no explanation for this desicion.


texts: Given a single discrete data set, how should I divide it into training data and test data?

texts:
I have a dataset in libSVM format consisting of 6000 entries, each with 5 indices, and each index has a binary value 1 or 2. Each of the 6000 entries has a label of 1 or 0, and I am trying to use various machine learning algorithms to determine the correct label (0 or 1) given a particular set of 5 indices/values.

For example, consider the following dataset (the real one is 6000 lines):

0 101:1 102:1 103:0 104:1 105:1
0 101:0 102:1 103:0 104:1 105:1
0 101:0 102:1 103:1 104:1 105:1
1 101:1 102:1 103:1 104:1 105:1
1 101:0 102:1 103:0 104:0 105:1
1 101:1 102:1 103:1 104:0 105:0
1 101:0 102:1 103:0 104:0 105:0

For an algorithm that predicts binary classification, like xgboost, conceptually, how do I first use my dataset to train the model, and then apply the model to the data?

I ask because xgboost asks for two files, a data training set and a data test set. It seems to me that the algorithm should just require a single full set of data, use all of the data to train and build a model, and then apply that model to the original data set and determine if the labels are being assigned "0 or 1" accurately.

Any help in understanding this concept is much appreciated.


texts: What are the use cases for Apache Spark vs Hadoop

texts: With Hadoop 2.0 and YARN Hadoop is supposedly no longer tied only map-reduce solutions. With that advancement, what are the use cases for Apache Spark vs Hadoop considering both sit atop of HDFS? I've read through the introduction documentation for Spark, but I'm curious if anyone has encountered a problem that was more efficient and easier to solve with Spark compared to Hadoop.


texts:
I've coded the 3D Region Growing from PCL to Python, thanks to the pseudo code from : https://pcl.readthedocs.io/projects/tutorials/en/latest/region_growing_segmentation.html
But I had a question, imagine instead of taking seed points as minimal curvature, as a classical Region growth, we decided to take custom seed points to initiate the algorithm from specific place to improve segmentation. I have in mind taking, for example, 6 seed points from 6 tree trunks detected by something before, and doing region growing from these points to segment the tree crown.
I wonder how this code could be transformed. Any library or advices are welcomed. Many thanks !
import math
import numpy as np
from sklearn.neighbors import KDTree

unique_rows = np.loadtxt("my/path/file.txt")
tree    = KDTree(unique_rows, leaf_size=2)
dist,nn_glob = tree.query(unique_rows[:len(unique_rows)], k=30)

def normalsestimation(pointcloud, nn_glob, VP=[0,0,0]):
ViewPoint = np.array(VP)
normals = np.empty((np.shape(pointcloud)))
curv    = np.empty((len(pointcloud),1))

for index in range(len(pointcloud)):

    nn_loc = pointcloud[nn_glob[index]]
    COV = np.cov(nn_loc,rowvar=False)
    eigval, eigvec = np.linalg.eig(COV)
    idx = np.argsort(eigval)
    nor = eigvec[:,idx][:,0]

    if nor.dot((ViewPoint-pointcloud[index,:])) > 0:
        normals[index] = nor

    else:
        normals[index] = - nor

    curv[index] = eigval[idx][0] / np.sum(eigval)

return normals, curv

def regiongrowing(pointcloud, nn_glob, theta_th = 'auto', cur_th = 'auto'):

normals,curvature = normalsestimation(pointcloud,nn_glob=nn_glob)
order             = curvature[:,0].argsort().tolist()
region            = []

if theta_th == 'auto':
    theta_th          = 15.0 / 180.0 * math.pi

if cur_th == 'auto':
    cur_th            = np.percentile(curvature,98)

while len(order) > 0:
    region_cur = []
    seed_cur   = []
    poi_min    = order[0]

    seed_cur.append(poi_min)
    region_cur.append(poi_min)
    order.remove(poi_min)
    seedval = 0

    while seedval < len(seed_cur):
        nn_loc  = nn_glob[seed_cur[seedval]]
        for j in range(len(nn_loc)):
            nn_cur = nn_loc[j]

            if all([nn_cur in order , np.arccos(np.abs(np.dot(normals[seed_cur[seedval]],normals[nn_cur]))) < theta_th]):

                region_cur.append(nn_cur)
                order.remove(nn_cur)

                if curvature[nn_cur] < cur_th:
                    seed_cur.append(nn_cur)

        seedval+=1

    region.append(region_cur)

return region

region  = regiongrowing(unique_rows,nn_glob)


texts: How to update the posterior belief when we are observing a stream of correlated data from a fixed but unknown data source

texts:
I want to build a [probabilistic] model that aims to infer the true value of an unknown categorical variable, $y \in \{1,2,..., K\}$.

We have a dataset $(X,y): \mathbb{R}^d\rightarrow \{1,2,..., K\}$ and we can train a classifier that gives $d$-dimensional data, $X$, and estimates the output $y$.

Now, suppose that $X$s are correlated and all coming from a fixed $y$. I mean, we are observing $X^1, X^2,...., X^T,...$ over time and we know that $y$ is fixed for all of them.

For example:

We receive $X^1$ (at time $t=1$) and our previously trained classifier produces a guess about $\hat{y}^1$.
Then, we receive $X^2$, and we again use the classifier to guess $\hat{y}^2$.
Then, we receive $X^3$, and so on.

So, at time $t=T$ we have $\hat{y}^1, \hat{y}^2, ..., \hat{y}^T$.

Now, the question is: How can I make a model to use these estimations ($\hat{y}^1, \hat{y}^2, ..., \hat{y}^T$) and improve my belief about the true $y$ over time, considering that:

dimension $d$ is not small. e.g. $d >50$
data samples, $X$s, are not i.i.d. but all coming from a fixed unknown $y$.
classifier is not optimal (just trained on some available data) and at each round gives an estimate about the $\hat{y}^t$ for the current $X^t$.

I have been reading some materials and came across the following but I am not sure which one is better to investigate more into:

Sequential Hypothesis Testing
Optimal stopping
Sequential probability ratio test
HDI+ROPE decision rule: highest density interval (HDI) region of practical equivalence (ROPE)

Or is there any specific Bayesian framework for it?


texts: Is there any problem using Euclidean Distance for a data set that includes a lot of values between 0 and 1?

texts:
I have a data set which includes 3 types of financial data (P/E, ROA, EPS growth) for each stock in a list of stocks. My goal is to use K-Medoids to cluster these stocks into groups based on this data.

I standardized each data point so it is represented as a standard deviation of the group of financial data it is apart of. However, most of my data is now between 0 and 1 and I was going to use Euclidean distance to sort the stocks.

Would there be any problem using Euclidean distance in this problem since the square root of a fraction makes the fraction larger?


texts: How to calculate change in Loss Function w.r.t Weights for a fully connected NN

texts:
I read the article Optimal Brain Damage by Yann LeCun, John S. Denker and Sara A. Solla where the authors discuss estimating the saliency of each weight of a neural network (which they define by the change in the loss function upon perturbing the weight). The larger the saliency, the more effect the weight has on the learning.
Is there any code/API in Keras or Tensorflow to calculate such "weight saliency" for a simple fully connected neural network (with one or two hidden layers)?


texts: how to convert feature_log_prob_ to exponential feature importance for BernoulliNB

texts:
I am using BernoulliNB classification for problem
from sklearn.naive_bayes import BernoulliNB
from sklearn.metrics import classification_report
from sklearn.metrics import classification_report,confusion_matrix
clf2 = BernoulliNB()
clf2.fit(X_train, y_train)
y__NB__prediction=clf2.predict(X_test)
sk_report = classification_report(
                  digits=2,
                  y_true=y_test,
                  y_pred=y__NB__prediction)
print(sk_report)

for calculating feature importance i am using feature_log_prob_. But there are a few issues for me:

feature_log_prob_  depends on the number of classes and the number of features for each class, but how to determine which class it belongs to.?
feature_log_prob_ don't give the probability directly but its logarithm. So you need to apply exponential in order to get back the probability. How should this be done?


texts: How can my loss be stable while the gradient keeps growing?

texts:
I have been working on an Offline/Batch Reinforcement Learning problem where I am using a BCQ-DDQN model as a Q-table. The model input is a state of 8 dimensions, and the output is a vector of Q-values for 8 different actions. The hidden layers have respectively 64 and 128 neurons.
I have run multiple experiments in different configurations, i.e., with different reward functions and different datasets (as we are in an Offline RL case).
For all experiments (with multiple seeds), I have noticed a weird behavior during the training of my agents.

While the loss of my models seem to have almost stabilized, my gradients keep growing for some reason. In the images, the "Grads (unclipped)/Mean" graph is computed using tf.global_norm() from Tensorflow on the different parameters of the network, i.e., the function is fed with the tensors of the gradients of the weights and biases of each network layer.
Note that in one of the experiment, it has happened that the gradient stabilised to a non-value zero while the loss was stable too.
With a stabilised loss, I would expect the norm of my gradients to converge to 0, meaning that the agent would have converge to some optima. However, in my case, the signals seem to be contradictory. The loss suggests that an optima has been reached, while a growing gradient suggests the opposite.
How is this possible? What should I conclude about my model, is it close to convergence? Or not yet?


texts: What are true positive, false positive, true negative, false negative in the given example?

texts:
Assume there are 100 images, 30 of them depict a cat, the rest do not.
A machine learning model predicts the occurrence of a cat in 25 of 30 cat images. It also predicts absence of a cat in 50 of the 70 no cat images.

In this case, what are the true positive, false positive, true negative and false negative?


texts: How exp(-z) is working in a sigmoid function in neural networks while z is a matrix?

texts:
function g = sigmoid(z)

%SIGMOID Compute sigmoid function
%J = SIGMOID(z) computes the sigmoid of z.

g = 1.0 ./ (1.0 + exp(-z));

end

I'm going through the Andrew Ng Coursera course. I doubt that how exp(-z) is computed directly while z is a matrix?