texts: Determining the optimal number of cluster and clustering method in hierarchical clustering

texts:
I'm working with some patient data and decided to cluster the patients.
I came up with a distance metric, created a distance matrix, and performed hierarchical clustering.
Now the question is, what linkage method of hierarchical clustering is best? And what is the optimal
number of clusters. In order to answer this question, I tried a bunch of different methods and cluster numbers. Here is the result: Rows are cluster numbers. Columns are methods
       ward.D    ward.D2     single   complete    average   mcquitty     median   centroid
2  0.07012487 0.06110526 0.07008440 0.06106565 0.07008440 0.07008440 0.07008440 0.07008440
3  0.04649798 0.05063088 0.06569336 0.05302616 0.05288394 0.04670727 0.06569336 0.06431294
4  0.04913864 0.04923650 0.05607585 0.05362537 0.05214325 0.04614849 0.05771631 0.05771631
5  0.05562508 0.05307553 0.06191763 0.05383284 0.04695595 0.05097810 0.05651751 0.05737195
6  0.05676926 0.05734915 0.04975863 0.05568472 0.05614707 0.05863589 0.05509910 0.05784693
7  0.06000339 0.06015014 0.05416420 0.05975802 0.05877047 0.06028632 0.05773932 0.05554587
8  0.06325530 0.06270295 0.05181852 0.06358805 0.06437938 0.06284041 0.05749092 0.04712364
9  0.06227251 0.06389455 0.05368045 0.06593435 0.06503654 0.06315090 0.05342649 0.05380309
10 0.06461370 0.06328118 0.05361237 0.06385926 0.06407600 0.06173574 0.06226608 0.05326900

The value you see in the matrix is the average silhouette width from silhouette analysis. I used the silhouettes to iteratively improve each clustering. I'll show this with an example
Consider the fourth row and third column (i.e. 5,single). If I run a hierarchical clustering using the "single" method and cut the dendogram at 5 clusters, I get the following silhouette plot

  cluster size ave.sil.width
1       1  354          0.06
2       2    1          0.00
3       3    1          0.00
4       4    1          0.00
5       5    1          0.00

This is clearly a terrible clustering. So I take each data point that had a negative silhouette and assign it to its neighbouring cluster (that's what a negative silhouette means. That it's closer to its neighbouring cluster than its own). I then recalculate the silhouettes again. Whatever points have negative silhouettes, assign them to their neighbour cluster. I do this 50 times (enough to for convergence). And in the end, the clustering has improved

  cluster size ave.sil.width
1       1  278          0.07
2       2   12          0.01
3       3   20          0.03
4       4   21          0.02
5       5   27          0.06

I then take the average silhouette width of this plot. And put it into the data frame at position (5,single)
BTW, at each iteration of this cluster-improvement process, I kept track of the average silhouette width. Here's what it looked like
So I did this for 8 different methods and 2-9 clusters. Now the question is, using this heatmap, what is the best method and number of clusters?
Is there a better way that I could determine this?


texts:
I am new to data science so please accept my apology in advance. I am trying to predict the value using ARIMA. I have got weekly value for the current year. Based on the available weekly values, I predict the remaining weekly values of the year. I am using ARIMA model. After forecasting the value, I plotted the predicted value and confidence interval as shown in the figure:

The red dotted line is the prediction and grey area shows the confidence interval. The problem is that I want area of confidence interval gets wider as we are closer to the end of the year (in my case year starts on 1st of April and ends on 31st March every year), like a funnel shape. I am using Python for forecasting. Here is my code:
from statsmodels.tsa.arima_model import ARIMA

model = ARIMA(df['Value'],order=(2,1,1))
model_fit = model.fit(disp=0)
forecast,stderr,conf_int = model_fit.forecast(27,alpha=0.05)

I read somewhere that it might happen if you don't have a large data sample size which is also my case. In my case prediction is based on 25 weekly values. However, I predicted another column (let say column B) from the same data set, the confidence interval of predicted column B value was better as shown below:

Could anyone please guide me on how to overcome this issue? or correct me if I am thinking in the wrong directions?


texts:
Is there a R implementation of isolation forest for anomaly detection?

Similar to the implementation from sklearn.


texts: NER: Extracting entities from an article

texts:
Description : I have dataset of categorised articles and to extract specific values from respective categorised article I have regex created for each category.

Aiming for:

Nlp techniques which learns the context of the content and avoids/minimizes the use of regex
If some new (similar) article comes up, depending on the learning (from 1) it tries to give the specific values.

Steps taken:

Created a dataframe with various features like : 'Name of the author', 'published date', etc. and got the values from the dataset by using regex

Research_work:

I was considering these options ahead this stage :

Using CNN : it will classify new articles depending on the feature values it learnt on and then use regexs for entity extraction. (It wont achieve the first aim)
Using CRF (medium_article): making use of POS+IOB tagging

Is there any other way around ?
cons of above stated methods?


texts:
-I want to make a python program using the LSTM model to predict an output value that is 1 or 0.
-My data is stored in a .csv file of the form: (Example of the line)
Date              time         temperature     wind  value-output
10-02-2020        10:00            25           10         1
-I found several courses, several examples of LSTM but I don't find my classification problem to do the same thing, there are many examples on translation.
-I am stuck on how to prepare my program my data to give them to the LSTM model.
-I want to take into consideration my temperature and wind inputs in addition to the time to predict the output value.

(I have already made a python program based on a simple ANN to predict my output value by following a tutorial), but for the LSTM I find it difficult.

Thanks in advance for your help.


texts: Could anyone please recommend a good frequent itemset package in python? I only need to find frequent itemset, no need of finding the association rules. Thanks!


texts: I understand how the convolution layers are applied after selective search finds the regions of interest in vanilla R-CNN and so the back-propagation or any weight updating is done in the individual convolution networks. But in Faster R-CNN a convNet is applied first and then RPN works out the regions of interest in a separate convNet. Where exactly are the weights updated? Is it between the initial convNet and the RPN. But weights can only be updated after comparison with the ground truth values. Or is it that after a full forward pass of both networks, a backward pass is done. In this case how are both considered simultaneously? If not, Where and how is this comparison done? Any pointers to any resource will be highly appreciated.


texts: What are common problems around HADOOP storage?

texts:
I've been asked to lead a program to understand why our Hadoop storage is constantly near capacity.  What questions should I ask?

Data age,
Data size?
Housekeeping schedule?
How do we identify the different types of compression used by different applications?
How can we identify where the duplicate data sources are?
Are jobs designated for edge nodes only on edge nodes?


texts: Do I need to apply a Ranking Algorithm for this?

texts:
I have data of the form :

Id1      A_Id2      B_Id2       C_Id2       D_Id2      E_Id2      F_Id2
1         6           3           9           23         20         5
1         4           7           8           9          11         56
1         2           36          98          73         2          4
1         9           5           2           7          32         24
1         14          7           5           9          12         5
2         34          4           7           10          7         12
2         5           57          23          91          4         6
2         7           .           .           .           .         .
2         3           .           .           .           .         .
2         .           .           .           .           .         .
.
.
100      .            .           .           .           .         .

Basically, I want to build a model such that I can get a best match of Id1, given top 5 Id2 matches of each attribute(A_Id2, B_Id2,...,F_Id2). Now every match should be computed keeping in mind A has highest priority, followed by B and C, followed by D and least priority to E and F.

So the output will look like this :

Id1    Match_Id2
1        9
2        7
3        .
4        .

I hope the problem is clear, if not please ask.
How should I go about building a Machine Learning model for this? I was wondering if ranking algorithm will help ?


texts: LSTM with keras/tensorflow implementation

texts:
I'm trying to learn a sufficient statistic from data. My input are sequences of samples x_samples which are real numbers. The length of the input sequence varies and has no fixed value. There is no memory between the sequences - each sequence is completely independent from all the others.

The output y is the sufficient statistic given by the sum of log(p_1(x)/p_2(x)) for all the samples in a given sequence (thus a RNN is needed). p_1 and p_2 are two unknown functions (they are the PDFs of some process). For example if a sequence has 3 samples x1,x2,x3 the output is given by y = log(p_1(x1)/p_2(x1)) + log(p_1(x2)/p_2(x2)) + log(p_1(x3)/p_2(x3)).

I am wondering how to input the sequences into my network. I've seen a few solutions to this problem but none of them seem relevant for me.

I don't want to use zero padding. The reason is that the sequence has no maximum length - it depends on other factors independent of this problem and may theoretically take any value. The LSTM is part of a bigger RL system. The data is generated along the way  so i don't have a fixed data set.
I don't want to use 1 hot encoding for my input since x_samples are just real values. It is possible to quantize the input but i think this is a bad solution.

I think it makes sense to input the the samples one by one with a start and end tokens for each sequence. Does this make sense?

Additionally I wonder if during the training I should reset the memory of the LSTM in the end of each sequence (with model.reset_states(), have I understood it correctly?) or it is enough to set a start and end tokens in the begging and the end of every sequence.


texts:
I am consistently seeing higher validation loss when I train & evaluate a model on AWS GPU vs local CPU.

I am using the exact same train/eval datasets and the exact same Tensorflow code and configuration for both runs. Has anyone else experienced this before and know how to explain?

See screenshots below comparing GPU vs CPU loss for equivalent steps.


texts: Why is the dimensionality reduction and/or feature selection techniques are considered to be not always effective?

texts: Can you please explain when (under what circumstances) the dimensional reduction and/or feature selection techniques are not effective to mitigate the curse of dimensionality?


texts: How to control the amount of positives in classification?

texts:
I have a basic, yet quite complex problem to solve right now. Let's say we have a training set of 20,000 samples in my training set, out of which 3 to 4% is flagged as "True", the rest is flagged as "False". I want to train a classifier (typically XGBClassifier or LGBMClassifier are those I worked a bit with).

What I'm currently doing is that I find the best parameters using GridSearchCV. But my objective is to minimize the amount of samples I would flag as "True" once I try on the test set. Should I train the algorithm with a typical F1 metric, and ONLY THEN find the best threshold that suits my needs? Or should I create a custom metric that would implicitly force the algorithm not to flag too many samples as positive?

Hope this is clear!


texts: How can I adjust the legend when visualizing clusters in two dimensions?

texts:
How can I change the legend as we can see now the legend has some cluster numbers missing. How can I adjust the legend so that it can show all the cluster numbers (such as Cluster 1, Cluster 2 etc, no it's only 0 3 6 9)? (codes I followed this link: Perform k-means clustering over multiple columns)

kmeans = KMeans(n_clusters=10)
y2 = kmeans.fit_predict(scaled_data)

reduced_scaled_data = PCA(n_components=2).fit_transform(scaled_data)
results = pd.DataFrame(reduced_scaled_data,columns=['pca1','pca2'])
sns.scatterplot(x="pca1", y="pca2", hue=y2, data=results)
#y2 is my cluster number

plt.title('K-means Clustering with 2 dimensions')
plt.show()

Edit: the legend seem not the same, the cluster 0 should be the lightest color.


texts: Dimension reduction : binary description and regression

texts:
Let assume that I own a model agency (that is a far-fetched example), and I would like to create a tool for my HR department, that gives an estimate salary for any given profile.

I have a huge database of profiles, where each name has a salary, sex,age, height,eye color, country of origin, weight, experience...

For example : $($Mister $1234$ ,Male, $\$200$k/year,$6'3$ or $190$cm , $200$ pounds or $90$kg, Blue eyes, Brown hair, $22$ years old,...$)$

The idea is that every time there is a new applicant, I input his/her characteristics into the tool, and it returns a salary.

The first idea that came in my mind is to assume that the salary $S_{\delta}$ is written as $$S_{\delta}=S(\delta_1,...,\delta_n)=\prod_{k=1}^n S_k^{\delta_k}$$

where $\delta=(\delta_1,...,\delta_n)$, $S_k$ is a coefficient  related to the $k^{th}$ criteria, and $\delta_k$ equals one(resp. zero) when the $k^{th}$  criteria is matched(resp. not matched). $\delta$ entirely defines a model profile.

I can rewrite $S_{\delta}$ as $$log(S_{\delta})=\sum_{k=1}^{n}{\delta_klog(S_k)}$$

or

$$Y_{\delta}=\sum_{k=1}^{n}{\delta_kY_k}$$

with $Y_k=log(S_k)$ and $Y_{\delta}=log(S_{\delta})$

I guess you know what I want to do next : I look at my database and I create a binary table for all profiles.

For example ,
$$
\begin{array}{c|lcr}
Names &  \text{Salaries}& \text{Male}& \text{Female}& \text{under 185 cm}& \text{above 185cm} & \text{17-20yo} & \text{21-24yo} \\
\hline
C.C & 200 & 1 & 0 &0 &1&0 &1  \\
A.A & 300 & 0 & 1 &1 &0&1 &0  \\
B.B & 400 & 0 & 1 &1 &0&0 &1
\end{array}
$$

Let assume I have a sample of size $N_{sample}$, I will solve
$$\min_{Y_1,...,Y_n}\sum_{l=1}^{N_{sample}}{\left(Y_{\delta_l}-Salary(l)\right)^2}$$

where $\delta_l$ and $Salary(l)$ are respectively the criteria , and the salary   of the $l^{th}$model.

$Excel$ can solve such a problem quickly by using the function $LINETEST$.

My issue is not about solving the equation , but how to pre-process the data when the input is a binary table.

Indeed, in another context , I would use the classical tool such as PCA on correlation matrix. However, here, I have a binary table , and in that case, we can clearly distinguish causality and correlation .

Besides using common sense to get rid of irrelevant factors( ex : male and above 185 cm always go along, one factor is redundant) , are there any other techniques to reduce the dimension of the problem in that context ?


texts: Should I seperate the data for a k-means model?

texts: I would like to cluster some user reviews and I'm doing this with k-means. In my dataset I have English and German reviews. Is this manipulating the cluster result if I don't seperate them? Or should I do a k-means prediction on each language?


texts: Matplotlib WIndow Not responding when loop is initialized

texts:
I am initializing empty figure when executing plt.figure(). It openes fine. But as soon as I have the following code implemented opened graph gets not responding. Any idea why this window is getting not responding?

import matplotlib.pyplot as plt
plt.figure()   # It opens graph fine

with open("data.txt") as file_data:

    while True:
        pass


texts: I have time series data from mobile sensors for different motions such as walking, pushups, dumbellifts, rowing and so on. All these motions have different length of time series. For classifying them using Dynamic Time Warping (DTW), how do I choose an appropriate window size that will give good results?


texts: Reward dependent on (state, action) versus (state, action, successor state)

texts:
I am studying reinforcement learning and I am working methodically through Sutton and Barto's book plus David Silver's lectures.

I have noticed a minor difference in how the Markov Decision Processes (MDPs) are defined in those two sources, that affects the formulation of the Bellman equations, and I wonder about the reasoning behind the differences and when I might choose one or the other.

In Sutton and Barto, the expected reward function is written $R^a_{ss'}$, whilst in David Silver's lectures it is written $R^a_{s}$. In turn this leads to slightly different formulations of all the Bellman equations. For instance, in Sutton and Barto, the equation for policy evaluation is given by:

\begin{align}
v_{\pi}(s) = \sum_a \pi(a|s) \sum_{s'} P_{ss'}^a(R_{ss'}^a + \gamma v_{\pi}(s'))
\end{align}

Whilst David Silver's lectures show:

\begin{align}
v_{\pi}(s) = \sum_a \pi(a|s) \left(R_{s}^a + \gamma \sum_{s'} P_{ss'}^a v_{\pi}(s') \right)
\end{align}

In both cases:

$\pi(a|s)$ is policy function - probability of choosing action $a$ given state $s$.
$\gamma$ is discount factor.
$P_{ss'}^a$ is transition function, probability of state changing to $s'$ given $s, a$

I understand that $R_{ss'}^a$ and $R_{s}^a$ are related (via $P_{ss'}^a$), so that these two sources are explaining the exact same thing. Note that the first equation can also be written as

\begin{align}
v_{\pi}(s)
&= \sum_a \pi(a|s) \sum_{s'} (P_{ss'}^aR_{ss'}^a + \gamma P_{ss'}^av_{\pi}(s'))\\
&= \sum_a \pi(a|s) \left( \sum_{s'} P_{ss'}^aR_{ss'}^a + \sum_{s'} \gamma P_{ss'}^av_{\pi}(s') \right) \\
&= \sum_a \pi(a|s) \left( \sum_{s'} P_{ss'}^aR_{ss'}^a + \gamma \sum_{s'}  P_{ss'}^av_{\pi}(s') \right)
\end{align}

Hence, it must be true that $R_{s}^a = \sum_{s'} P_{ss'}^a R_{ss'}^a$.

My question is whether there is any reason I should prefer to use one or the other notation?

I started with Sutton and Barto, and find that notation more intuitive - the reward may depend on the eventual state, and this is explicit in the equations. However, it looks like in practice that the notation used in the video lectures describes more efficient calculations (essentially $R_{s}^a = \sum_{s'} P_{ss'}^a R_{ss'}^a$ is cached, if the formula is translated directly to code). Is that all there is to it?


texts: Train, test split of unbalanced dataset classification

texts:
I have a model that does binary classification.

My dataset is highly unbalanced, so I thought that I should balance it by undersampling before I train the model. So balance the dataset and then split it randomly. Is this the right way ?  or should I balance also the test and train dataset ?

I tried balancing only the whole dataset and I get train accuracy of 80% but then on the test set I have 30% accuracy. This doesn't seem right ?

But also I don't think that I should balance the test set because it could be considered as bias.

What is the right way to do this?

Thanks

UPDATE: I have 400 000  samples, 10% are 1s and 90% 0s. I cannot get more data. I tried to keep the whole dataset but I don't know how to split it into train and test set. Do I need the same distribution in the train and test dataset ?


texts: How to interpret agglomerative coefficient agnes() function?

texts:
how to interpret agglomerative coefficient in agnes() function of cluster package?

Example:

res.agnes <- agnes(dataset,method = 'ward')
res.agnes$ac
0.934


texts: Can You Purposely Bias A Clustering Model?

texts:
We have a large amount (Billions) of high cardinality, mixed nominal & numerical data, and are performing some clustering on it as an experiment.  There is a small subset of these data, however, that we know should belong in its own cluster.

Is it possible to purposely make sure these data end up in their own cluster, while still clustering the remaining data?  If so, how does one approach doing so?


texts: How can I identify and remove outliers in R

texts:
I am performing regression analysis on prices of product that we have purchased, based on size and other attributes.

However there are often buys in odd circumstances which factor into the price, that is not (and cannot be) addressed directly in the features of the analysis.

Each time I run a regression, I will check the 20 with the largest error manually, and 90%+ of the time they will be odd buys like mentioned before, and for my purposes can be completely ignored.

I have been looking into cooks distance to remove these, however I'm not sure how to best set the threshold, or if there is a better method to use.


texts: EagerTensor object does not support item assignment

texts:
I'm trying to assign a new Value to a TF-Array. Here's my Code:

import tensorflow as tf
x = tf.zeros(shape=[5],dtype=tf.float32)
x[1]=0

The error message:

'tensorflow.python.framework.ops.EagerTensor' object does not support item assignment

What am I doing wrong?


texts: When would not normalizing input values have higher accuracy?

texts:
Right now I'm training a deep neural network for a binary classification problem, with a feature set of winrates. As such, each winrate is bigger or equal to 0 but smaller than 100.

I've been getting promising results without normalizing the input data, until I normalized it and got staggeringly worse accuracy.

The input feature is a 2d matrix of size 20, and the network has four layers with differing numbers of nodes in each layer. I'm using sgd optimizer and ReLU activation for the hidden layers, and the softmax activation function for the output layer.

The thing I'm wondering is why I'm getting better results with the neural network without the normalization? Is it because the optimal hyperparameters required for the network with the normalized input are different from when it is not normalized?


texts: Keras neural network entirely different results on two different hardwares (one is AWS)

texts:
I have a Keras model. which is defined as follows:

nn_model = Sequential()
nn_model.add(Dense(300, activation="relu",input_shape=(4,))) # because we have 4 features
nn_model.add(Dropout(0.3))
nn_model.add(Dense(150, activation="relu"))
nn_model.add(Dense(50, activation="relu"))
nn_model.add(Dense(30, activation="relu"))
nn_model.add(Dense(1))

optimizer = optimizers.RMSprop(0.001)
nn_model.compile(loss='mse',optimizer=optimizer,metrics=['mae'])

This model is used as a regressor for predicting a certain pattern.
For the SAME training and testing data AND SAME code, the model is making two entirely different predictions on two different hardwares. In one case, the weights become entirely corrupt and makes flat predictions (training data has no flat curve).

Versions of libraries being used are same on both the machines.

Here are the specs of two machines:

Correct predictions machine:
Lenovo ThinkPad E570
model type: Intel(R) Core(TM) i7-7500U CPU @ 2.70GHz

Wrong prediction (AWS VM):
Machine type: c5.large

Can anyone point out any reason of which this might be happening?


texts: How similar is Adam optimization and Gradient clipping?

texts:
According to the Adam optimization update rule:
$$m \leftarrow \beta_1 m + (1 - \beta_1)\nabla J(\theta)$$
$$v \leftarrow \beta_2 v + (1 - \beta_2)(\nabla J(\theta) \odot \nabla J(\theta))$$
$$\theta \leftarrow \theta - \alpha \frac{m}{\sqrt{v}}$$
From the equations, it is clear that $m$ is accumulated gradient for each $\theta$ based on an exponential decay function and $v$ is doing a similar thing (kind of) but with the magnitude of the gradient. Then, when we update the parameters $\theta$, we divide the accumulated gradient with square root of the accumulated magnitude to only update those parameters largely which haven't been updated much and vice-versa.
In gradient clipping, we do a kind of similar thing by scaling the gradient vector with respect to a threshold. My question is, why do we need gradient clipping to solve the problem of gradient explosion when we can use Adam optimizer to do a controlled search of the space for the minima.


texts: Why aren't Genetic Algorithms used for optimizing neural networks?

texts:
From my understanding, Genetic Algorithms are powerful tools for multi-objective optimization.

Furthermore, training Neural Networks (especially deep ones) is hard and has many issues (non-convex cost functions - local minima, vanishing and exploding gradients etc.).

Also I'm that conceptually training a NN with GA is feasible. I was wondering, why aren't they used in practice? Is it a matter of performance?


texts:
I have a twitter data-set and I wanna extract their related topics. So, I decided to classify my Tweets into clusters using an unsupervised machine learning algorithm like k-means. This choice is made due the time consuming of the training process in the supervised approaches.

So, as a first step after cleaning my tweets, I will extract features (eg. Hashtags...) from them, and enrich them with side information from knowledge bases (eg. Wikipedia). Secondly, they will be represented in a Vector space. Next, using k-means and for given K=6 clusters, my tweets already enriched will be classified into 6 clusters.

However, I don’t know how to identify automatically the topics related to these clusters. Is there any solutions?


texts: How to convert a time series data which can be fed to a Machine Leanring model for unsupervised learning?

texts:
I am very new to time series data. I am working with server data and want to classify if the bunch of server times are healthy or not.

My data looks like this in pandas

+-------+-----------+------------+----------------+-------+
|  ID   | timestamp | IP_address |   event_type   | usage |
+-------+-----------+------------+----------------+-------+
| 12xcv |       456 |     123573 | server_load    |  56.0 |
| 12xcv |       457 |     123573 | server_process |  59.0 |
| 12xcv |       459 |     123573 | unload         |  43.0 |
| 23vbz |       461 |     123573 | server_load    |  78.0 |
| 23vbz |       461 |     123573 | unload         |  78.0 |
+-------+-----------+------------+----------------+-------+

My questions are:

I want to use unsupervised clustering algorithm. How do I convert this data so that I can use something like k-means with two clusters such that one cluster denotes healthy and other denotes unhealthy.
What other unsupervised methods exists for classification in time series data?


texts: making logical inference from a simuation generated data

texts:
I have data collected from a computer simulation of football games which seem to have recurring patterns of the following form.

if madrid plays arsernal and the match ends under 3 goal, then on their next match against each others, madrid will win. if madrid happens to loose and then plays against chelsea next, they will win 90% of the time.

how do I find such inferences from simulation generated data like this. There are other forms of hidden patterns that I believe exists in the dataset.


texts: Sklearn feature selection stopping criterion (SelectFromModel)

texts:
Sklearn has several functions for feature selection that lets the user determine the size of the chosen subset. An example of this is SelectKBest where the user determines the value of "k", which is the number of top performing features.

Does anyone know what stopping criterion SelectFromModel uses when it selects a feature subset? The documentation mentiones a "threshold"-parameter that determines which features are important enough, and that this parameter is set to "median" OR "mean" by default.


texts:
I am trying to make a system that can recognize poses that are attained by a humanoid robotic arm; like that of asimo.

I am currently looking into the libraries that detect real human poses like openpose, as the visual similarity is of a high degree between the arms.

I was wondering if I could retrain a model that already recognizes human arms.


texts: Hyperparameter tuning for stacked models

texts:
I'm reading the following kaggle post for learning how to incorporate model stacking

http://blog.kaggle.com/2016/12/27/a-kagglers-guide-to-model-stacking-in-practice/
in ML models. The structure behind constructing the 5 folds and creating out of sample predictions on the training data makes sense for the purpose of building the meta model or the model on top of the base models. However i'm not sure how it uses hyper parameter tuning especially for the base models.

So the concept of getting out of sample predictions makes sense to me. We essentially for each of the 5 folds use the other 4 folds to train and then predict on the fifth. So how do we actually hyper parameter tune the base models on this same dataset without adding bias, it's seems to me that this is not possible?

Note i'm making the assumption that there is no more data available to use. I'd appreciate any help!


texts: How do you choose an appropriate $k$ to achieve $k$-anonymity for data?

texts: How do you choose an appropriate $k$ to achieve $k$-anonymity for a data? What methods exist that are agnostic to the business context for the problem?


texts: how to make CBOW model suitable for only 10 outpus?

texts:
I want to modify CBOW model to output only 10 classes. my question is how to make this model perform better even if the number of output is much less than the number of vocabulary output? should I use hierarchical softmax or softmax activation function ?
thanks in advnace


texts: Minimum number of trees for Random Forest classifier

texts:
I am searching for a theoretical or experimental estimation of the lower bound for the number of trees in a Random Forest classifier.

I usually test different combinations and select the one that (using cross-validation) provides the median best result.

However, I think that there may be a lower bound on the number of trees to use, given $m$ observations and $n$ features, to reduce the variance contribution to the error. Is there some test or reference I may check to correctly tune my classifier?


texts: I just studied K-Fold cross validation technique for finding model parameters and something seemed to be very confusing. Every tutorial I follow says that for K-Fold validation, the whole dataset will be split into K portions and K models will be fit with one portion as validation dataset each time. So, my doubt is that this method will generate K models. Which model are we supposed to use during actual inference? Or is it that same model will be trained K times with different portion of data as hold-out set?


texts: Finding correlation between MNIST digits

texts: What way would be correct to calculate the correlation between say digit '1' and digit '7' images from MNIST? Taking average values of all digit '1' pixels and digit '7' pixels to compute correlation between those would be a correct?


texts: One hot encoding for multiple label(trainy) in .fit() method?

texts:
I have a mobile price classification dataset in which I have 20 features and one target variable called price_range. I need to classify mobile prices as low, medium, high, very high.

I have applied a one-hot encoding to my target variable. After that, I split the data into trainX, testX, trainy, testy. So my shape for trainX and trainy is (1600,20) and (1600,4) respectively.

Now when I try to fit trainX and trainy to logisticRegresion, i.e -> lr.fit(trainX,trainy) I am getting an

  error and it says: bad input (1600,4)

So, I understood that I have to give trainy value in shape (1600,1) but by one hot encoding I have got array of 4 columns for each individual price_range as per the concept of one hot encoding.

So now I am totally confused how people use one hot encoding for target variable in practice? please help me out.


texts: How to calculate compression ratio when using autoencoder in neural network

texts:
For example, if I use an autoencoder to compress a 1000 dimensional data set to 25 dimensions. Is the compression ratio is 40:1?

Other info:
The dataset contains 5000 samples.
2 million parameters are needed for the encoder.
Another 2 million parameters are needed for the decoder.

Does these numbers have something to do with the compression ratio calculation?


texts: How can I regularize the output of a layer from scratch (without using Keras)?

texts:
I am trying to build a Convolutional Neural Network after reading notes from Stanford's cs231n course. I use ELU activation as activation function, and SoftMax as my classifier. Architecture is simple:

Convolution(5 x 5 x 32) - ELU - MaxPool(2 x 2) - Convolution(5 x 5 x 64) - ELU - MaxPool(2 x 2) - FullyConnected(hidden units = 256) - SoftMax

I wrote the architecture without using API's and it turns out fine. When I apply L2 regularization to the weights, I get test accuracy of 57% (I didn't do hyperparameter tuning; learning rate = 0.001, regularization strength = 0.001).

Recently, I found out that in keras API, there is something call activity regularization, which puts penalty based on the output(activation) of a layer. I tried to do so in my model, by putting the regularization term to my loss:

L2_regularization_weight = np.sum(W1 ** 2) + np.sum(W2 ** 2) + np.sum(W3 ** 2) + np.sum(W4 ** 2)
activity_regularization = np.sum(conv_out_1 ** 2) + np.sum(conv_out_2 ** 2)

loss += 0.5 * self.reg * (activity_regularization + L2_regularization_weight)

_, dW1, _ = conv_backward_fast(conv_out_1, conv_cache_1)

_, dW2, _ = conv_backward_fast(conv_out_2, conv_cache_2)

grads['W1'] += self.reg * (dW1 + W1)

grads['W2'] += self.reg * (dW2 + W2)

grads['W3'] += self.reg * W3

grads['W4'] += self.reg * W4

W1, W2 , W3, W4 are weights of first convolution layer, second convolution layer, Linear filter in fully connected layer, and linear filter in classification layer. Here I am regularizing based on L2 of weights and L2 of convolution layers' output. Moreover, here is my documentation for conv_backward_fast function:

conv_backward_fast(dout, cache)

Input : dout = gradient w.r.t layers after this convolution layer in the model
       (e.g. If the function is called by second convolution layer, then dout is the "product" of rates of change of loss w.r.t ELU - MaxPool - FullyConnected - SoftMax.)

       cache = All inputs to current convolution layer

Output : dx = gradient w.r.t input
         dw = gradient w.r.t weights
         db = gradient w.r.t bias

So, my first question is "Am I doing it correctly?" (Although I feel like I don't since the validation accuracy and training accuracy fall down to 27%). My second question would be "Any suggestion to do activity regularization and backward calculation correctly?"


texts: Unsynchronized time series visualization

texts:
I would like to visualize a large amount of events composed of time serie windows. A typical event would be:

Problem is, my events are not synchronized, and so if I plot them all, it would look like:

Question
Is there any way to visualize all my events so I can see their original/"typical" shape (preferably in the time domain) despite their unsynchronization ?
What I have tried so far:

Visualize features: approach is good but I have to guess what I am looking for.
Synchronized events: in the example above it might be possible, but for some other cases it won't be. (such as multiple peaks spaced with different time lapses)
Use statistical values to visualize: that was my first shot, to plot not all events but only their mean, median, p95,... Problem is, using this approach gives a deformed rendering of the data because of the unsynchronization:

What I have thought about
The main problem of applying the mean, p95 or whatever statistical function is that it is applied along an axis that is not synchronized between events. I don't know if it exists or if it is even feasible, but I was thinking about an approach that would display those statistical values checking their temporal neighbors ? I know Dynamic Time Warping deals with this unsynchronization but I am not sure if and how I could use it to plot my events in the way I'd like.
An idea of a possible output could be (any other output idea is welcomed!):

Any help, idea, would be greatly appreciated ! Thanks


texts: One-Class discriminatory classification with imbalanced, heterogenous Negative background?

texts:
I'm working on improving an existing supervised classifier, for classifying {protein} sequences as belonging to a specific class (Neuropeptide hormone precursors), or not.

There are about 1,150 known "positives", against a background of about 13 million protein sequences ("Unknown/poorly annotated background"), or about 100,000 reviewed, relevant proteins, annotated with a variety of properties (but very few annotated in an explicitly "negative" way).

My previous implementation looked at this as a binary classification problem:
Positive set = Proteins marked as Neuropeptides.
Negative set: Random sampling of 1,300 samples (total) from among the remaining proteins of a roughly similar length-wise distribution.

That worked, but I want to greatly improve the machines discriminatory abilities (Currently, it's at about 83-86% in terms of accuracy, AUC, F1, measured by CV, on multiple randomly sampled negative sets).

My thoughts were to:
1) Make this a multiclass problem, choosing 2-3 different classes of protein that will definetly be negatives, by their properties/functional class, along with (maybe) another randomly sampled set.
 (Priority here would be negative sets that are similar in their characteristics/features to the positive set, while still having defining characteristics) .
2) One class learning - Would be nice, but as I understand it, it's meant just for anomaly detection, and has poorer performance than discriminatory approaches.

*) I've heard of P-U learning, which sounds neat, but I'm a programming N00b, and I don't know of any existing implementations for it. (In Python/sci-kit learn).

So, does approach 1 make sense in a theoretical POV? Is there a best way to make multiple negative sets? (I could also simply use a massive [50K] pick of the "negative" proteins, but they're all very very different from each other, so I don't know how well the classifier would handle them as one big , unbalanced mix).
Thanks!


texts:
Noticed that spark xgboost does not have a API trees_to_dataframe() as that in Python API, I am trying to parse the getModelDump result, but I am confused on its format, which fields represents what etc.
 // train xgb_model in spark version of xgboost
scala> xgb_model
res18: ml.dmlc.xgboost4j.scala.spark.XGBoostClassificationModel = xgbc_89286dd04aa3

scala> xgb_model.nativeBooster.getModelDump(null, true);
res19: Array[String] =
Array("0:[f1<53] yes=1,no=2,missing=2,gain=58047.7812,cover=336165
        1:[f3<53.9500008] yes=3,no=4,missing=3,gain=24677.3848,cover=63748.25
                3:leaf=-0.0531237721,cover=53626.5
                4:leaf=0.031994272,cover=10121.75
        2:[f16<1.66669905] yes=5,no=6,missing=6,gain=10181.9785,cover=272416.75
                5:leaf=-0.0937986076,cover=268367
                6:leaf=-0.0139159411,cover=4049.75
", "0:[f1<51] yes=1,no=2,missing=2,gain=52816.4062,cover=336097.594
        1:[f8<369.570007] yes=3,no=4,missing=4,gain=22681.3555,cover=60529.668
                3:leaf=-0.0121749714,cover=37363.5625
                4:leaf=-0.0751453713,cover=23166.1055
        2:[f16<1.67979908] yes=5,no=6,missing=6,gain=10274.8359,cover=275567.906
                5:leaf=-0.089068912,cover=271300.188
                6:leaf=-0.0108754979,cover=4267.74268
", "0:[f1<56] yes=1,no=2,missing=2,gain=4887...

scala> res19.size
res20: Int = 200

My Model parameter settings is below :
xgbParams = {'n_estimators': 200, 'max_depth': 2, 'eta': 0.05, 'lambda':1, 'gamma':4, 'alpha':0.1, 'subsample':0.8, #'min_child_weight': 1,
         'colsample_bytree':0.8, 'objective': 'binary:logistic', 'colsample_bylevel':0.8,
         'eval_metric':'logloss', 'seed': 1122, 'missing': -999999999}

I think the res19.size = 200 makes sens, as I have set n_estimators to 200. I am confused on each string in res19, which all formatted as below:
I think f2 must represent some certain feature, but how can I find the examply feature name ? Also, what does 0, 1, 2 represent? and what does yes=3, no=4 mean ?
I have also checked understanding-python-xgboost-model-dump-output-of-a-very-simple-tree but that posts only explain the value represented by the leaf node.
Thanks in advance !!
0:[f2<0.380098999] yes=1,no=2,missing=1,gain=732.850342,cover=72529.7266
        1:[f47<31.9999981] yes=3,no=4,missing=3,gain=753.887451,cover=67352.3594
                3:leaf=4.21585646e-05,cover=63820.7422
                4:leaf=0.0237709191,cover=3531.61987
        2:[f4<1050] yes=5,no=6,missing=6,gain=410.277802,cover=5177.3667
                5:leaf=0.00518732425,cover=1373.32422
                6:leaf=-0.0266880095,cover=3804.04224

```


texts: SVM overfitting with consistent validation results

texts:
I have some imbalanced (1400 samples of which 250 are +ve) data for a binary classification problem and I am running an SVM grid search optimising for precision. I am trying 3,4,5,6,7,and 8 stratified and shuffled k-folds and in all cases I am finding the precision to be higher in training than validation (by "all" I mean all the cases of the search that return anything worth using - whilst I don't care too much about recall I can't have 5 TPs in my results).
I'm typically getting 90% training precision and 60% test precision and the SD for all k-fold runs never exceeds 5% for the test data.
This seems to go against my intuition of what overfitting is. My next steps will be to ensemble 6/7 undersampled data sets, review my feature space (reduce dimensions/try other features combinations) or try a different model altogether.
Can someone explain what might be going on here and other ways I could remedy?


texts: How to approximate constant parameters of any generalized function?

texts:
I am new to Data Science and Machine Learning.
I am trying to approximate constant parameters of any generalized not necessarily polynomial function given the input, desired output and the form of the function using neural network.
Is there any other way to do it using neural network or should I use some other technique?
For example this is one of the functions whose parameters par6 and par7 I am trying to approximate:

I have the values for x and f2(x).
Any help will be appreciated.
Thank You.


texts: selecting sample from population using genetic algorithm

texts:
I have a set of 31390 individuals with an associated weight (kg) and I want to select a number of them such that the maximum weight is 3000kgs and that the weight distribution around several variables describing the sample are close to some targets.

I was approaching the problem by considering each individual as a gene of a chromosome and then multiplying the chromosome by the weight of the sample. In this case 1 is for being included in the sample while 0 is being excluded from the sample.

I followed the example used in https://www.r-bloggers.com/genetic-algorithms-a-simple-r-example/ although my constraint is different as I want to minimize the difference between the weight distribution I get from the selected sample and my target.

Taking 1000 iterations and populations of 50 and 100 I don't get satisfactory solutions. I have read that people take populations that are (# genes)^2 but I will run out of memory if I take a population of that size.

I think crossover could help but it seems as if the genalg library only uses mutation.

Any suggestions on how to solve the problem? Thanks. I have posted the code below

Weight.Limit<-30*100

evalFunc <- function(x) {
  current_weight <- as.numeric(x %*% ind.data$weight)
  sample.dist<-x %*% as.matrix(ind.data[,4:ncol(ind.data)])/current_weight
  current_solution_dist <-  sum(sqrt(abs(constraints.data-sample.dist)))

  if (current_weight> Weight.Limit)
    return(0) else return(current_solution_dist)
}

GAmodel <- rbga.bin(size = 31390, popSize = PopSize, iters = iter, mutationChance = MutationChance, elitism = T, evalFunc = evalFunc)

cat(summary(GAmodel))


texts:
I have digit images as below which I would like to identify:

Some are of slightly worse quality :

The images are not of a fixed resolution but are mostly in the range (80*20 to 130 *40).
Due to lack of enough labelled data(~3K rows), I had to go for an open source digit dataset. I tried both MNIST and SVHN with not much luck. I've detailed both my approaches below..

Note : The only preprocessing that I've used is converting to grayscale ,resizing the images to a fixed size and doing a mean subtraction on the image.

1. SINGLE DIGIT MNIST and SLIDING WINDOW FOR TESTING : I trained a Convnet  on the MNIST set with ~99% on the validation set.Then I used a sliding window to move over my test image and predict for each. This approach had two problems : First, as my images are not of a fixed resolution often I couldn't get the right sliding window (one which contains just one digit completely). Second, I cropped out individual digits from my test images, and tested my MNIST model on these, the results were 'wildly off', out of the ~15 samples that I had cropped out, my MNIST model couldn't get even one right. Here's a cropped out image for reference.

SINGLE DIGIT SVHN I thought that the performance of my MNIST model on my test set maybe bad as these two are different type of images, numbers in the SVHN dataset are more similar to the ones I posted.
I repeated the same single digit exercise with SVHN, with far better results, on my small cropped out set of 15 numbers,this time I got 10 right but the sliding window problem still remained.
MULTI DIGIT SVHN : As house numbers in SVHN are of smaller lengths(max 5) than my numbers, I created 50k numbers of length 8 by horizontally stacking different numbers from the SVHN multi digits dataset.The best validation loss I've been able to achieve from this model is 6.70, not very great and obviously not very effective at predicting. I used a  ConvNet with the following architecture:

model_input = Input(input_shape)

  x = Conv2D(32, (3, 3),data_format='channels_first', activation='relu', name='conv_32_1',padding='same')(model_input)
  x = Conv2D(32, (3, 3), data_format='channels_first',activation='relu',name='conv_32_2',padding='same')(x)
  x = MaxPooling2D(pool_size=(2, 2))(x)
  x = Dropout(0.35)(x)

  x = Conv2D(64, (3, 3),  data_format='channels_first',activation='relu',name='conv_64_1',padding='same')(x)
  x = Conv2D(64, (3, 3), data_format='channels_first',activation='relu',name='conv_64_2',padding='same')(x)
  x = MaxPooling2D(pool_size=(2, 2))(x)

  x = Conv2D(128, (3, 3), data_format='channels_first',activation='relu',name='conv_128_1',padding='same')(x)
 x = Dropout(0.35)(x)

 x = Flatten()(x)

 x = Dense(768, activation='relu')(x)
 x = Dropout(0.50)(x)
 output_list = [Dense(11, activation='softmax',name='digit_'+str(i))(x)    for i in range(1,9)]
 adam = Adam(lr=0.0005)

 model = Model(inputs=model_input, outputs=output_list)

 model.compile(loss='categorical_crossentropy', optimizer=adam, metrics=['accuracy']).

And this is the loss curve:

My questions,going from the last approach to the 1st :

Is my MULTI SVHN model too small for this problem/dataset? If not, how do I improve its performance?
Is there a way to generate better candidate sliding windows, as my single digit SVHN seems to be the best model I have among the three.
Is my reasoning correct that the MNIST model didn't work well on my images as the digits are somewhat different (handwritten vs computer generated) ?
Are there other preprocessing  techniques that I could use to improve my performance espeically for images like the 2nd one (18458882)?
Is there any other approach that I could try to solve this?


texts: How should rolling window of parameter estimates look like?

texts:
I am using Orstein-Uhlenbeck model to model inflation: $dI_t=\theta(\mu-I_t)dt+\sigma dW_t$.
I have plotted rolling window estimates of all the parameters. However, I do not understand what to conclude from them. I know that this model assumes long-term mean and it would be nice to have convergence is estimates. I would appreciate it if you could help me with conclusions. Red is $\sigma$, pink is $\mu$ and blue is $\theta$.


texts: Efficient implementation of seperable convolution in tensorflow

texts:
It seems like the native implementation of separable convolution in tensorflow is not efficient.
https://github.com/tensorflow/tensorflow/issues/12940

Is anyone aware how can we get an efficient implementation of separable convolution in tensorflow from somewhere?

If not is there any working/efficient implementation of separable convolution in other libraries?


texts: Getting count of frequent itemsets in Python mlxtend

texts:
I have a dataset:

[['Apple', 'Beer', 'Chicken', 'Rice'],
['Apple', 'Beer', 'Rice'],
['Apple', 'Beer'],
['Apple', 'Bananas']]

I am using the frequent itemsets tools, specifically apriori, to find frequent datasets.  I convert to a OneHotDataset, then to a Data Frame:

oht = OnehotTransactions()
oht_ary = oht.fit(tempList).transform(tempList)
oht_ary

df = pd.DataFrame(oht_ary, columns=oht.columns_)
df

Then I use the Apriori function:

frequent_itemsets = apriori(df, min_support=0.01, use_colnames=True)

I see this table, which is what I expect:

support                      itemsets
0      1.00                       [Apple]
1      0.25                     [Bananas]
2      0.75                        [Beer]
3      0.25                     [Chicken]
4      0.50                        [Rice]
5      0.25              [Apple, Bananas]
6      0.75                 [Apple, Beer]
7      0.25              [Apple, Chicken]
8      0.50                 [Apple, Rice]
9      0.25               [Beer, Chicken]
10     0.50                  [Beer, Rice]
11     0.25               [Chicken, Rice]
12     0.25        [Apple, Beer, Chicken]
13     0.50           [Apple, Beer, Rice]
14     0.25        [Apple, Chicken, Rice]
15     0.25         [Beer, Chicken, Rice]
16     0.25  [Apple, Beer, Chicken, Rice]

The question I have is, is there any way to print a suppport "count" of occurrences of the datasets-meaning how many times that itemset appears in the transactions?  One thing I messed around with from the mlxtend site was this being added to add an additional column of length but couldn't get a count piece:

frequent_itemsets['length'] = frequent_itemsets['itemsets'].apply(lambda x: len(x))

EDIT:  Ok I think I am confusing folks.  What I would like to see is:

      support     itemsets    count
0      1.00       [Apple]       4
1      0.25      [Bananas]      1
2      0.75         [Beer]      3
3      0.25       [Chicken]     1

I guess I could just create a column that uses the total support and multiplies by the relative support, but was looking for a more systematic way to do it.


texts: How to do face recognition without using any kind of Neural Networks like CNN?

texts:
Is/was there any way to perform face recognition, instead of using the Convolution Neural Network which uses the technique of mapping(encoding) the face using 128-D vector and then using classifier (like knn/SVM) on it?

Before the invention of the CNN approach, what did we use for face recognition?

From a comment by @rahulreddy:

Custom object detection is possible using Voila-Jones Haar cascade or using the histogram of oriented gradients (HOG) technique or now by CNN.


texts:
I am trying to run a keras model on vast.ai using multiple GPUs. For that I am using keras.utils.multi_gpu_model , however I keep having this error:

if multi_GPU and n_GPUs > 1:
    model = multi_gpu_model(model)

  AttributeError: module 'tensorflow_core._api.v2.config' has no
  attribute 'experimental_list_devices')

I am using this default docker :

Official docker images for deep learning framework TensorFlow

  Successfully loaded tensorflow/tensorflow:nightly-gpu-py3

I have also checked the available GPUs and all the GPUs are detected correctly:

Any ideas?

cheers


texts: Constraining a neural net during training

texts:
I have simple "image-like" objects that contain non-linearly encoded information about real images. Each real image is zero except for two pixels, whose float values sum to unity.

I created a simple neural net in pytorch and began training with 1000 "image-like" objects. It sort of works without employing any constraints (two float pixels summing to unity, rest zero), but not very well. Eventually the loss function and learning rate bottom out after about 200 epochs. So I thought I would try employing the constraints.

I first tried doing this in the forward method of my neural net. I was not optimistic, but I tried it anyway because it was easy. This attempt didn't disappoint, it was truly awful.

Clearly, I have to employ Lagrange multiplier constraints in the loss function. I've seen numerous web articles on how to create custom loss functions, but I have no idea how to implement such constraints in them. Any help would be greatly appreciated.


texts: How to compare experiments run over different infrastructures

texts:
I'm developing a distributed algorithm, and to improve efficiency, it relies both on the number of disks (one per machine), and on an efficient load balance strategy. With more disks, we're able to reduce the time spent with I/O; and with an efficient load balance policy, we can distribute tasks without much data replication overhead.

There are many studies on the literature that deal with the same problem, and each of them runs different experiments to evaluate their proposal. Some experiments are specific of the strategy presented, and some others, like weak scaling (scalability) and strong scaling (speedup), are common to all of the works.

The problem is the experiments are usually executed over entirely different infrastructures (disks, processors, # machines, network), and depending on what is being evaluated, it may raise false/unfair comparisons. For example, I may get 100% of speedup in my application running on 10 machines with Infiniband connection, whereas I could get the same or even worse results if my connection was Ethernet.

So, how can one honestly compare different experiments to point out efficiency gains?


texts: How to fix the index value that I use for X_test and y_test?

texts:
I am writing code for SVR. Therefore, I have generated my code as the requirements. But I am stuck on writing codes for the index of for-loop of X_test and y_test. I have to write code as it should be associated with the line in the datasets just next to the X_train and y_train. So their index should be +1 of the ending index of X_train and y_train.

For Example:

In the first iteration (i.e. when i=0), we are using the first 1000 rows for training and the next row (i.e. the 1001st row) for testing
In the second iteration (i.e. when i=1), we are using the rows from 1 to 1001 for training and the next row (i.e. the 1002nd row) for testing
In the third iteration (i.e. when i=2), we are using the rows from 2 to 1002 for training and the next row (i.e. the 1003rd row) for testing and so on.

My full code:

import pandas as pd
import numpy as np

# Make fake dataset
dataset = pd.DataFrame(data= np.random.rand(2000,22))
dataset['age'] = np.random.randint(2, size=2000)

# Separate the target from the other features
target = dataset['age']
data = dataset.drop('age', axis = 1)

X_train, y_train = data.loc[:1000], target.loc[:1000]

X_test,  y_test  = data.loc[1001], target.loc[1001]

X_test = np.array(X_test).reshape(1, -1)
print(X_test.shape)

SupportVectorRefModel = SVR()
SupportVectorRefModel.fit(X_train, y_train)

y_pred = SupportVectorRefModel.predict(X_test)
y_pred

y_pred_list = []
y_test_list = []

for i in range(1, 2000):

    X_train, y_train = dataset.iloc[i:1000+i], target.iloc[i:1000+i]
    X_test, y_test = dataset.iloc[i], target.iloc [i]

    X_test = np.array(X_test).reshape(1, -1)
    print(X_test.shape)

    SupportVectorRefModel = SVR()
    SupportVectorRefModel.fit(X_train, y_train)
    y_pred = SupportVectorRefModel.predict(X_test)

    y_pred_list.append(y_pred)

    y_test_list.append(y_test)

print(y_test_list, y_pred_list)

I want to update my code on this below:

 X_test, y_test = dataset.iloc[i], target.iloc [i]

So how may I update this index line as an above requirement?


texts: PyTorch cross_entropy with 3D data (RNN/LSTM)

texts:
I am working on LSTMs and I want to compute cross_entropy loss, given X and y.
X.shape: (batch_size, time_steps, number_of_classes)
y.shape: (batch_size, time_steps)
y contains the ground truth indices, such as 0,1,2,3,..,10.
When I use torch.nn.functional.cross_entropy(X, y)
I get error:
ValueError: Expected target size (1000, 10), got torch.Size([1000, 1])

What is the correct way to use PyTorch's cross_entropy with this kind of Sequential model?


texts: Can we use two independently measured features in a same ML model?

texts: Two features are measured at different times but belong to the same target. In which conditions or form these features can be modeled together? Or they shouldn't be used in the same model but modeled separately? E.g. I am trying to identify users by looking at their mouse dynamics data and keyboard dynamics data, however, these two are measured independently but they belong to the same person (I have this scenario for many users). To explain it better, imagine it like I gave a text to users to repeatedly write them using the keyboard (and I recorded their keyboard usage) and similarly I gave them a different task to use mouse repeatedly (and I recorded that too). The sample size is also different for each feature.


texts: Modern Feature Selection Review/Resources

texts:
I found this review paper by Guyon and Elisseeff in a 2003 JMLR publication but, although not outdated, it is quite old. Is there a more recent review or resource on the topic of feature selection?

JMLR Review 2003


texts: Visualizing categorical data in scatter plot with Orange (version > 3.18)

texts: Why in Orange versions after 3.18 is not possible to visualize categorical data in Scatter Plot widget?


texts: Help framing a sequence prediction problem

texts:
I've found lots of tutorial/examples that focus on sequence prediction, which use previous time steps of the input variable(s) in order to create a forecast e.g. predict stock market price based on previous price.

What if I want to make a sequence prediction for a variable which is not part of the inputs (e.g. using sequences of X1 and X2 to predict a sequence for X3 at the same time step(s)). Is this the same kind of problem in the previously mentioned tutorials, just in a different context?

I've been trying to learn how to make a Sequential model in Keras, but first wanted to get some clarity on my problem and make sure i'm thinking about it in the right way.

If anyone can provide some insight into clearly framing the problem for me to start trying to build a model it would be most appreciated.


texts: How to minimalize noise from autoencoder output in audio reconstruction

texts:
I am training an autoencoder that takes an audio sample and outputs a variation of the input. The network is working as expected by the final output contains noise and it's not very clear. I am suspecting that this is normal since the autoencoder can't compress the data loosely in the bottleneck. I was wondering if there are tricks to improve the quality of the final result.
Here is the model that I am using:
class CAE(nn.Module):
    def __init__(self, **kwargs):
        super(CAE, self).__init__()
        self.encoder = nn.Sequential(
            nn.Conv1d(in_channels=1, out_channels=32, kernel_size=2048, stride=1024, padding=1024, bias=True),
            nn.Tanh()
        )
        self.decoder = nn.Sequential(
            nn.ConvTranspose1d(in_channels=32, out_channels=1, kernel_size=2048, stride=1024, padding=1024, bias=True),
        )

    def forward(self, x):
        x = self.encoder(x)
        #print("Encoder Output Shape: {}".format(x.shape))
        x = self.decoder(x)
        return x
```


texts: Given two know vector x, and b (dimension 3*1 for example), what are the ways to approximate the matrix A (dimension 3*3) so that the equality Ax=b is as close as possible (something like least square), knowing that very likely the system does not have an answer.


texts: SVR - RMSE is much worse after normalizing the data

texts:
I'm building a model using a custom kernel SVR that looks into a few of my dataframe's features and checks the proximity/distance between each pair of datapoints. The features are weigthed and the weights were calculated using cross validation.
Initially my dataframe was not normalized, and the model's results were not very good (RMSE higher than 25% of the target range). Because I had read that SVR is sensible to scale, I decided to try to normalize the data, which resulted in much worse "predictions".
The original results in terms of Root Mean Squared Error were as follows:
RMSE (test_set):  6.59
RMSE (training_set):  6.56
RMSE (validation_set):  5.90

The new results, with the normalised data are the following:
RMSE (test_set):  2404.68
RMSE (training_set):  148.06
RMSE (validation_set):  2546.44

These values are very much outside the normal range of my outcome variable. This makes me suspect I'm doing something wrong.
Notes:

I recalculated the kernel's weights after normalizing.
I normalized the data after splitting it into train/test/validation.
I didn't normalize the y (outcome variable) vector.

To normalize the data, I used the following code:
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()

def normalize(df):
    x = df.values
    df_scaled = sc.fit_transform(x)
    return pd.DataFrame(df_scaled, columns=df.columns)

X_train = normalize(X_train)
X_test = normalize(X_test)
X_validation = normalize(X_validation)

Then I use these 3 variables to train and test the model.
regressor = SVR(kernel=my_kernel)
regressor.fit(X_train, y_train)

Any tips on what I'm doing wrong? Thanks.


texts: How best to embed large and noisy documents

texts:
I have a large corpus of documents (web pages) collected from various sites of around 10k-30k chars each, I am processing them to extract relevant text as much as possible, but they are never perfect.
Right now I creating a doc for each page, processing it with TFIDF and then creating a dense feature vector using UMAP.
My final goal is to really pick out the differences in the articles, for similarity analysis, clustering and classification - however at this stage my goal is to generate the best possible embeddings.
For this type of data what is the best method to create document embeddings?
Also is it possible (is yes how) to embed various parts of the page; title, description, tags separately and them maybe combine this into a final vector?


texts:
I have a behaviours vector representing some identity. I need to binary classify [malicious or benign] each instance [ideally with a normalised severity score].

For that I can use a variety of linear classifiers/kernelized SVM/Random Forest etc...

The issue is that once the classifier has been trained I'd like to allow the user the ability to configure which behaviours are more (or less) critical.

For example, one behaviour might be encryption done by some process and a user fearing ransomware might want to make this behaviour more significant.

Given linear classifier (which I'd like to avoid) simply multiplying the given W with the given configuration would do the trick. What can be done in kernelized SVM/Random Forest/DNN etc for equivalent result?


texts:
I have a set of 1000 images of the dorsal side of the hand. I need to identify the different joints in the fingers and measure the distance between them. I have already cropped the images to same dimensions, centered the hand, and labeled the required points.

Can someone point me to the next steps and a general direction I should be looking at for training the machine I need? Which algorithms would be the closest for my need?


texts: No data point at some combination of categorical independent variables when doing linear regression

texts:
I am running a linear regression model which contains a few categorical independent variables. During the EDA, I found that some combination of these categorical variables doesn't really have data point.

Below image show the cross table/contigency table of two of the variables(var_1 and var_2). The value in the table show the number of data points available in the dataset. Some combination has plenty of data points but some have only one or two data points or even zero data points.

I am not sure if having no or very few data points at some of the combination of the variables would damage the following regression or I might potentially violate some assumptions or rules.

Let me know your thoughts. Thanks!


texts: which is the most effective(accurate) face detection method in python

texts:
i try haar_cascade for face detection and LBPH for face recognition , but the result wasn't good enough, please suggest good ways to detect and recognize faces.

my aim is to create an app which take a photograph of students and by scanning this one photograph, it will predict which student is present and which is absent


texts: Derivation of backpropagation for Softmax

texts:
So, after a couple dozen tries I finally implemented a standalone nice and flashy softmax layer for my neural network in numpy. All works well, but I have a question regarding the maths part because there's just one tiny point I can't understand, like at all.

Having any kind of activation function in the output layer, back-propagation looks like:

$$ z = x\cdot w+b \\ a = f(z) \\ E = \frac{1}{N} \sum^{N}_{i=1} (a_{i} - y_{i})^{2}
 \\ \frac{\partial E}{\partial w} := \frac{\partial E}{\partial a_{i}} \cdot\frac{\partial a_{i}}{\partial z} \cdot \frac{\partial z}{\partial w}  \\ \frac{\partial E}{\partial w} := (a_{i} - y_{i}) \cdot f^{'}(z) \cdot x $$

So by differentiating $ a_{l} $ with respect to $ z_{l} $, the result is the derivative of the activation function with $ z_{l} $ itself. Now, with Softmax in the final layer, this does not apply. If I use $ Softmax'(z_{l}) $ I get incorrect results, but I rather need $ Softmax'(a_{l}) $. Looking at a couple online materials like LINK in the Backpropagation phase section in the python code, the author also uses the activation itself in the argument for softmax's derivative and I've been differentiating there and back for the last 2 weeks, ... I don't understand why it works like that.

In short, my question is: With specifically Softmax in the output layer, why is my $ \frac{\partial a_{l}}{\partial z_{l}} $ becomes $ Softmax'(a_{l}) $ instead of $ Softmax'(z_{l}) $ like with pretty much any other normal activation function that's out there?


texts: Decision Trees - C4.5 vs CART - rule sets

texts:
When I read the scikit-learn user manual about Decision Trees, they mentioned that

  CART (Classification and Regression Trees) is very similar to C4.5,
  but it differs in that it supports numerical target variables
  (regression) and does not compute rule sets. CART constructs binary
  trees using the feature and threshold that yield the largest
  information gain at each node.

I don't understand where we compute rule sets for the C4.5 algorithm(and I dont even know what rule sets mean). Its essentially same as the CART, except that it uses gini index instead of cross entropy.

Can someone please explain what rule sets are and how they are used in C4.5 in detail?


texts:
I have a set of 20 observations, the target variable which is a binary variable (0 or 1), in this set has one unique value which is 1.

I want to predict this variable for a large sample, so I want to use a regression algorithm. The problem is that the target variable take one value, there is no observation that can describe the 0 class.

Is there any solution to handle this situation.

Thank you in advance.


texts: trade offs between number of features with its score

texts:
I am running k-mean clustering on ~200000 samples. The dataset has in total 14 features. One feature is id and the rest are categorical.

I have been playing with which features to include in the clustering and the metric Im using is Silhouette.

I would like advice on how to decide which analysis is better. A cluster with fewer features and a higher score (i.e., .8) or a cluster with more features and a lower score (i.e., 30)

My assumption is that the one with more features and a lower score is better because the algorithm has more information that describes the sample. However, those extra features may be making it harder for the algorithm to put the samples into groups.

Any advice/tips?


texts: How to provide classified feature to a neural network

texts:
Let say I have a feature that may have one of 4 values, 1,2,3,4. I want to provide it as a NN input, what is proper way to do that?

I can map it like 1 -> -1.0 | 2 -> -0.3 | 3 -> 0.3 | 4 -> 1.0 , or something similar to have mean of 0.0 and std near 1.0. But in this example 1 is much different than 4 compared to difference of 3 and 4, and I don't want such discrimination cos 1 and 4 are equally different to me as 3 and 4.

Another way is to have 4 features where each of them relates to a class 1,2,3,4 and has value of 1 if initial feature has value that matches it, and has value of 0 if it doesn't match it.

Like this 1 -> [1,0,0,0], 2 -> [0,1,0,0], 3 -> [0,0,1,0], 4 -> [0,0,0,1]
But I don't like here the fact that this feature gets to much weight, especially if it has many classes.

I was thinking to make separated layer just for this feature, do you have some better solution?


texts: How to interpret Variance Inflation Factor (VIF) results?

texts:
From various books and blog posts, I understood that the Variance Inflation Factor (VIF) is used to calculate collinearity. They say that VIF till 10 is good. But I have a question.

As we can see in the below output, the rad feature has the highest VIF and the norm is that VIF till 10 is okay.

How does VIF calculate collinearity when we are passing an entire linear fit to the function? And how to interpret the results given by VIF? Which variables are collinear with which variables?

lm.fit2 = lm(medv~.+log(lstat)-age-indus-lstat, data=Boston)
> summary(lm.fit2)

Call:
lm(formula = medv ~ . + log(lstat) - age - indus - lstat, data = Boston)

Residuals:
     Min       1Q   Median       3Q      Max
-15.3764  -2.5604  -0.3867   1.8456  25.2255

Coefficients:
              Estimate Std. Error t value Pr(>|t|)
(Intercept)  53.942455   4.823309  11.184  < 2e-16 ***
crim         -0.126273   0.029185  -4.327 1.83e-05 ***
zn            0.021993   0.012238   1.797 0.072934 .
chas          2.270669   0.768911   2.953 0.003296 **
nox         -13.959428   3.187365  -4.380 1.45e-05 ***
rm            2.619831   0.378737   6.917 1.43e-11 ***
dis          -1.374045   0.166350  -8.260 1.35e-15 ***
rad           0.286993   0.057004   5.035 6.72e-07 ***
tax          -0.010756   0.003033  -3.546 0.000428 ***
ptratio      -0.840540   0.116431  -7.219 1.99e-12 ***
black         0.008015   0.002402   3.336 0.000913 ***
log(lstat)   -8.672865   0.530188 -16.358  < 2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 4.258 on 494 degrees of freedom
Multiple R-squared:  0.7904,    Adjusted R-squared:  0.7857
F-statistic: 169.3 on 11 and 494 DF,  p-value: < 2.2e-16

> vif(lm.fit2)
      crim         zn       chas        nox         rm        dis
  1.755719   2.269767   1.062622   3.800515   1.972845   3.418391
       rad        tax    ptratio      black log(lstat)
  6.863674   7.279426   1.770146   1.340023   2.827687


texts: Machine Learning: Balanced training set but highly unbalanced prediction set? How to adjust?

texts:
I am trying to train a model to detect gender in a dataset of CEO speeches. Here are the datasets that I have:

Final Dataset: 20K CEO voices analyzed (around 95% male)
Testing dataset (?): 1K CEO voices analyzed from the final dataset, less unbalanced because I added more females on purpose (80% male)
Training dataset: 6K voices analyzed from audiobooks and TED talks (55% male).

For now, I have been trying different models by training and splitting dataset #3 (70%, 30%). I get good accuracy (95%) using this method. However, when I apply the trained model to dataset #2, I get an accuracy of 85%.

I am not sure what to do. Should I undersample women in the training dataset so that its distribution is more similar to the final dataset?


texts: Training detector without bounding box data

texts:
From what I can see most object detection NNs (Fast(er) R-CNN, YOLO etc) are trained on data including bounding boxes indicating where in the picture the objects are localized.

Is there any model that can be trained with image+label data (without bounding box data) to predict the bounding box(es) ?

I am referring following papers

Is object localization for free? – Weakly Supervised Object
Recognition with Convolutional Neural Networks
Self-taught object localization with deep networks

These papers are not recent and I am not able to find any recent papers related to this . Please let me know if there are any other resources related to available .

Thanks in advance


texts: Pros and Cons of Positive Unlabeled learning?

texts:
I've been looking for papers that discuss the pros and cons of positive unlabeled learning but I haven't been able to find anything.
I'm looking to compare the general differences between creating a positive-unlabeled based problem vs a regression classification. I have a biological dataset where it's hard to definitively define a sample as negative but I can make rules that would find something as close as possible to negative - with my idea being that I can assign scores to samples (e.g. like 0.1 instead of 0 to imply the lack of certainty but the closeness of the sample  to being negative). However, I am trying to get an understanding of if I should consider positive unlabelled learning (in theory I could label my positive samples and ignore everything else even if other samples are capable of having a close to negative label/score) but I'm struggling to find information on pros and cons of trying positive-unlabelled learning.


texts: Does Tensorflow support a Decision Tree Classifier?

texts: I am trying to implement decision tree classifier to classify my data set. I am using Python. Now it is easy to implement in scikit learn, but how can I implement this in tensorflow.


texts: For Incremental Learning ML Model do we have to perform any kind of label encoding?

texts:
Please guide me on Online / Incremental Learning ML model, I am using Creme tool for my hands-on, where as my dataset has some categorical features, I did tried to do encoding but still getting error as TypeError: unsupported operand type(s) for -: 'str' and 'float'. Please do let if we need any kind of label encoding or we should not do any encoding, I did tried passing the raw data itself, it also failed.
For example : Restaurants dataset has some features like "genre_name" which is categorical, and also has Date datatype.
I did tried with Categorical data for Label / Class, and there it works without any encoding transformation. However with features having issue. Please guide.
from creme import datasets
from creme import preprocessing
from creme import linear_model
from creme import metrics
from creme import optim

X_y = datasets.Restaurants()

model = preprocessing.StandardScaler()

model |= linear_model.LogisticRegression(optimizer=optim.SGD(.1))

for x, y in X_y:
  y_pred = model.predict_one(x)
  metric = metric.update(y, y_pred)
  model = model.fit_one(x, y)

For this as well getting same error as "TypeError: unsupported operand type(s) for -: 'str' and 'float'".


texts: Confusion about Decoder labels for training seq-to-seq models

texts: So in seq-to-seq models for say NMT, the decoder is a sequence model for the right-shifted intended output.  My question is, during training, are the inputs and outputs of the decoder supposed to be the desired labels? Or are just the outputs the desired labels and the inputs are the actual predicted output from the last timestep?


texts:
Some articles say that naive Bayes is naive because of "independence of attributes". Whereas others say "independence of attributes within a class". Can anybody please clear this confusion?
Thanks


texts: Random forest classifier is predicting only one class even when the dataset is not imbalanced

texts:
This is a binary classification task, I have 15K 1's and 11K 0's (target)

I have tried the following:

X = feature_cols
y = department_wise[['Threshold']]
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.2, random_state=1)
model = RandomForestClassifier()
model.fit(X, y)
predicted_labels = model.predict(X_test)

X_test predicts only 0 and the accuracy comes around 88%. I don't understand why since my data set is not even imbalanced.
Whatever other classifier I am trying, it shows the same result with high accuracy. Please let me know where I am going wrong.


texts: Statistical test using Gmean to compare multiple algorithms on multiple datasets

texts:
I am new in this area. I am facing some issues while comparing the algorithms using statistical test. I have following result of Gmean of some classification algorithm. Abalone, Balance-scale, Car, Chess are the datasets here and ROS, RUS, RFS, NoS are the algorithms.

If I want to compare which dataset is better than others using some statistical test like, t-test, Friedmen test, Wilcoxon test etc then is it possible that I can compare the algorithm using the following table?

                  ROS       RUS     RFS     NoS
Abalone           0.003     0.0036  0.0039  0
Balance-scale     0.8858    0.8065  0.8966  0.9417
Car               0.9191    0.7216  0.9056  0.9094
Chess             0.4912    0.1973  0.5084  0.1438

If anyone has any idea about it please help me. Or you can share any references from where I can find the solutions. I studied these statistical test, null hypothesis, p-value etc but couldn't understand whether it is possible or not to compare these algorithms using Gmean.

***G-mean = Geometric Mean, used to evaluate the performance of multiclass classifiers

Thanks in advance.


texts: Meaning of Instance Space and Concept Class, (PAC Learnable)

texts:
I'm studying Probably approximately correct learning, and I don't understand what an Instance Space and a Concept is.

I have see that wikipedia https://en.wikipedia.org/wiki/Probably_approximately_correct_learning  provides various examples, but it's still rather an abstract concept.

Could you provide me with an intuitive definition and some tangible examples?


texts: Need help with confusing dataset formats for Images and annotations

texts:
I have a more general question, which I could not answer after quite a bit of google. Actually, I got more confused. I hope you can share some light.

I am familiar with deep learning, Keras but come from time series analysis and now start in image processing.

Background

What I noticed, that most tutorial use open datasets to explain implementations of CNNs. Each dataset (COCO, ImageNet, CIFAR,…) chooses all different formatting of the annotations and representation of the images.

For example, COCO just e.g. uses the name of the image in json while others use the flattened RGB information of the image in csv.

Now I want to create my own dataset with hand-gestures. Generally, I could just give each image one tag and be finished for image classification. However, I see many implementations of complex Network architectures like RetinaNet, which use formats such as COCOs also for object (here hand) recognition.

The Question(s):

Not knowing if I will later need more sophisticated annotations, should I start with a COCO format? But if so, which should I choose of the many formats? COCOs, ImageNets,… ?

Secondly, if I later would write a converter from one to the other (for comparison), I guess it would be best if I already have the most sophisticated formatting, as downgrading is easier than upgrading the information level.

Could you give me a hint, your opinion, or an overview?

Maybe some other points which could give a clearer picture:

What is, in your opinion, the best format for hand gesture analysis?
(different gestures, different background). Or more general, which
formats suits which tasks the best?
Can you use the classic train-test split function with the COCO
format?
Does using CSV mean in most cases to put the flattened image into it,
while json the image name?
Which most know, preset network architectures from Github use which
formatting. (For using them as pre-trained networks for transfer
learning)

Thanks a lot

p.s. my answer to you might be delayed, thanks for the patience


texts: SAS procedure IF THEN ELSE loop did not recognize the Roman letters or numbers as value

texts:
I have written the following SAS procedure to display Grades based on the salary:

Not Working: This displays only 'I' for all observations irrespective of the else condition that says any salary less than 35000 should have a Grade as 'II'

DATA Test (Keep = FirstName LastName Salary Grade);
set orion.sales;
if salary < 35000 then grade = 'I';    /* Did not interpret 'I' */
else grade = 'II';    /* Did not interpret 'II' */
run;

proc print data = test;
run;

Working:

DATA Test (Keep = FirstName LastName Salary Grade);
set orion.sales;
if salary < 35000 then grade = 'G1';    /* interprets 'G1' */
else grade = 'G2';    /* interprets 'G2' */
run;

proc print data = test;
run;

So basically, the SAS did not interpret the Roman letters or numbers. Does anyone has an idea of why this is happening or any workaround to display the Roman numbers or letters?


texts: Accessing and Multiplying Individual Elements of a Layer's Output in Keras

texts:
Problem Description:

I am trying to access the individual elements (i.e., scalar) from a softmax layer's output (with dimension (,2)) and multiply this with a tensor from another model, which has a dimension of (,10). A mock-up digram describing my problem is shown in the attached Figure.

I am using Keras with Tensorflow as the back-end. So far, my approach has been the following: Lets say the output dimension of the softmax layer is (,2) (i.e., a vector of size 2). First, my plan is  to access the individual elements of this vector (tensor according to the Keras/tensorflow) using the keras.backend.gather(a_k_p,0). Where, the variable a_k_p references the SoftmaxLayer.
However, gather simply gives the entire row and does not give the individual element. So, my first question is how to access individual elements of a layer's output?

Assuming I get an answer to the above question, I am describing my idea to multiply a scalar with a tensor as follows. First, replicate this scalar element to create a vector (i.e., a tensor) of size that matches with the DenseLayerA and DenseLayerB (in this case (,10)) and perform the multiplication. Now, I am not sure if this is the right approach since I am unsuccessful in retrieving the individual elements of the Sofmax Layer's output. So, is my approach correct? if not, what is the right way to solve my problem.


texts:
I am trying to make a model of this image. Here is the relevant code:

base_model = VGG16(weights='imagenet')
conv4_3, conv3_3, conv2_2, conv1_2 = base_model.get_layer('block4_conv3').output,
    base_model.get_layer('block3_conv3').output,
    base_model.get_layer('block2_conv2').output,
    base_model.get_layer('block1_conv2').output

# Use the output of the layers of VGG16 on x in the model
conv1 = Convolution2D(256, 1, 1, border_mode='same')(BatchNormalization()(conv4_3))
conv1_scaled = resize(conv1, 56)
.
.
.
conv5 = Convolution2D(3, 3, 3, border_mode='same')(merge([ip_img, conv4], mode='sum'))
op = Convolution2D(2, 3, 3, border_mode='same')(conv5)

for layer in base_model.layers:
    layer.trainable = False

model = Model(input=base_model.input, output=op)
model.compile(optimizer='sgd', loss=custom_loss_fn)

I have a bunch of colored images in a directory. The input image should be the grayscale of an image stacked thrice(224x224x3) and the op should be the UV planes of the image(224x224x2) which I can add to the grayscale(224x224x1) to get the YUV image. The custom loss function works on the UV of the original image and UV of the prediction.

How do I train it?


texts:
I was just looking at the docs on Pytorch for different available schedulers and I found one that I am having some trouble understanding here.

The others seem to make sense: As training progresses, the learning rate gradually decreases. But in my study so far, I am yet to come across a model that needs such a "kickstart" mechanism.
Could someone please help me figure out why we need this?


texts: Visualizing Variance / Standard Deviation for categories

texts:
Data Structure:

Method Category   Variance for X
1         A             20
1         B             14
1         C             16
2         A             14
2         B             19

Where X was not used for classification, but is evaluation criteria. The objective is select method which produces classification with minimum possible variance for X for most of class / overall least variance.

My question: is there some standard (or obscure) method of visualizing variance for large number of categories?


texts: What is the class of this optimization problem?

texts:
I have the following optimization problem:
Find $\mathbf{w}$ such that the following error measure is minimised:

$E_u = \dfrac{1}{N_u}\sum_{i=0}^{N_u-1}\lVert \mathbf{w}^Tx(t_{i+1})-\mathbf{F}(\{\mathbf{w}^Tx(t_j)_{j=0,i}\})\rVert$,
$t_i \text{ being the i-th timestamp and } \lVert \cdot \rVert \text{ the } L_2 \text{ norm}$
$\mathbf{F}$ is something of the form $\sum_{j=0}^{i}\alpha_j\mathbf{w}^Tx(t_j)$ with $\sum_{j=0}^i \alpha_j = 1$.

It's important to note the $\alpha$'s are fixed (because they are from a subsystem).

With the constraints that: $\mathbf{w}>\mathbf{0}$ and $\mathbf{w}<\mathbf{L}$. Both $\mathbf{0}$ and $\mathbf{L}$ are vectors in $\mathbb{R}^6$, $\mathbf{L}$ being a vector of positive arbitrary limits I set.

Unfortunately, this doesn't look like the standard least-squares problem, due to that pesky $\mathbf{w}$ term that pops in both places (this is fixed in a certain epoch). Essentially, is like least squares but the target $\mathbf{y}$ is the value of the series at the next timestamp.

Is this another class of problems? Unfortunately, I don't have enough background on this area, but I've read something about Recursive Least Squares and Kalman filters - is this something that could be solved with this?


texts: What approach is to be taken to convert a code snippet to simple English?

texts:
Suppose a is a variable and I have a DSL (Domain Specific language) or a code snippet that says

a.size() > 0

My task is to convert it to simple English and print Number of a's is greater than zero

More examples:

Input:  a != null , Output a is not null

Input:  a != "" , Output a is not empty

Input: a.parent.children.size()>3, Output: Parent of 'a' has more than 3 children

Input: a.children().forAll(e|e.value<4), Output: All the children of 'a' have a value less than 4

I want to know what the right approach to this kind of problem would be.

I basically want to translate the code snippet to simple English, similar to how humans can have a fair/high level understanding of what the code snippet is trying to do by just reading it.

I've considered the below approaches:

One approach that I've considered to to write something like a compiler or a parser.
Another approach that I'm considering is to use machine translation, similar to how RNNs are used to translate one language to another.

How would one approach this problem?

I asked the same question on SO here but I didn't get any response yet, so decided to ask here


texts:
I am planning to use the Orange Data Mining Tool for easy data exploration and model generation. What is still unclear to me is: after finding a good model, what can I do with it, how can I use or deploy it in production?
I already found out that there is no Orange server which can run the Orange workflow. But is it at least possible to load the model in Python and use it there which was generated and saved via the Orange UI?


texts: Why to exclude features used for label generation during modeling?

texts:
I have a dataset like below without labels

But with the help of experts opinion, we generate labels based on the below 3 rules (all 3 rules has to be met to label it as 1)

So now the dataset looks like below

As you can see that my final dataset has the labels.

Now I can run a ML model for classification. Am I right?

But I read that during model building process, features that were used to create the labels will have to be excluded because they might result in perfect separation of classes and model might fail. what does it mean by fail? Aren't we aiming for separation of classes through classification algorithms?

May I know why do we have to exclude these features (Ex: RG, FG and BP features which were used to derive labels)?

It's basically my model will be built on below dataset. But aren't we losing the predictive power? why do we have to build a model by excluding those features (that were used to derive labels)?


texts:
I'm actually training a Unet model for lung tumor segmentation.
I tried out different backbones of Unet and I found out that the Resnet34 is better* than Resnext50 which is better than VGG19 for my purpose.
I want now to try the Unet ++ architecture.
I was asking myself if I could assume that best performances of Unet++ will be obtained with Resnet34 since it is what I could conclude from my experiments on Unet.
I know that Unet++is an improved version of Unet for medical images analysis but I don't know if the before mentionned assumption is True.
Thank you in advance for your help.
*better: it means that I obtained a better IoU on test dataset.


texts:
Problem
This is a question my friend in mechanical engineering asked me. However, I am not an expert in computer vision/image processing, so I come here to seek help.
Data
The following are two frames of a video, which records the positions of a 3D printer arm (the vertical bright curve).

Objective
The objective is to track the positions of the arm and do some computation based on this. I know there are basically two steps to do tracking

Step 1: identify the object in each frame of the video and draw a bounding box around the object.
Step 2: measure the relative position difference of bounding boxes of two consecutive frames.

Difficulty
The difficulty here is objective detection.
More specifically,

If I try to use model-based approach. It is not easy to model the shape of the curve since it is deformable.
If I try to use some object detection package (deep learning-based approach). I am not sure which package supports the tracking of a curve like this.

Could someone help me on this. Thank you in advance.


texts: Calculate weighted mean for two columns and hundreds of rows?

texts:
I'm trying to teach myself basics of R and I couldn't find the answer:

Say, I have a csv file and I want to calculate weighted mean for each subject such that I have a mean mu = 0.015*1030+0.16*26930+0.24*0+0.87*250+0.29*310+0.77*6240+0.98*3730+0.98*0+0.08*1400 for Subject A, for example.

How would I accomplish it in R or Excel?

Also, if I need to multiply each pair by corresponding value from the same row from the column salary:

0.015*1030*111+0.16*26930*222+0.24*0*333+0.87*250*444+0.29*310*555+0.77*6240*666+0.98*3730*777+0.98*0*888+0.08*1400*999

Thanks

UPDATE:


texts:
I have a file with 50000 rows from a library platform. Each individual row saves a user, and shows the order in which the user, has selected. The books could be from various categories (e.g. roman, history, etc..). There are a total of 10 categories. The categories that user has selected could be for example: 334664.
This means this user has selected a book from categories 3, 4 and 6. How can I use this data to build a recommendation system using the k-means cluster algorithm. If anybody can help me how I can go through the whole process step by step.


texts: How to maximize recall score for specific label in multiclass classification?

texts:
Thought I had solved the problem but I'm having inconsistent issues with it so reaching out here.
I have a multilabel classification problem with four labels ['--','-','+','++'] and with a basic random forest model, I have significant performance issues with one label '-', while the other three labels are performing pretty decently.
model = RandomForestClassifier(random_state=42)
model_name = 'RFC_base'

grid_params={}

pipe = GridSearchCV(make_pipeline(model),scoring='recall_weighted',cv=5,param_grid=grid_params,n_jobs=-1)

          recall avg    recall (++) recall (+)  recall (-)  recall (--)
RFC_base    0.848485    0.840909    0.871795    0.771429    0.893617

I'm well aware that GridSearchCV is over engineering, but I like to keep code consistent between tinkers while optimising.
I wanted to improve the recall score for '-', so created a custom scoring function that I thought would be maximising the recall score for '-'.
recall_neg_scorer = make_scorer(recall_score,average=None,labels=['-'],greater_is_better=True)

I've been tinkering with this to optimise it, hence trying to define the 'greater_is_better' param. So I do an actual GridSearchCV now changing some of the params.
model = RandomForestClassifier(random_state=42)
scaler = StandardScaler()

grid_params = {
    'randomforestclassifier__n_estimators': [81], #np.arange(start=60,stop=90,step=1),
    'randomforestclassifier__max_depth': np.arange(1,10,step=1),
    'randomforestclassifier__criterion': ['gini'], #,'entropy'],
    'randomforestclassifier__max_features': [7], #np.arange(1,X.shape[1]+1,step=1),
    'standardscaler': [StandardScaler(),MinMaxScaler(),None]
}

model_name = 'RFC_gscv_recall_neg'

pipe = GridSearchCV(make_pipeline(scaler,model),scoring=recall_neg_scorer,param_grid=grid_params,cv=5,n_jobs=-1)

What I'm finding is my scoring function doesn't actually optimise '-' when 'greater_is_better = True'
For instance, when GSCV runs on 'max_depth' it chooses 9 which gives :
          recall avg    recall (++) recall (+)  recall (-)  recall (--)
RFC_gscv_9  0.836364    0.818182    0.871795    0.742857    0.893617

Which is performing worse on '-' than even the base case. When I force max_depth = 8 it gives :
          recall avg    recall (++) recall (+)  recall (-)  recall (--)
RFC_force_8 0.842424    0.818182    0.871795    0.8        0.87234

When I choose 'greater_is_better = False' it actively tries to minimise the score. Is there something I'm screwing up here or is there a known issue that I'm missing out?
Also I'm a bit new to stack so let me know if there's something I'm missing.


texts:
I'm running into a problem while working on clustering. I work on data with white Gaussian noise. All of the methods I have come across use some sort of random initialization to set up the mean and covariance matrix of the clusters.

My question is: Since the initialization is random, there is a chance that I get a really bad starting point which gives me bad results. How do I deal with this?

One specific initialization I'm considering is the K-Means++ which is better than strictly random because it at least attempts to use the data to make informed initialization, but it too is random in the end.

Do people usually do multiple runs and take the best initialization?

What about that for streaming data?


texts: Reinforcement Learning on real time data over a web server

texts:
Question: is it possible to implement a reinforcement learning model over a NodeJS server?

This server would be receiving binary forms of data (open /close; yes/no) in real time.

The objective for the model is to learn from the incoming data for a few days and then predict values for next day (similar to time series, but not exactly). The learned parameters need to be flushed every week.

Note: the training data I have doesn't allow for supervised learning - i.e. there is no data for the target class.


texts: Applying dimensionality reduction on OneHotEncoded array

texts:
I have a really large data set with mixed variables. I have converted categorical variables to numerical using OneHotEncoding and it has resulted in more than a couple of thousand different features, combined that is.

Is it possible to apply dimensionality reduction algorithms on OneHotEncoded data which looks like [[1. 0. 1. 0.]...[0. 0. 0. 0.]] or should it be done by merging with the original data set?


texts: Is $R^2$ an appropriate evaluation metric for k-Nearest Neighbors?

texts:
I found a source that stated that $R^2$ is the ”percentage of the response variable variation that is explained by a linear model.”
(Source)

Since kNN is not a linear model (it is nonparametric), is it appropriate to use $R^2$ to evaluate it? If not, what error metric would be appropriate? I'm using the kNN classifier provided by scikit-learn in python.


texts: training one SVM model for predicting more than one response variable

texts:
Please I want to know if there is any SVM R package that can handle more than one response variable (y) at a time. that is to train one model for predicting more than one response variable. it could be regression or multi class classification problem.

Thanks for your help


texts: Does the R hts-Package work with time series of different units?

texts:
currently I'm having a look at the wonderful hts-Package. Its idea is to forecast hierarchical time series (like described here). In this example the unit of measurement across all time series is the same, e.g. people.

Does this approach (especially the combinef-Function in optimal reconciliation mode) also works for mixed unit time series? I'm thinking for example of weather-data where the temperature of one time series is measured in C° and of another in F° (same quantity, different unit). Or even more complex: Currency in different countries with daily changing exchange rates.

Aggregating such data would at least render the level 0 result useless, but does it work on the level 1, assuming that all levels below 1 use the same unit (e.g. $ or F°)?