I am working on this problem in which I have a dataset of $n$-dimensional examples that come from different and unknown distributions.  Given a new sample, I wish to find $k$ examples from the dataset that come from distribution(s) closest to the new sample.  Which measure (Kullback-Leibler vs Hellinger Distance) might be more suitable for this and why?
120678
I'm looking for an intuitive explanation for the following questions:  In statistics and information theory, what's the difference between Bhattacharyya distance and KL divergence, as measures of the difference between two discrete probability distributions?  Do they have absolutely no relationships and measure the distance between two probability distribution in totally different way?
27851
Suppose we have 2 kernel functions $K_1(x,y)$ and $K_2(x,y)$. We know, that the dataset ($(x_1,y_1),\ldots,(x_l,y_l),$ $y_i \in \{-1,1\}$ ) is separated with the first one (that is, there are $w,$ $w_0$: $$y_i(K_1(w,x_i)-w_0)&gt;0 $$ for all $i=1,\ldots,l$ ), and not separated with the second kernel function. What we can say about kernel function $K_1(x,y)+K_2(x,y)$ ? How I can show that the same dataset is separated with it?
88907
This is for my hw, and if anyone can solve the first part of the question it will be great. Here is the question:  Assume a two-class problem with equal a priori class probabilities and Gaussian class-conditional densities as follows:  $$p(x\mid w_1) = {\cal N}\left(\begin{bmatrix} 0 \\ 0 \end{bmatrix},\begin{bmatrix} a &amp; c \\ c &amp; b \end{bmatrix}\right)\quad\text{and}\quad p(x\mid w_2) =  {\cal N}\left(\begin{bmatrix} d \\ e \end{bmatrix},\begin{bmatrix} 1 &amp; 0 \\ 0 &amp; 1 \end{bmatrix}\right)$$  where $ab-c^2=1$.   Find the equation of the decision boundary between these two classes in terms of the given parameters, after choosing a logarithmic discriminant function. Determine the constraints on the values of a, b, c, d and e, such that the resulting discriminant function results with a linear decision boundary.
22100
I want to sample from a univariate density $f_X$ but I only know the relationship:  $$f_X(x) = \int f_{X\vert Y}(x\vert y)f_Y(y) dy.$$  I want to avoid the use of MCMC (directly on the integral representation) and, since $f_{X\vert Y}(x\vert y)$ and $f_Y(y)$ are easy to sample from, I was thinking of using the following sampler:   For $j=1,\dots, N$.  Sample $y_j \sim f_Y$. Sample $x_j \sim f_{X\vert Y}(\cdot\vert y_j)$.   Then, I will end up with the pairs $(x_1,y_1),...,(x_N,y_N)$, and take only the marginal samples $(x_1,\dots,x_N)$.  Is this correct?
38078
I need to calculate the sample Mahalanobis distance in R between every pair of observations in a $n \times p$ matrix of covariates. I need a solution that is efficient, i.e. only $n(n-1)/2$ distances are calculated, and preferably implemented in C/RCpp/Fortran etc. I assume that $\Sigma$, the population covariance matrix, is unknown and use the sample covariance matrix in its place.  I am particularly interested in this question since there seems to be no "consensus" method for calculating pairwise Mahalanobis distances in R, i.e. it is not implemented in the dist function nor in the cluster::daisy function. The mahalanobis function does not calculate pairwise distances without additional work from the programmer.  This was already asked here , but the solutions there seem incorrect.   Here is a correct but terribly inefficient (since $n \times n$ distances are calculated) method:  set.seed(0) x0 &lt;- MASS::mvrnorm(33,1:10,diag(c(seq(1,1/2,l=10)),10)) dM = as.dist(apply(x0, 1, function(i) mahalanobis(x0, i, cov = cov(x0))))   This is easy enough to code myself in C, but I feel like something this basic should have a preexisting solution. Is there one?  There are other solutions that fall short: HDMD::pairwise.mahalanobis() calculates $n \times n$ distances, when only $n(n-1)/2$ unique distances are required. compositions::MahalanobisDist() seems promising, but I don't want my function to come from a package that depends on rgl, which severely limits others' ability to run my code. Unless this implementation is perfect, I'd rather write my own. Anybody have experience with this function?
85242
I have a process which consists of a number of events and what is known is the timings between the events. What I'm trying to determine is a distribution that allows me to determine a likelyhood that a new sample fits the distibution.  The issue is mainly that if you have lots of samples you can approximate the result using a standard gaussian and use the mean and standard deviation. But if you only have a handful of samples, the gaussian does not accurately represent the situation.  From what I've read it is common to model waiting times using the gamma distribution. Looking at how the process evolves it looks like it matches well. The unknown is the scale parameter, since the shape parameter I think should be the number of samples. What I've worked out so far is that given the timings $X_1 ... X_N$ you can say:  $$ \sum_{n=1}^N X_n \sim \Gamma(N,\theta) $$  ($N$ is known and fixed)  However, $\theta$ is unknown, but the maximum likelihood parameter is the average of the $X_i$ (according to wikipedia anyway).   My question is, can I use this to estimate a distribution for $X_i$, that is, since the $X_i$ are independent:  $$ N X_i | \sum_n X_n \sim \Gamma(N, \tfrac{1}{N}\sum_n X_n) $$  Something else I've wondered about. Suppose I do have information about $\theta$, say a distribution. How can I incorporate this into the model?  Edit: Clarified that N is fixed.
62001
I want to use the Kolmogorov-Smirnov test to test if a sample is drawn from a Pareto distribution. Unfortunately, the only way to estimate the distribution's parameters is from the sample.  Does anybody know of the existence of tables with modified critical values or any other way to perform the test in this case?
27718
I have seen somewhere that classical distances (like Euclidean distance) become weakly discriminant when we have multidimensional and sparse data. Why? Do you have an example of two sparse data vectors where the Euclidean distance does not perform well? In this case which similarity should we use?
32777
Definition of "Consistent nearest neighbour", runs our usual KNN classifier but instead of viewing k as a hyper-parameter it always sets k = ceil[log(n)].  So far, I looked-up many references and websites and researched on how to determine if a method is between parametric or non-parametric. I came up with below definitions,     A parametric algorithm has a fixed number of parameters. In contrast, a non-parametric algorithm uses a flexible number of parameters, and the number of parameters often grows as it learns from more data.   From .  Moreover, I found,     A parametric model, we have a finite number of parameters, and in nonparametric models, the number of parameters is (potentially) infinite.    From .  My question is, why do we count it as a Non-parametric model, while    the number of parameters can increase to infinity in the mentioned    method and it will gets more complicated by increasing test data?
49905
How do  methods compare in terms of model variance and bias?   My question is partly motivated by this thread: . The answer there suggests that models learned with leave-one-out cross-validation have higher variance than those learned with regular $K$-fold cross-validation, making leave-one-out CV a worse choice.  However, my intuition tells me that in leave-one-out CV one should see relatively lower variance between models than in the $K$-fold CV, since we are only shifting one data point across folds and therefore the training sets between folds overlap substantially.  Or going in the other direction, if $K$ is low in the $K$-fold CV, the training sets would be quite different across folds, and the resulting models are more likely to be different (hence higher variance).  If the above argument is right, why would models learned with leave-one-out CV have higher variance?
2191
I have read through existing answers on CrossValidated (plus elsewhere online) and can't find what I'm looking for, but do please point me to existing sources if I've missed them.  Let's say I have a data set of N=1000 records, each of which can be manually sampled and labelled as either 'Valid' or 'Invalid' (or True/False, Right/Wrong, etc).    I want to achieve a given level of confidence that all records in the data set are Valid.  As I sample records, if I find a single Invalid one I would go back and amend how the data set is created to rectify that and similar problems.    So, after some iterations of spotting Invalids, fixing and recreating the data set, I do some sampling that only includes Valid records.  If I want to be (say) 99% or 95% sure that all records are Valid, how big does my sample have to be? (Ideally as a function of N.)  I've tried playing around with Hypergeometric tests () - in that context I want to know what k should be, but I don't have a fixed value of K. Rather I want to choose k such that K is likely to be equal to N - but setting K=N obviously works out to a Probability of 1! I'm also wondering if I need to use a Bayesian approach but I don't understand Bayesian stats enough.
74366
When we do k-fold cross validation, should we just use the classifier that has the highest test accuracy? What is generally the best approach in getting a classifier from cross validation?
121830
I want to compare the classification performance of two heuristics: $h_0$ and $h_1$. I have around 700 samples for which I know the output class $c$ and the results of the two heuristics $c_{\textit{h_0}}, c_{\textit{h_1}}$.  That is, for each sample I have $\text{sample}_{\textit{id}}, c, c_{\textit{h_0}}, c_{\textit{h_1}}$.   $h_0$ has around ~71% correct predictions of the output class and $h_1$ has around ~77% correct. What I want to calculate now is whether this performance difference is just by chance or if the prediction performance of $h_1$ is statistical significant better than that of $h_0$.  I have found , but I don't know whether I can just put my data in there like the following matrix to calculate a p-value.  $$   \begin{matrix}                                &amp; h_0 &amp; h_1 \\     \text{correct prediction} &amp; 497 &amp; 539 \\     \text{wrong prediction}   &amp; 203 &amp; 161   \end{matrix} $$  I don't know if I can calculate the p-value like that or if I'm completely on the wrong track. I would really appreciate it if someone could point me in the right direction.
80432
A large amount of computing resources is required to compute exact probabilities for the Kruskal–Wallis test. Existing software only provides exact probabilities for sample sizes less than about 30 participants. These software programs rely on asymptotic approximation for larger sample sizes. Exact probability values for larger sample sizes are available. Spurrier (2003) published exact probability tables for samples as large as 45 participants.[7] Meyer and Seaman (2006) produced exact probability distributions for samples as large as 105 participants.[8]
78383
I have data set with date features like 01/01/2019 and I would like to use KNN. However, I cannot find a good transformation for dates that has a meaningful distance result for the last feature.  For example:  f1 | 1 | 2 | 3 | 4 | 01/01/2019  f2 | 10 | 3 | 12 | 1 | 14/01/2019  Does anyone have any recommendations?
52588
We know that the original KS-test has a limitation, requiring the Null hypothesis testing underlying distribution to be fully specified, rather than estimated. But in practice, we usually need to test goodness of fit for the fitted distribution.  It is suggested to conduct a Monte Carlo Simulation by . But there are very few limited literature review about such simulation. There are several ideas come up in my mind.    Approach 1  It seems that the reason why original KS-test is no longer valid for testing estimated distribution, is that we need a smaller critical threshold KS Distance (d value) to reject the null hypothesis underlying distribution, as it is fitted around the data.  The  solves such problem. Although it is only for Normal distribution, we can extend its method.   Find the KS Distance ($D_{0}$) between the data (n data points) and the estimated underlying distribution, like original KS-test. Generate n data points by the estimated underlying distribution for M times (around millions), find their KS Distance $D_{m}$ set = {$D_{1}$, $D_{2}$, $D_{3}$, ..., $D_{M}$} for each time. Compare the $D_{0}$ value and the percentile of $D_{m}$ set as the p-value.   $P(&gt;D_{0}|Null Hypothesis Distribution)$    Approach 2  Some post said that the original KS-test can not be applied to estimated distribution, because it does not take into account the standard error of estimation (or fitting).  So they suggest bootstrap from raw data as a simulation, but they have not described the details. I think the approach can be as follow.   Find the KS Distance ($D_{0}$) between the data (n data points) and the estimated underlying distribution A, like original KS-test. Bootstrap from raw data and fit distribution again, getting a new fitted distribution B. Find the KS Distance ($D_{1}$) between this new raw sample and new distribution B. Repeat step 2 for M time, getting their KS Distance $D_{m}$ set = {$D_{1}$, $D_{2}$, $D_{3}$, ..., $D_{M}$} for each time. Compare the $D_{0}$ value and the percentile of $D_{m}$ set, to see whether it is out of the 95% empirical interval.   It seems that this takes into account the estimated error (or confidence interval) about the estimation, which affecting the original KS-test. But it seems that this approach does not follow the p-value definition, or hypothesis definition, assuming Null hypothesis and find the sample statistics probability.    Approach 3  My fellow also comes up an idea, like the approach 2.   Find the KS Distance ($D_{0}$) between the data (n data points) and the estimated underlying distribution A, like original KS-test. Set the Null hypothesis that the data is from the distribution in question (A). Generate n points from the distribution in question (A) and fit the same distribution to this new sample which gives parameter set B. Measure the KS distance $D_{1}$ between this sample and the distribution in question with parameter set B.  Repeat step 3 a large number of times (M) which gives a distribution of KS distances $D_{m}$ assuming the actual data is from the distribution in question using our fitting method. If distance $D_{0}$ is greater than 95% of the distances calculated in 4 then reject the Null hypothesis that the data is from the distribution in question with parameter set A.   It seems that it is something parametric bootstrap from approach 2. But it appear strange to generate such sample and fit data again. And it seems still not following the hypothesis definition.    Do you have any idea about these three approaches? or any modification?  Or do you find any formal literature saying simulation details for KS-test for estimated distribution?  I only find  with details about this. But sorry I can not find the full text about this book. Does anyone have any other electronic version of similar details about simulation KS-test?  Does Anderson-Darling test also suffer from this estimated parameter issue?  Any ideas or discussion are highly appreciated.  Thank you.
75272
I have a cluster of p-dimensional points and given a new p-dimensional point $x$ I want to determine whether or not it is likely to belong to this cluster.   The cluster is made up of $n$ p-dimensional points, I am making the assumption that these points are drawn from a multivariate Gaussian distribution with sample mean ${\hat{\mu_X}}$ and sample covariance matrix $\hat{\sigma_X}$.  Given a new point $x$ I am trying to decide if it is likely to belong to this cluster using the following threshold on the : $$\frac{n}{(n-1)^2}\left(X_i-\hat{\mu}_X\right)'\hat{\sigma}_X^{-1}\left(X_i-\hat{\mu}_X\right)&gt;B_{0.95}\left(\frac{p}{2},\frac{n-p-1}{2}\right)$$  However, some clusters have very few sample points $n$ in which case calculating the inverse covariance matrix $\hat{\sigma_X}^{-1}$ becomes impossible.  Are there any other equivalent or more appropriate measure I can use in this case?