I am looking for a right correlation metric to measure correlation between two vectors. The dimensionality of the vectors is around 100. The first vector has real values as the elements and the second vector has 0,1 values.   Spearman rho is widely used as a correlation metric between two real-valued vectors, but my case is slightly different (0,1 values in one of the vectors). Whether Mann–Whitney U test is more appropriate in my case than Spearman Rho?  I appreciate your help.
7015
I would like to find the correlation between a continuous (dependent variable) and a categorical (nominal: gender, independent variable) variable. Continuous data is not normally distributed. Before, I had computed it using the Spearman's $\rho$. However, I have been told that it is not right.  While searching on the internet, I found that the boxplot can provide an idea about how much they are associated; however, I was looking for a quantified value such as Pearson's product moment coefficient or Spearman's $\rho$. Can you please help me on how to do this? Or, inform on which method would be appropriate?  Would Point Biserial Coefficient be the right option?
71130
According to some articles (e.g. ) correlation is just a centered version of cosine similarity. I use the following code to calculate the cosine similarity matrix of the column vectors of a matrix X (slightly modified from ):  cos.sim &lt;- function(ix)  {   A = X[,ix[1]]   B = X[,ix[2]]   return(t(A)%*%B/sqrt(sum(A^2)*sum(B^2))) }    n &lt;- ncol(X)  cmb &lt;- expand.grid(i=1:n, j=1:n)  C &lt;- matrix(apply(cmb,1,cos.sim),n,n)   My question  Which modifications of the code above are needed to get the correlation matrix cor(X) instead of the cosine similarity matrix. I guess the changes are minimal but I can't see them at the moment.
110290
I have a basic question.  Say I have two random variables, $X$ and $Y$.  I can standardize them by subtracting the mean and dividing by the standard deviation, i.e., $X_{standardized} = \frac{(X - E(X))}{(SD(X))}$.  Is the correlation of $X$ and $Y$, $Cor(X, Y)$, the same as the covariance of the standardized versions of $X$ and $Y$?  That is, is $Cor(X, Y) = Cov(X_{standardized}, Y_{standardized})$?
117134
I am trying to figure out how R computes lag-k autocorrelation (apparently, it is the same formula used by Minitab and SAS), so that I can compare it to using Excel's CORREL function applied to the series and its k-lagged version. R and Excel (using CORREL) give slightly different autocorrelation values.  I'd also be interested to find out whether one computation is more correct than the other.
56977
I've recently been reading about the Mann-Whitney U test. It turns out that to carry out this test in R you actually need to run a Wilcoxon test!  My question: is the W statistic of wilcox.test in R identical to the U statistic?
97343
What is the best statistical test for investigating if there is any correlation between 2 categorical variables? Both are satisfaction scores: 1st variable is:  Overall satisfaction with the service. 1: Not at all satisfied; 10: Completely satisfied  2nd variable is:  Satisfaction with the availability of information for the service&quot; 1: Not at all satisfied; 10: Completely satisfied.
101933
I've looked through a lot of literature available online, including this forum without any luck and hoping someone can help a statistical issue I currently face:  I have 5 lists of of ranked data, each containing 10 items ranked from position 1 (best) to position 10 (worst). For sake of context, the 10 items in each lists are the same, but in different ranked orders as the technique used to decide their rank is different.  Example data:              List 1      List 2      List 3     ... etc Item 1     Ranked 1    Ranked 2    Ranked 1      Item 2     Ranked 3    Ranked 1    Ranked 2 Item 3     Ranked 2    Ranked 3    Ranked 3 ... etc   I am looking for a way to interpret and analyse the above data so that I get a final result showing the overall rank of each item based on each test and its position, e.g.  Result Rank 1 = Item 1 Rank 2 = Item 3 Rank 3 = Item 4 ... etc   So far I have attempted to interpret this information from performing Pearson's Correlation, Spearman's Correlation, Kendall Tau's B, and Friedman tests. I have found however, that these results have generally paired my lists (i.e. compared list 1 to list 2, then list 1 to list 3 .. etc), or have produced results such as Chi-Square, P-Values etc about the overall data.   Does anyone know how I can interpret this data in a statistically sound method (at a post graduate / PhD applicable level) so that I can understand the overall ranks signalling the importance of each item in the list across the 5 tests please? Or, if there is another type of technique or statistical test I can look into I would appreciate any hints or guidance.   (It maybe also worth noting, I have also performed the simpler mathematical techniques such as sums, averaging, minimum - maximum tests etc, but do not feel these are statistically important enough at this level).  Any help or advice would be greatly appreciated, thank you for your time.
45053
Which inter-rater reliability methods are most appropriate for ordinal or interval data?  I believe that "Joint probability of agreement" or "Kappa" are designed for nominal data. Whilst "Pearson" and "Spearman" can be used, they are mainly used for two raters (although they can be used for more than two raters).  What other measures are suitable for ordinal or interval data, i.e. more than two raters?
3721
For building a recommendation system, I also use the Pearson correlation coefficient. This is the definition:  $r(x, y)=\frac{\sum_{i=1}^n (x_i-\bar{x})(y_i-\bar{y})}{\sqrt{\sum_{i=1}^n (x_i-\bar{x})^2 \cdot \sum_{i=1}^n (y_i-\bar{y})^2}}$  $x$ and $y$ are part of $\mathbb{R}$.  Now for coding, it is important to take care of all potential outcomes. For example, if the denominator is zero, you will have to filter that or throw an exception.  I came up with some arguments, one of them being that if all values of $x_i$ and/or $y_i$ were equal to the average of $x$ and/or $y$, then the denominator would be zero.  But how can I prove that the coefficient is either undefined (zero denominator) or in between -1 and 1? What is the best approach?
16601
I have a question regarding correlation inference. Consider, I have two sets of variables X and Y. For an x element of X I know the correlation to an unknown variable z. I also have the covariance matrix X and Y. Moreover, I know a set of (y1,y2,y3) element of Y, which model best my variable x. So how to infer the correlation of (y1,y2,y3) and z?  I need a function: f(corr(x,z), corr(x,(y1, y2, y3)), cov(X,Y)) --> corr((y1,y2,y3),z)  This might be a very simple question, but I have no idea how to do it or for what I can search in the literature.  Thank you in advance!
62898
I’m trying to generate correlated random sequence with mean = $0$, variance = $1$, correlation coefficient = $0.8$. In the code below, I use s1 &amp; s2 as the standard deviations, and m1 &amp; m2 as the means.   p = 0.8  u = randn(1, n) v = randn(1, n) x = s1 * u + m1 y = s2 * (p * u + sqrt(1 - p^2) * v) + m2   This gives me the correct corrcoef() of 0.8 between x and y. My question is how can I generate a series means if I want z that is also correlated with y (with the same correlation $r=0.8$), but not with x. Is there a particular formula I need to know? I found  but couldn't understand it.
19356
I'm trying to understand better the difference between  covariance and correlation, besides the fact that the correlation coefficient is a dimensional and has values between $-1$ and $1$.   One unclear point is the following: the correlation coefficient can highlight only a linear relation between two variables, but does  covariance only work for linear relations too? Supposing that the relation is not linear, is the covariance not zero? Is it bigger than in the case in which the relation is linear?   Cauchy inequality says $$|\sigma_{xy}|&lt;\sigma_{x} \sigma_{y}$$  Which seems to say that the covariance is maximum when there is a perfect linear relation.    So I do not get what does happen if the relation is not linear, and, if covariance is not a suitable parameter in that cas I don't get the reason for that, since simply looking at the definition:  $$\sigma_{xy}=\frac{\sum(x-\bar{x})(y-\bar{y})}{N}$$  it looks like that this should be non zero for any kind of relation (like parabolic).  So can this be a difference between the use of covariance and correlation coefficient? (The first is useful for any kind of relation, the second one only for linear ones).
30822
I've received a results from a Mann-Whitney rank test that I don't understand.  The median of the 2 populations is identical (6.9). The uppper and lower quantiles of each population are:   6.64 &amp; 7.2 6.60 &amp; 7.1   The p-value resulting from the test comparing these populations is 0.007. How can these populations be significantly different? Is it due to the spread about the median? A boxplot comparing the 2 shows that the second one has far more outliers than the first. Thanks for any suggestions.
109616
Suppose we have $N$ measurable variables, $(a_1, a_2, \ldots, a_N)$, we do a number $M &gt; N$ of measurements, and then wish to perform  on the results to find the axes of highest variance for the $M$ points in $N$-dimensional space.  (Note: assume that the means of $a_i$ have already been subtracted, so $\langle a_i \rangle = 0$ for all $i$.)  Now suppose that one (or more) of the variables has significantly different characteristic magnitude than the rest.  E.g. $a_1$ could have values in the range $10-100$ while the rest could be around $0.1-1$.  This will skew the axis of highest variance towards $a_1$'s axis very much.  The difference in magnitudes might simply be because of an unfortunate choice of unit of measurement (if we're talking about physical data, e.g. kilometres vs metres), but actually the different variables might have totally different dimensions (e.g. weight vs volume), so there might not be any obvious way to choose "comparable" units for them.  Question:  I would like to know if there exist any standard / common ways to normalize the data to avoid this problem.  I am more interested in standard techniques that produce comparable magnitudes for $a_1 - a_N$ for this purpose rather than coming up with something new.  EDIT:  One possibility is to normalize each variable by its standard deviation or something similar.  However, the following issue appears then: let's interpret the data as a point cloud in $N$-dimensional space.  This point cloud can be rotated, and this type of normalization will give different final results (after the SVD) depending on the rotation.  (E.g. in the most extreme case imagine rotating the data precisely to align the principal axes with the main axes.)  I expect there won't be any rotation-invariant way to do this, but I'd appreciate if someone could point me to some discussion of this issue in the literature, especially regarding caveats in the interpretation of the results.
17128
I have two datasets, one containing the percentage of people with flu and one which has the number of searches for Tea-recipes in google. Problem is, I get different results when i choose to include the percentage number instead of the real number of people with flu.  Im using Excel function Correl(peoplwithflupercentage, teasearches)  My question is, can I somehow use percentages to look for correlation with real numbers. Like correlation between a dataset (percentage numbers) and another dataset (real numbers)? I know this probably is a simple question, but  can't seem to find a good example in the forum or google.
6696
I'd like to estimate the correlation between:  An ordinal variable: subjects are asked to rate their preference for 6 types of fruit on a 1-5 scale (ranging from very disgusting to very tasty) On average subjects use only 3 points of the scale.  A continuous variable: the same subjects are asked to quickly identify these fruits, which results in an mean accuracy for the 6 fruits.   Is Spearman rho the best method to analyze these data and/or are there other good methods I could consider?
57590
I've got another moment generating function question. I've uploaded the question as a link.   Joint Moment Generating Question:   My Solution: Not exactly sure how to approach this one  Also just another clarification. If the  correlation between random variables X and Y is zero, are X and Y are independent? I do know that if Covariance(X,Y) = 0, X and Y are said to be uncorrelated. But how about for correlation?  Thank you so much for your help guys!! Much appreciated =)