I have two vectors, a1 and a2. They're identical except for a certain lag:    I expected their cross-correlation to have a maximum of 1 at their lag. However that's not the case: their max cross-correlation is close to 1, but not quite (0.9998). (If I just remove the lag manually and correlate, I of course get a correlation of exactly 1.)  My intuitive understanding was that cross-correlation took two vectors, "slide one over" by a certain amount, and calculated a correlation. Clearly that's not the case, because I don't get a value of exactly 1. Why isn't the peak 1?  (This question is followup to , where the discrepancy is much bigger, so I take this is a general feature of ccf.)  Code  # make vectors a0 = runif(25) a1 = c(rep(0.5, 2), a0) a2 = c(a0, rep(0.5, 2))  # plot require(ggplot2) ggplot() + geom_line(aes(x=1:27, y=a1), color="red") + geom_line(aes(x=1:27, y=a2), color="blue")  # cross-correlate tmp = ccf(a1, a2, type="correlation", plot=F) print(paste("maximum correlation is", max(tmp$acf)))
122491
I have two sigmoid functions, sig1 and sig2, made with this function sigmoid = function(x, A =1, mu=0, ss = 1)  A*1 / (1 + exp(-(x-mu) * ss)). Since they have a true offset of 10 (mu1=50 and mu2=60) I expected their cross-correlation function to peak at a lag of 10. My non-mathematical intuition is that cross-correlation "slides" one of the curves over by a given lag, correlates, and repeats for multiple lags. When I do this by myself in code (my.ccf in the code below) I do directly recover the lags I designed into the curves. However, ccf, the real R cross-correlation function, returns max lag of 4 (see below). What's going on?    To replicate, I did the same thing for two sines. They have a designed "lag" of 10 (see below). Here, the max lag returned by ccf (9) is closer to the designed-in value, but isn't exactly equal. My intuitive function returns 10, the "correct" answer.    Why doesn't the max of the two cross-correlation functions (4 and 9) exactly equal the lag I coded into the curves (10 and 10, respectively)? What's wrong with my intuition? Edit: As pointed out by Whuber, why isn't the max of ccf 1, since these are identical and perfectly aligned vectors.  (code)  # sigmoid function sigmoid = function(x, A =1, mu=0, ss = 1)  A*1 / (1 + exp(-(x-mu) * ss))  # my intuition my.ccf = function(x,y, lag=20) {   lags = -lag : lag    # add padding to y   y.padded = c(rep(NA,lag), y, rep(NA,lag))    # correlate   rr = numeric(length(lags))   for (ii in 1:length(lags)) {     # apply lag to y.padded     I = (1:length(x)) + (ii-1)     y.lagged = y.padded[I]      rr[ii] = cor.test(x, y.lagged)$estimate   }   return(rr) }  # make sigmoids and cross-correlate sig1 = sigmoid(1:100, mu=50, ss=1/3) sig2 = sigmoid(1:100, mu=60, ss=1/3) ccf.sig=ccf(sig1, sig2, plot=F) rr.sig = my.ccf(sig2,sig1,lag=16)  # do the same with sines sin1 = sin((1:100) * 4*pi/100) sin2 = sin(((1:100) - 10) * 4*pi/100) ccf.sine=ccf(sin1, sin2,plot=F) rr.sine = my.ccf(sin2,sin1,lag=16)  # plot sigmoids + ccf.sig p1 = ggplot() + geom_line( aes(x=1:100,y=sig1)) +    geom_line(aes(x=1:100,y=sig2)) + ggtitle("sigmoids 1 and 2") p2 = ggplot(data.frame(lag=ccf.sig$lag, corr=ccf.sig$acf), aes(x=lag, y=corr)) +    geom_line() + ggtitle("ccf function") p3 = ggplot() + geom_line(aes(x=-16:16, y=rr.sine)) + ggtitle("my intuition") p1 + p2 + p3  # plot sines + ccf.sig p1 = ggplot() + geom_line( aes(x=1:100,y=sin1)) +    geom_line(aes(x=1:100,y=sin2)) + ggtitle("sines 1 and 2") p2 = ggplot(data.frame(lag=ccf.sine$lag, corr=ccf.sine$acf), aes(x=lag, y=corr)) +    geom_line() + ggtitle("ccf function") p3 = ggplot() + geom_line(aes(x=-16:16, y=rr.sig)) + ggtitle("my intuition") p1 + p2 + p3
19356
I'm trying to understand better the difference between  covariance and correlation, besides the fact that the correlation coefficient is a dimensional and has values between $-1$ and $1$.   One unclear point is the following: the correlation coefficient can highlight only a linear relation between two variables, but does  covariance only work for linear relations too? Supposing that the relation is not linear, is the covariance not zero? Is it bigger than in the case in which the relation is linear?   Cauchy inequality says $$|\sigma_{xy}|&lt;\sigma_{x} \sigma_{y}$$  Which seems to say that the covariance is maximum when there is a perfect linear relation.    So I do not get what does happen if the relation is not linear, and, if covariance is not a suitable parameter in that cas I don't get the reason for that, since simply looking at the definition:  $$\sigma_{xy}=\frac{\sum(x-\bar{x})(y-\bar{y})}{N}$$  it looks like that this should be non zero for any kind of relation (like parabolic).  So can this be a difference between the use of covariance and correlation coefficient? (The first is useful for any kind of relation, the second one only for linear ones).
46062
In my understanding, highly correlated variables won't cause multi-collinearity issues in random forest model (Please correct me if I'm wrong). However, on the other way, if I have too many variables containing similar information, will the model weight too much on this set rather than the others?   For example, there are two sets of information (A,B) with the same predictive power. Variable $X_1$,$X_2$,...$X_{1000}$ all contain information A, and only Y contains information B. When random sampling variables, will most of the trees grow on information A, and as a result information B is not fully captured?
105422
If you have two points, $A$ and $B$, at $(1,1,1)$ and $(1,1,7)$ respectively, and a unit vector $C (0,0,1)$. What's a way to find if unit vector C, will cross B if C extends forever. (Unit vector C has a starting point at A)  Basically, imagine point A is a person, point B is an object. C is a unit vector. Is person A looking directly at object B given A's unit vector?  Since  $A\cdot B = ||A||\;||B||\cos\theta$ could I say, if the dot product of A and B is equal to $||A|| \; ||B||$, then it would mean $\cos θ = 1$, therefore $θ = 0$ and the line crosses B.
18045
Let's say I test how variable Y depends on variable X under different experimental conditions and obtain the following graph:    The dash lines in the graph above represent linear regression for each data series (experimental setup) and the numbers in the legend denote the Pearson correlation of each data series.  I would like to calculate the "average correlation" (or "mean correlation") between X and Y. May I simply average the r values? What about the "average determination criterion", $R^2$? Should I calculate the average r and than take the square of that value or should I compute the average of individual $R^2$'s?
37401
If I run the R code below, it will generate two independent vectors then test them to see if they are related in some way (i.e. p-value &lt; 0.05).  If I repeat this 1,000 times, then 50 of these (5%) will have a false positive, with a p-value &lt; 0.05. This is a type 1 error.  If we increase sampleSize to 1000, or even 100,000, the results are the same (5% false positives).  I am struggling to understand this, as I would have thought that if we collected enough samples, the chance of a false positive would drop towards 0 (just like correlation).  So, I guess my question is: "How is it possible that the number of false positives, based on p-values generated from comparing two independent datasets, be independent of sample size?".  # R code to demonstrate that with a large dataset, we can still  # get significant p-values, purely by chance.  # Change this to 1000, and we still get the same number of false positives (50, or 5%). sampleSize = 20  cat("Sample size:",sampleSize,"\n")  set.seed(1010093) n=1000 pValues &lt;- rep(NA,n) for(i in 1:n){     y &lt;- rnorm(sampleSize)     x &lt;- rnorm(sampleSize)     pValues[i] &lt;- summary(lm(y ~ x))$coeff[2,4] } # Controls false positive rate fp = sum(pValues &lt; 0.05) cat("Out of ",n," tests, ",fp," had a p-value &lt; 0.05, purely by by chance.\n",sep="")  ----output---- Running "true positives - none.R" ... Sample size: 1000  Out of 1000 tests, 52 had a p-value &lt; 0.05, purely by by chance.
103313
Is there any adjustment required when computing the correlation coefficient between:   continuous and discrete variable  two discrete variables ?    Thanks !
19588
I was wondering if anyone could give me the intuition behind the cross product of two vectors $\textbf{a}$ and $\textbf{b}$. Why does their cross product $\textbf{n} = \textbf{a} \times \textbf{b}$  give me a vector which is perpendicular to a plane?  I know I can just check this by using dot product but I'm not totally satisfied with "it just works" answer =)  Thank you for any help! =)
43201
In linear models we need to check if a relationship exists among the explanatory variables. If they correlate too much then there is collinearity (i.e., the variables partly explain each other). I am currently just looking at the pairwise correlation between each of the explanatory variables.  Question 1: What classifies as too much correlation? For example, is a Pearson correlation of 0.5 too much?  Question 2: Can we fully determine whether there is collinearity between two variables based on the correlation coefficient or does it depend on other factors?  Question 3: Does a graphical check of the scatterplot of the two variables add anything to what the correlation coefficient indicates?
9311
I did a toy experiment with linear regression, but getting different results for $R^2$, could any one help me?  library(ggplot2) fit=lm(price~carat+depth+table+x+y+z-1,data=diamonds) summary(fit) sse=crossprod(diamonds$price-fit$fitted.values) sst=crossprod(diamonds$price-mean(diamonds$price)) 1-sse/sst
71130
According to some articles (e.g. ) correlation is just a centered version of cosine similarity. I use the following code to calculate the cosine similarity matrix of the column vectors of a matrix X (slightly modified from ):  cos.sim &lt;- function(ix)  {   A = X[,ix[1]]   B = X[,ix[2]]   return(t(A)%*%B/sqrt(sum(A^2)*sum(B^2))) }    n &lt;- ncol(X)  cmb &lt;- expand.grid(i=1:n, j=1:n)  C &lt;- matrix(apply(cmb,1,cos.sim),n,n)   My question  Which modifications of the code above are needed to get the correlation matrix cor(X) instead of the cosine similarity matrix. I guess the changes are minimal but I can't see them at the moment.
22555
You can use the decathlon dataset {FactoMineR} to reproduce this. The question is why the computed eigenvalues differ from those of the covariance matrix.  Here are the eigenvalues using princomp:  &gt; library(FactoMineR);data(decathlon) &gt; pr &lt;- princomp(decathlon[1:10], cor=F) &gt; pr$sd^2       Comp.1       Comp.2       Comp.3       Comp.4       Comp.5       Comp.6  1.348073e+02 2.293556e+01 9.747263e+00 1.117215e+00 3.477705e-01 1.326819e-01        Comp.7       Comp.8       Comp.9      Comp.10  6.208630e-02 4.938498e-02 2.504308e-02 4.908785e-03    And the same using PCA:  &gt; res&lt;-PCA(decathlon[1:10], scale.unit=FALSE, ncp=5, graph = FALSE) &gt; res$eig           eigenvalue percentage of variance cumulative percentage of variance comp 1  1.348073e+02           79.659589641                          79.65959 comp 2  2.293556e+01           13.552956464                          93.21255 comp 3  9.747263e+00            5.759799777                          98.97235 comp 4  1.117215e+00            0.660178830                          99.63252 comp 5  3.477705e-01            0.205502637                          99.83803 comp 6  1.326819e-01            0.078403653                          99.91643 comp 7  6.208630e-02            0.036687700                          99.95312 comp 8  4.938498e-02            0.029182305                          99.98230 comp 9  2.504308e-02            0.014798320                          99.99710 comp 10 4.908785e-03            0.002900673                         100.00000   Can you explain to me why the directly computed eigenvalues differ from those? (the eigenvectors are the same):  &gt; eigen(cov(decathlon[1:10]))$values  [1] 1.381775e+02 2.350895e+01 9.990945e+00 1.145146e+00 3.564647e-01  [6] 1.359989e-01 6.363846e-02 5.061961e-02 2.566916e-02 5.031505e-03   Also, the alternative prcomp method gives the same eigenvalues as the direct computation:  &gt; prc &lt;- prcomp(decathlon[1:10]) &gt; prc$sd^2  [1] 1.381775e+02 2.350895e+01 9.990945e+00 1.145146e+00 3.564647e-01  [6] 1.359989e-01 6.363846e-02 5.061961e-02 2.566916e-02 5.031505e-03   Why do PCA/princomp and prcomp give different eigenvalues?
63384
I am interested in the geometric meaning of the multiple correlation $R$ and coefficient of determination $R^2$ in the regression $y_i = \beta_1 + \beta_2 x_{2,i} + \dots + \beta_k x_{k,i} + \epsilon_i $, or in vector notation,  $$\mathbf{y} = \mathbf{X \beta} + \mathbf{\epsilon}$$  Here the design matrix $\mathbf{X}$ has $n$ rows and $k$ columns, of which the first is $\mathbf{x}_1 = \mathbf{1}_n$, a vector of 1s that corresponds to the intercept $\beta_1$.   The geometry is more interesting in the $n$-dimensional subject space rather than in the $k$-dimensional variable space. Define the hat matrix:  $$\mathbf{H} = \mathbf{X \left(X^\top X \right)}^{-1} \mathbf{X}^\top$$  This is an orthogonal projection onto the column space of $\mathbf{X}$, i.e. the   through the origin spanned by the $k$ vectors representing each variable $\mathbf{x}_i$, the first of which is $\mathbf{1}_n$. Then $\mathbf{H}$ projects the vector of observed responses $\mathbf{y}$ onto its "shadow" on the flat, the vector of fitted values $\mathbf{\hat{y}} = \mathbf{Hy}$, and if we look along the path of the projection we see the vector of residuals $\mathbf{e} = \mathbf{y} - \mathbf{\hat{y}}$ forms the third side of a triangle. This should furnish us with two routes to a geometric interpretation of $R^2$:   The square of the multiple correlation coefficient, $R$, which is defined as the correlation between $\mathbf{y}$ and $\mathbf{\hat{y}}$. This will appear geometrically as the cosine of an angle. In terms of lengths of vectors: for instance $SS_\text{residual} = \sum_{i=1}^{n}e_i^2 = \|\mathbf{e}\|^2$.   I would be delighted to see a brief account which explains:   The finer details for (1) and (2), Why (1) and (2) are equivalent, Briefly, how the geometric insight lets us visualise basic properties of $R^2$, for instance why it goes to 1 when the noise variance goes to 0. (After all, if we can't intuit from our visualisation then it's no more than a pretty picture.)   I appreciate this is more straightforward if the variables are centred first, which removes the intercept from the question. However, in most textbook accounts which introduce multiple regression, the design matrix $\mathbf{X}$ is as I laid out. Of course it's fine if an exposition delves into the space spanned by the centred variables, but for insight into the textbook linear algebra, it would be very helpful to relate this back to what's happening geometrically in the uncentred situation. A really insightful answer might explain what exactly is breaking down geometrically  - i.e. when the vector $\mathbf{1}_n$ is removed from the spanning set. I don't think this last point can be addressed by considering the centred variables alone.
33655
I'm trying to interpret variance inflation factors using the vif function in the R package car. The function prints both a generalised $\text{VIF}$ and also $\text{GVIF}^{1/(2\cdot\text{df})}$. According to the , this latter value     To adjust for the dimension of the confidence ellipsoid, the function   also prints GVIF^[1/(2*df)] where df is the degrees of freedom   associated with the term.   I don't understand the meaning of this explanation in the help file, so I'm not sure if I should be using $\text{GVIF}$ or $\text{GVIF}^{1/(2\cdot\text{df})}$. For my model these two values are very different (maximum $\text{GVIF}$ is ~$60$; maximum $\text{GVIF}^{1/(2\cdot\text{df})}$ is ~$3$).   Could someone please explain to me which one I should be using, and what is meant by adjusting the dimension of the confidence ellipsoid?
14203
Let's say I have two 1-dimensional arrays, $a_1$ and $a_2$. Each contains 100 data points. $a_1$ is the actual data, and $a_2$ is the model prediction. In this case, the $R^2$ value would be: $$ R^2 = 1 - \frac{SS_{res}}{SS_{tot}} \quad\quad\quad\quad\quad\ \ \quad\quad(1).  $$ In the meantime, this would be equal to the square value of the correlation coefficient,  $$ R^2 = (\text{Correlation Coefficient})^2 \quad (2). $$ Now if I swap the two: $a_2$ is the actual data, and $a_1$ is the model prediction. From equation $(2)$, because correlation coefficient does not care which comes first, the $R^2$ value would be the same. However, from equation $(1)$, $SS_{tot}=\sum_i(y_i - \bar y )^2$, the $R^2$ value will change, because the $SS_{tot}$ has changed if we switch $y$ from $a_1$ to $a_2$; in the meantime, $SS_{res}=\sum_i(f_i-\bar y)^2$ does not change.   My question is: How can these contradict each other?  Edit:    I was wondering that, will the relationship in Eq. (2) still stand, if it is not a simple linear regression, i.e., the relationship between IV and DV is not linear (could be exponential / log)? Will this relationship still stand, if the sum of the prediction errors does not equal zero?
25624
I have several hundred measurements. Now, I am considering utilizing some kind of software to correlate every measure with every measure. This means that there are thousands of correlations. Among these there should (statistically) be a high correlation, even if the data is completely random (each measure has only about 100 datapoints).  When I find a correlation, how do I include the information about how hard I looked for a correlation, into it?   I am not at a high level in statistics, so please bear with me.
30175
I'm not a statistic major, so my knowledge of statistics is quite limited but I've found myself in need of learning about and using mutual information. I believe I understand the concept and formula, but it seems counter-intuitive that perfectly similar data would have 0 mutual information. I would expect two sets of data with perfect similarity to have mutual information of 1.  I'm writing a program which calculates Mutual Information between two columns (i and j) in a pairwise protein sequence, and I'm not sure if I should manually fix the MI to be 1 if the columns are exactly the same. For example, if the two columns (written as lines) were:  CLLYFFDTTQGILMIGCL IILLIIIIFLIVILVIFV   Then I've calculated the MI by hand to be equal to 0.8384. However, if the two columns (written as lines) were:  GGGGGGGGGG GGGGGGGGGG   The mutual information will be equal to 0 because the formula will result in:  $$\sum \left( p(1) \times \log \left( \frac{p(1)}{p(1) p(1)} \right) \right) = 1 \times \log \left( \frac{1}{1} \right) = 0$$  In my program I have it check if the columns are identical (i = j) and just manually assign Mi = 1. So I'm wondering: Is this bad practice? Does it only consider one possible case of many? Should I not do this at all? The professor I'm working with suggested I incorporate a pseudo-count for every other non-existing amino acid and ignoring a manual fix for when i = j.  Would a pseudo-count fix this? Counting each existing amino acid as 1.1 and everything else as 0.1, or would it result in the same mutual information? Again, I'm far from a statistics major so if you can explain simply, that would be best. Thanks.