i) What is the main role of "only" trying to find orthogonal components in PCA?   I can understand, that we would not want a zero-solution as well as find directions that are orthogonal in order to explain most of the variance. When we look at the problem in terms of finding a projection matrix that preserves the gram matrix of mean centered data,   ii) What would an optimization under a non orthonormal constraint produce, as long as we make sure the solution is non-zero? Why would it be or not be useful?   I am aware of the concept of non-orthogonal eigen-functions as well, just in case you want to generalize the answer to a Kernel PCA, that would be fine as well.
69305
I am trying to understand some descriptions of PCA (the first two are from Wikipedia), emphasis added:     Principal components are guaranteed to be independent only if the data set is jointly normally distributed.   Is the independence of principal components very important? How can I understand this description?     PCA is sensitive to the relative scaling of the original variables.     What does 'scaling' mean there? Normalization of different dimensions?     The transformation is defined in such a way that the first principal component has the largest possible variance and each succeeding component in turn has the highest variance under the constraint that it be orthogonal to the preceding components.   Can you explain this constraint?
121647
Is there a name for a matrix whose rows (or columns) are non-zero orthogonal vectors ?  It seems to me that "orthogonal matrix" would be a good name, but this is already taken -- it refers to a matrix whose rows (or columns) form an orthonormal set of vectors.
76066
As far as I understand,  principal components are obtained by rotating the coordinate axes to align them with the directions of maximum variance.   Nevertheless, I keep reading about "unrotated principal components" and my statistics software (SAS) gives me varimax-rotated principal components as well as the unrotated ones. Here I am confused: when we compute principal components, the axes are already rotated; so why is there another rotation needed? And what does "unrotated principal component"  mean?
9746
I am trying to understand the differences between the linear dimensionality reduction methods (e.g., PCA) and the nonlinear ones (e.g., Isomap).  I cannot quite understand what the (non)linearity implies in this context. I read from  that     By comparison, if PCA (a linear dimensionality reduction algorithm) is   used to reduce this same dataset into two dimensions, the resulting   values are not so well organized. This demonstrates that the   high-dimensional vectors (each representing a letter 'A') that sample   this manifold vary in a non-linear manner.   What does     the high-dimensional vectors (each representing a letter 'A') that   sample this manifold vary in a non-linear manner.   mean? Or more broadly, how do I understand the (non)linearity in this context?
97919
I have carried out a principal components analysis of six variables $A$, $B$, $C$, $D$, $E$ and $F$. If I understand correctly, unrotated PC1 tells me what linear combination of these variables describes/explains the most variance in the data and PC2 tells me what linear combination of these variables describes the next most variance in the data and so on.   I'm just curious -- is there any way of doing this "backwards"? Let's say I choose some linear combination of these variables -- e.g. $A+2B+5C$, could I work out how much variance in the data this describes?
102747
While trying Gaussian Mixture Models , I found these 4 types of covariances.   'full' (each component has its own general covariance matrix), 'tied' (all components share the same general covariance matrix), 'diag' (each component has its own diagonal covariance matrix), 'spherical' (each component has its own single variance).   I googled a lot to find more details about each of these types but found very high-level descriptions (such as ) only.   Appreciate if someone can help me understand these, or at least direct me to somewhere I can read about these.
3893
Principal component analysis (PCA) can be used for dimensionality reduction. After such dimensionality reduction is performed, how can one approximately reconstruct the original variables/features from a small number of principal components?  Alternatively, how can one remove or discard several principal components from the data?  In other words, how to reverse PCA?    Given that PCA is closely related to singular value decomposition (SVD), the same question can be asked as follows: how to reverse SVD?
61886
I know the definition of symmetric positive definite (SPD) matrix, but want to understand more.   Why are they so important, intuitively?  Here is what I know. What else?   For a given data, Co-variance matrix is SPD. Co-variance matrix is a important metric, see this  for intuitive explanation. The quadratic form $\frac 1 2 x^\top Ax-b^\top x +c$ is convex, if $A$ is SPD. Convexity is a nice property for a function that can make sure the local solution is global solution. For Convex problems, there are many good algorithms to solve, but not for non-covex problems. When $A$ is SPD, the optimization solution for the quadratic form $$\text{minimize}~~~ \frac 1 2 x^\top Ax-b^\top x +c$$ and the solution for linear system $$Ax=b$$ are the same. So we can run conversions between two classical problems. This is important because it enables us to use tricks discovered in one domain in the another. For example, we can use the conjugate gradient method to solve a linear system.  There are many good algorithms (fast, numerical stable) that work better for an SPD matrix, such as Cholesky decomposition.   EDIT: I am not trying ask the identities for SPD matrix, but the intuition behind the property to show the importance. For example, as mentioned by @Matthew Drury, if a matrix is SPD, Eigenvalues are all positive real numbers, but why all positive matters. @Matthew Drury had a great answer to flow and that is what I was looking for.
102210
When computing the covariance matrix of a sample, is one then guaranteed to get a symmetric and positive-definite matrix?  Currently my problem has a sample of 4600 observation vectors and 24 dimensions.
14594
Say that the raw data is $N$-dimensional, where $N$ is a large positive integer. If we apply PCA to the dataset, and compute the reconstruction error (in $\ell^2$-norm) using the first $d \leq N$ components, will the reconstruction error monotonically decrease to $0$ as $d \rightarrow N$?
6548
Can someone point me to a paper, or show here, why symmetric matrices have orthogonal eigenvectors? In particular, I'd like to see proof that for a symmetric matrix $A$ there exists decomposition $A = Q\Lambda Q^{-1} = Q\Lambda Q^{T}$ where $\Lambda$ is diagonal.
41587
Can anyone explain me please how to see this statement: the only orthogonal projection that is also unitary from $\Bbb C^n$ to $\Bbb C^n$ is the Identity.  how can I prove formally that? or how can I see that only the Identity satisfy it?
32859
Suppose I am running a regression $Y \sim X$. Why by selecting top $k$ principle components of $X$, does the model retain its predictive power on $Y$?   I understand that from dimensionality-reduction/feature-selection point of view, if $v_1, v_2, ... v_k$ are the eigenvectors of covariance matrix of $X$ with top $k$ eigenvalues, then $Xv_1, Xv_2 ... Xv_k$ are top $k$ principal components with maximum variances. We can thereby reduce the number of features to $k$ and retain most of the predictive power, as I understand it.   But why do top $k$ components retain the predictive power on $Y$?   If we talk about a general OLS $Y \sim Z$, there is no reason to suggest that if feature $Z_i$ has maximum variance, then $Z_i$ has the most predictive power on $Y$.   Update after seeing comments: I guess I have seen tons of examples of using PCA for dimensionality reduction. I have been assuming that means the dimensions we are left with have the most predictive power. Otherwise what's the point of dimensionality reduction?
115144
I have a unitary matrix that I want to construct. I only care what happens to the first computational state, so the first column is specified. So far, I've been assigning each question mark to a variable and solving $UU^T = I$ analytically. But this 6x6 case is out of computational reach for this method.  Is there any general method, or any clever trick, to help me fill in the rest of matrices such as $U$?  Note: I actually would prefer if all the entries were real, so technically these are better called orthogonal matrices.  $U = \frac{1}{\sqrt{5}}\begin{bmatrix} 0 &amp; ? &amp; ? &amp; ? &amp; ? &amp; ? \\ 1 &amp; ? &amp; ? &amp; ? &amp; ? &amp; ? \\ 1 &amp; ? &amp; ? &amp; ? &amp; ? &amp; ? \\ 1 &amp; ? &amp; ? &amp; ? &amp; ? &amp; ? \\ 1 &amp; ? &amp; ? &amp; ? &amp; ? &amp; ? \\ 1 &amp; ? &amp; ? &amp; ? &amp; ? &amp; ? \\ \end{bmatrix} $
60074
Suppose $u$ is the vector that maximizes the variance of the projection of the data with design matrix $X$.   Now, I have seen materials that refer $u$ as the (first) principal component of the data, which is also the eigenvector with largest eigenvalue.   However, I have also seen that the principal component of the data is $X u$.   Obviously, $u$ and and $Xu$ are different things. Can anyone help me out here and tell me what is the difference between these two definitions of principal components?
3300
I know that in a regression situation, if you have a set of highly correlated variables this is usually "bad" because of the instability in the estimated coefficients (variance goes toward infinity as determinant goes towards zero).  My question is whether this "badness" persists in a PCA situation.  Do the coefficients/loadings/weights/eigenvectors for any particular PC become unstable/arbitrary/non-unique as the covariance matrix becomes singular?  I am particularly interested in the case where only the first principal component is retained, and all others are dismissed as "noise" or "something else" or "unimportant".  I don't think that it does, because you will just be left with a few principal components which have zero, or close to zero variance.  Easy to see this isn't the case in the simple extreme case with 2 variables - suppose they are perfectly correlated.  Then the first PC will be the exact linear relationship, and the second PC will be perpindicular to the first PC, with all PC values equal to zero for all observations (i.e. zero variance).  Wondering if its more general.
12577
In class we've seen that $a$ (the weights) must satisfy $$X^T (y-Xa) =0$$  Here $X$ is a $(n\times d)$ matrix (so we have $n$ samples in $\mathbb R^d$)  let's denote the residuals $r = y-Xa$. In our class notes, it is written that     The normal equations require the residuals to be orthogonal to each of the columns of $X$.   Why?     Therefore, the solution of the linear regression is a projection of $y$ onto the subspace spanned by $v_1 , \ldots , v_d$ (the columns of $X$)   Can you please explain this?