I'm working on an implementation of PCA that works on very large data sets.  Based on my understanding of the algorithm, the first step is to do an  of the input m x n matrix, X. This SVD looks like X = WΣVT. The "interesting" output Y of this process -- , "The PCA transformation that preserves dimensionality (that is, gives the same number of principal components as original variables)" -- is given by the following equation:    Based on my reading, if I can compute the W component of the SVD, then I can compute Y as:    The upshot here is that I'd only have to compute W. In terms of computational and memory complexity, this approach is significantly more efficient because the only matrix above and beyond the initial data set I'd have to load has size m x m, which (at least in my case) is much smaller than V, which would be n x n.  Is there some reason why this derivation won't work that I'm not seeing?
15315
Principal component analysis (PCA) is usually explained via an eigen-decomposition of the covariance matrix. However, it can also be performed via singular value decomposition (SVD) of the data matrix $\mathbf X$. How does it work? What is the connection between these two approaches? What is the relationship between SVD and PCA?  Or in other words, how to use SVD of the data matrix to perform dimensionality reduction?
79831
Let $T_1,T_2$ be two linear transformations from $\mathbb{R}^n$ to $\mathbb{R}^m$. Let $\{x_1,\cdots , x_n\}$ be a basis of $\mathbb{R}^n$. Suppose that $T_1x_i \neq 0$ for every $i=1,2,\cdots, n$ and that $x_i \bot \ker T_2$ for every $i=1,2,\cdots, n$. Which of the following is/are necessarily true?   $T_1$ is invertible. $T_2$ is invertible.  both $T_1,T_2$ are invertible.  neither $T_1$ nor $T_2$ is invertible.   My Approch:  Given dimension of $\mathbb{R}=n$ and  $x_i\perp Ker(T_2)$  So, $\text{Ker}T_2=0$  Row space and null space are perpendicular to each other.  From Rank Nullity Theorem:  let T : V → W be a linear map. Then the rank of T is the dimension of the image of T and the nullity of T is the dimension of the kernel of T, so we have  $\text{dim}(\text{Im}(T))+ \text{dim}(\text{Ker}(T))=\text{dim}(V)$  Applying above result we get,  $\text{dim}(\text{Im}(T))+ 0=n$  Therefore, $T_2$ is one-one.  I could not solve this problem completely. what to do next?
3893
Principal component analysis (PCA) can be used for dimensionality reduction. After such dimensionality reduction is performed, how can one approximately reconstruct the original variables/features from a small number of principal components?  Alternatively, how can one remove or discard several principal components from the data?  In other words, how to reverse PCA?    Given that PCA is closely related to singular value decomposition (SVD), the same question can be asked as follows: how to reverse SVD?
22157
I have a $20\times100$ matrix $X$, containing my $N=20$ samples in the $D=100$-dimensional space. I now wish to code up my own principal component analysis (PCA) in Matlab. I demean $X$ to $X_0$ first.  I read from someone's code that in such scenarios where we have more dimensions than observations, we no longer eigen-decompose the $X_0$'s covariance matrix. Instead, we eigen-decompose $\frac{1}{N-1}X_0X_0^T$. Why is it correct?  The normal covariance matrix is of size $D\times D$, each element of which tells us the covariance between two dimensions. To me, $\frac{1}{N-1}X_0X_0^T$ is not even of the correct dimensions! It is $N\times N$ matrix, so what would it tell us? Covariance between two observations?!
5242
I am trying to perform document-level clustering. I constructed the term-document frequency matrix and I am trying to cluster these high dimensional vectors using k-means. Instead of directly clustering, what I did was to first apply LSA's (Latent Semantic Analysis) singular vector decomposition to obtain the U,S,Vt matrices, selected a suitable threshold using the scree plot and applied clustering on the reduced matrices (specifically Vt because it gives me a concept-document information) which seemed to be giving me good results.   I've heard some people say SVD (singular vector decomposition) is clustering (by using cosine similarity measure etc.) and was not sure if I could apply k-means on the output of SVD. I thought it was logically correct because SVD is a dimensionality reduction technique, gives me a bunch of new vectors. k-means, on the other hand, will take the number of clusters as the input and divide these vectors into the specified number of clusters. Is this procedure flawed or are there ways in which this can be improved? Any suggestions?
55263
I've got an application where it'd be handy to cluster a noisy dataset before looking for subgroup effects within the clusters.  I first looked at PCA, but it takes ~30 components to get to 90% of the variability, so clustering on just a couple of PC's will throw away a lot of information.    I then tried t-SNE (for the first time), which gives me an odd shape in two dimensions that is very amenable to clustering via k-means.  What's more, running a random forest on the data with the cluster assignment as the outcome shows that the clusters have a fairly sensible interpretation given the context of the problem, in terms of the variables that make up the raw data.  But if I'm going to report on these clusters, how do I describe them?  K-means clusters on principal components reveal individuals who are nearby to one another in terms of the derived variables that comprise X% of the variance in the dataset.  What equivalent statement can be made about t-SNE clusters?  Perhaps something to the effect of:     t-SNE reveals approximate contiguity in an underlying high-dimensional manifold, so clusters on the low-dimensional representation of the high-dimensional space maximize the "likelihood" that contiguous individuals will not be in the same cluster   Can anyone propose a better blurb than that?
69690
I have collected some data and stored them in an $N\times P$ matrix $A$. Using SVD, we can rotate $A=UDV^T$ into a new basis, also discarding some dimensions: $A\approx U\tilde{D}V^T$, where $\tilde{D}$ is a diagonal matrix of the first $k&lt;P$ singular values, with the remaining diagonal values $0$.  Then I train some model using $U\tilde{D}$ as the source data. Now I have a new set of data $B$ that I want to test the model on. The first step in applying the model is rotating $B$ to correspond to the data $U\tilde{D}$ that I trained the model on. How can I do this?
48051
It's not immediately clear to me why this is true. My notes say that putting $n$ orthonormal vectors $ v_1, ..., v_n$ in the columns of $X$ gives $X^TX = I$, and it follows from this that the rows of $X$ are orthonormal.   Can you explain this better to me? I tried the $2\times 2$ case, but it isn't clear to me why this is true.
26471
Is it possible to use kernel principal component analysis (kPCA) for Latent Semantic Indexing (LSI) in the same way as PCA is used?  I perform LSI in R using the prcomp PCA function and extract the features with highest loadings from the first $k$ components. By that I get the features describing the component best.  I tried to use the kpca function (from the kernlib package) but cannot see how to access the weights of the features to a principal component. Is this possible overall when using kernel methods?
32558
I have a dataset with 11 variables and PCA (orthogonal) was done to reduce the data. Deciding on the number of components to keep it was evident for me from my knowledge about the subject and the scree plot (see below) that two principal components (PCs) were enough to explain the data and the remaining components were only less informative.   Scree plot with parallel analysis:  observed eigenvalues (green) and simulated eigenvalues based on 100 simulations (red). Scree plot suggests 3 PCs, whereas parallel test suggests only the first two PCs.      As you can see only 48% of the variance could be captured by the first two PCs.    Plotting observations on the first plane made by the first 2 PCs revealed three different clusters using hierarchical agglomerative clustering (HAC) and K-means clustering. These  3 clusters turned out to be very relevant to the problem in question and were consistent with other findings as well. So except the fact that only 48% of variance was captured everything else was tremendously fine.      One of my two reviewers said: one cannot rely much on these findings as only 48% of variance could be explained and it is less than required.      Question Is there any required value of how much variance should be captured by PCA to be valid? Is it not dependent on the domain knowledge and methodology in use? Can anybody judge on the merit of the whole analysis just based on the mere value of the explained variance?    Notes     Data are 11 variables of genes measured by a very sensitive methodology in molecular biology called Real-Time Quantitative Polymerase Chain Reaction (RT-qPCR).   Analyses were done using R.   Answers from data analysts based on their personal experience working on real-life problems in the fields of microarray analysis, chemometrics, spectometric analyses or alike are much appreciated.   Please consider supporting you answer with references as much as possible.
2977
In general, what is meant by saying that the fraction $x$ of the variance in an analysis like PCA is explained by the first principal component? Can someone explain this intuitively but also give a precise mathematical definition of what "variance explained" means in terms of principal component analysis (PCA)?  For simple linear regression, the r-squared of best fit line is always described as the proportion of the variance explained, but I am not sure what to make of that either. Is proportion of variance here just the extend of deviation of points from the best fit line?
109168
I have a set of data that has $n$ samples described by $m$ variables. I do a PCA to reduce it to just 2 dimensions so I can make a nice 2D plot of the data. I understand that the $x,y$ coordinates (i.e., the PCA scores) for the plot are calculated by basically summing the products of the original data (after centering) by the loadings for each variable, so:  $$\mathrm{PC}_1 = X_1L_1 + X_2L_2 + ... + X_mL_m.$$  My question is, if I pick an arbitrary point in the PCA space (i.e. a value for $\mathrm{PC}_1$ and $\mathrm{PC}_2$, or $x$ and $y$ in my plot), is there a convenient way to translate that back to a set of the original values (i.e., $X_1,X_2,\dots,X_m$)?    Note 100% reversal is obviously not expected (since I'm only using 2 PCs), so a decent approximation is fine.
60419
My original data has many more columns (features) than rows (users). I am trying to reduce the features of my SVD (I need all of the rows). I found one method of doing so in a book called "Machine Learning in Action" but I don't think it will work for the data I am using.  The method is as follows. Define SVD as $$A = USV^\top.$$  Set an optimization threshold (i.e., 90%). Calculate the total sum of the squares of the diagonal $S$ matrix. Calculate how many $S$ values it takes to reach 90% of the total sum of squares. So if that turns out to be 100 $S$ values, then I would take the first 100 columns of the $U$ matrix, first 100 rows of the $V^\top$ matrix, and a $100\times 100$ square matrix out of the $S$ matrix. I would then calculate $A = USV^\top$ using the reduced matrices.  However, this method does not target the columns of my original data, since the dimensions of the resulting $A$ matrix are the same as before. How would I target the columns of my original matrix?
121301
I have a couple of quick questions about PCA:    Does the PCA assume that the dataset is Gaussian?   What happens when I apply a PCA to inherently non-linear data?   Given a dataset, the process is to first mean-normalize, set the variance to 1, take an SVD, reduce rank, and finally map the dataset into the new reduced-rank space. In the new space, each dimension corresponds to a "direction" of maximal variance.     But is the correlation of that dataset in the new space always zero, or is that only true for data that is inherently Gaussian?   Suppose I have two datasets, "A" and "B", where "A" corresponds to randomly sampled points taken from a Gaussian, while "B" corresponds to points randomly sampled from another distribution (say Poisson).    How does the PCA(A) compare to the PCA(B)?   By looking at the points in the new space, how would I determine that the PCA(A) corresponds to points sampled from a Gaussian, while PCA(B) corresponds to points sampled from a Poisson?   Is the correlation of the points in "A" 0?   Is the correlation of points in "B" also 0?   More importantly, am I asking the "right" question?   Should I look at the correlation, or is there another metric that I should consider?
79192
Let's say I have a $p$-dimensional multivariate Gaussian distribution. And I take $n$ observations (each of them a $p$-vector) from this distribution and calculate the sample covariance matrix $S$. In this , the authors state that the sample covariance matrix calculated with $p &gt; n$ is singular.    How is it true or derived? Any explanations?
43154
Consider a set $X=(X_1; \dots; X_n)$ of $n$ data points such that $X_i \in \mathbb{R}^d$ is a column vector. Let $Y = \text{pca_proj}(X)$ denote the projection of points in $X$ according to the PCA components i.e.  $$Y_i \in \mathbb{R}^d\\ Y_i = W X_i$$  where $W \in \mathbb{R}^{d \times d}$ is the projection matrix obtained from PCA, and each row of $W$ is an eigen vector of $X^T X$.   Now let's define $Z = \text{pca_proj}(Y)$. My questions is: does $Y = Z$? If not, what happens if we keep applying PCA recursively on a matrix? Does it converge to a specific matrix?
39365
I have a training dataset of images: X (Visual) and Y (Infrared). Each set has $300$ training examples. I extract feature vectors from both sets of images. Thus my X and Y datasets are respectively $300\times 1920$ and $300\times 1536$ where $300$ is the number of sample observations (in  this case images) and $1920$ &amp; $1536$ are respectively the length of the feature vectors in visual and infrared spectrum.  A testing dataset consists of images of different subjects in both visual spectrum and infrared spectrum. I will have the visual spectrum data as my gallery. I will have the infrared images as the probe data. For each probe image, I need to retrieve the corresponding visual image from the gallery by using some kind of similarity measure.  Basic algorithm idea:   Start Training Phase. Read the images from visual and infrared spectrum dataset. Get the feature vectors by using desired descriptors and populate the matrices X and Y. Use CCA for subspace learning. Get the projection matrices Wx and Wy. Start Testing Phase. Given the gallery images (visual), read these images and use the Wx transformation matrix to convert them into the CCA subspace. For each probe image, convert it into the CCA subspace by using the Wy transformation and compare with each images of the gallery and compute a matching score. The image (or its label) in the gallery having the maximum score is returned.   Could anyone tell me if my approach is correct or not? Pointing me in the right direction would also be helpful. Please see the following paper for reference: .    I work in Matlab and use the following command to perform CCA:  [Wx,Wx,r,U,V] = canoncorr(X,Y);   %// DO CCA   The output I get is this :    Name           Size             Bytes   Class     Attributes    Wx         1920x297            890880  double                 Wx         1536x297            712704  double                 U           300x297             27840  double                 V           300x297             27840  double                 r             1x297               464  double          :   The projection matrices are Wx and Wy since they transform X and Y into the new space. The resulting projections of X and Y into the new space are U and V, respectively. The r vector represents the entries of the correlation matrix between U and V, which is a diagonal matrix.