I have a dataset with approximately 4000 rows and 150 columns. I want to predict the values of a single column (= target).  The data is on cities (demography, social, economic, ... indicators). A lot of these are highly correlated, so I want to do a PCA - Principal Component Analysis.   The problem is, that ~40% of the values are missing.  My current approach is: Remove target indicator and do PCA with mean/median imputation of missing values. Select x principal components (PC). Append target indicator to these PC. Use PC as predictors for the target variable and try common regression techniques, e.g. knn, linear regression, random forest etc.  With this approach, I'm getting quite good results. My metric is RMSE% - root mean squared relative prediction error. I tried this for all columns in the dataset, the RMSE% is between 0.5% and 8% (depending on the column). These errors are for values I actually know, NOT imputed values.  So, here's my problem: I'm not sure how much my data is distorted by replacing the missing values with the column mean/median. Is there any other way of imputing the missing values with minimal effect on the PCA results?
77400
I used the prcomp() function to perform a PCA (principal component analysis) in R. However, there's a bug in that function such that the na.action parameter does not work. ; two users there offered two different ways of dealing with NA values. However, the problem with both solutions is that when there is an NA value, that row is dropped and not considered in the PCA analysis. My real data set is a matrix of 100 x 100 and I do not want to lose a whole row just because it contains a single NA value.  The following example shows that the prcomp() function does not return any principal components for row 5 as it contains a NA value.  d       &lt;- data.frame(V1 = sample(1:100, 10), V2 = sample(1:100, 10),                        V3 = sample(1:100, 10)) result  &lt;- prcomp(d, center = TRUE, scale = TRUE, na.action = na.omit) result$x                                # $ d$V1[5] &lt;- NA                           # $ result  &lt;- prcomp(~V1+V2, data=d, center = TRUE, scale = TRUE, na.action = na.omit) result$x   I was wondering if I can set the NA values to a specific numerical value when center and scale are set to TRUE so that the prcomp() function works and does not remove rows containing NA's, but also does not influence the outcome of the PCA analysis.  I thought about replacing NA values with the median value across a single column, or with a value very close to 0. However, I am not sure how that influences the PCA analysis.  Can anybody think of a good way of solving that problem?
4429
What are the main differences between performing principal component analysis (PCA) on the correlation matrix and on the covariance matrix? Do they give the same results?
57446
For a given data matrix $A$ (with variables in columns and data points in rows), it seems like $A^TA$ plays an important role in statistics. For example, it is an important part of the analytical solution of ordinary least squares. Or, for PCA, its eigenvectors are the principal components of the data.  I understand how to calculate $A^TA$, but I was wondering if there's an intuitive interpretation of what this matrix represents, which leads to its important role?
3058
I'm new to feature selection and I was wondering how you would use PCA to perform feature selection. Does PCA compute a relative score for each input variable that you can use to filter out noninformative input variables? Basically, I want to be able to order the original features in the data by variance or amount of information contained.
86126
Is there any methodical way to compare the directions, magnitudes, etc of PCA results for different samples drawn from the same population?  I'm leaving the nature of the test deliberately vague because I'd like to hear all the various possibilities... e.g. there might be (and I'm speculating here) a test comparing the sizes of the first principal components, or a test comparing the directions of the principal components, or there's some kind of distance measure between PCA outcomes and a test statistic for their equality.  As far as a use case goes, I don't have on in mind. Just out of curiosity, maybe as an exploratory technique.
67276
I have a big dataset and I want to perform a dimensionality reduction.   Now everywhere I read that I can use PCA for this. However, I still don't seem to get what to do after calculating/performing the PCA. In R this is easily done with the command princomp.  But what to do after calculating the PCA? If I decided I want to use the first $100$ principal components, how do I reduce my dataset exactly?
26471
Is it possible to use kernel principal component analysis (kPCA) for Latent Semantic Indexing (LSI) in the same way as PCA is used?  I perform LSI in R using the prcomp PCA function and extract the features with highest loadings from the first $k$ components. By that I get the features describing the component best.  I tried to use the kpca function (from the kernlib package) but cannot see how to access the weights of the features to a principal component. Is this possible overall when using kernel methods?
82353
I am trying to estimate a midplane of a 3D model using the midpoints of paired landmarks, in order to reconstruct missing data (midplane refers here to the middle/saggital plane of the cranium which cuts the skull into two symmetrical halves, left and right).  I therefore need to estimate a plane from 27 points in 3D. I need to get the equation of the plane as $$ax+by+cz=d.$$  I have looked into orthogonal regression and principal component analysis (PCA) as methods, however I didn't take maths past A-levels and am struggling. I know I can supposedly use the eigenvectors to get the equation of the plane of best fit, but need someone to explain exactly how. I am using R for the PCA but am not great at R either.  Alternatively, if there's a better way to estimate the plane I would be glad to hear it!
116069
In Element of Statistical Learning, chapter 4-2 about linear regression of an indicator matrix, it is stated that the sum of predictor is equal to 1.  To bring a bit of context: We have $N$ training samples of size $p$ represented by a $N$ x $(p+1)$ matrix (intercept included) $\textbf{X}$. Each training sample $x_i$ belong to a category $g_i \in [| 1, .., K |]$. We use the $N$ x $K$ indicator matrix for the training samples which contains only one 1 per row. We try to fit a linear regression model for the parameter $\textbf{B}$ and we get $\hat{\textbf{B}} = (\textbf{X}^T\textbf{X})^{-1}\textbf{X}^T\textbf{Y}$.  For a new data, we compute the fitted output $\hat{f}(x)^T = (1,x^T)\hat{\textbf{B}}$ and the category is then $arg \max_{k \in [| 1, .., K |]} \hat{f}_k(x)$  They say that $\sum_{k \in [| 1, .., K |]} \hat{f}_k(x) = 1$ for any $x$  as long as there is an intercept in the model.  My question is thus why? I try to use the coefficient relation from the regression derivation ($\textbf{X}^T(\textbf{Y} - \textbf{X}\hat{\textbf{B}}) = 0$) but without success.
97104
I have a dataset containing categorical features, which has 4 labels, and 4 features. (It is a meta classifier, so outputs from base classifier serve as input into this classifier) Label  Feat1 Feat2 Feat3 Feat4  1      1     1      2     2  2      3     1      2     2   3      4     3      3     1       4      4     1      2     4  I'm using scikit learn, and am considering using Naive Bayes or a Decision Tree. The classifier needs to be able to deal with missing features, and I read on scikit learn's  that Decision Tree does not support missing values. What I am looking for is advice as to how to approach missing categorical values when using scikit learn. Also, any links to academic papers addressing this would be appreciated.
10595
It is a common practice to apply PCA (principal component analysis) before a clustering algorithm (such as k-means). It is believed that it improves the clustering results in practice (noise reduction).   However I am interested in a comparative and in-depth study of the relationship between PCA and k-means. For example, Chris Ding and Xiaofeng He, 2004,  showed that "principal components are the continuous solutions to the discrete cluster membership indicators for K-means clustering". However, I have hard time understanding this paper, and Wikipedia actually .  Also, the results of the two methods are somewhat different in the sense that PCA helps to reduce the number of "features" while preserving the variance, whereas clustering reduces the number of "data-points" by summarizing several points by their expectations/means (in the case of k-means). So if the dataset consists in $N$ points with $T$ features each, PCA aims at compressing the $T$ features whereas clustering aims at compressing the $N$ data-points.  I am looking for a layman explanation of the relations between these two techniques + some more technical papers relating the two techniques.
32558
I have a dataset with 11 variables and PCA (orthogonal) was done to reduce the data. Deciding on the number of components to keep it was evident for me from my knowledge about the subject and the scree plot (see below) that two principal components (PCs) were enough to explain the data and the remaining components were only less informative.   Scree plot with parallel analysis:  observed eigenvalues (green) and simulated eigenvalues based on 100 simulations (red). Scree plot suggests 3 PCs, whereas parallel test suggests only the first two PCs.      As you can see only 48% of the variance could be captured by the first two PCs.    Plotting observations on the first plane made by the first 2 PCs revealed three different clusters using hierarchical agglomerative clustering (HAC) and K-means clustering. These  3 clusters turned out to be very relevant to the problem in question and were consistent with other findings as well. So except the fact that only 48% of variance was captured everything else was tremendously fine.      One of my two reviewers said: one cannot rely much on these findings as only 48% of variance could be explained and it is less than required.      Question Is there any required value of how much variance should be captured by PCA to be valid? Is it not dependent on the domain knowledge and methodology in use? Can anybody judge on the merit of the whole analysis just based on the mere value of the explained variance?    Notes     Data are 11 variables of genes measured by a very sensitive methodology in molecular biology called Real-Time Quantitative Polymerase Chain Reaction (RT-qPCR).   Analyses were done using R.   Answers from data analysts based on their personal experience working on real-life problems in the fields of microarray analysis, chemometrics, spectometric analyses or alike are much appreciated.   Please consider supporting you answer with references as much as possible.
60756
Im following a tutorial here:  to gain a better understanding of PCA.  The tutorial uses the Iris dataset and applies a log transform prior to PCA:     Notice that in the following code we apply a log transformation to the continuous variables as suggested by [1] and set center and scale equal to TRUE in the call to prcomp to standardize the variables prior to the application of PCA.   Could somebody explain to me in plain English why you first use the log function on the the first four columns of the Iris dataset. I understand it has something to do with making data  relative but am confused what's exactly the function of log, center and scale.  The reference [1] above is to Venables and Ripley, Modern applied statistics with S-PLUS, Section 11.1 that briefly says:     The data are physical measurements, so a sound initial strategy is to work on log scale. This has been done throughout.
7378
Is singular value decomposition almost always useful in practice for enhancing the predicative power of a trained classification model?   E.x. A dataset for classification has 20,000 features. Run SVD to convert them to top principal components and transform them to 300 features, and trained a classification model. When predict the class of a test instance, convert it to a 300d principal component feature vector, and use the trained model to predict its class.  Are there some notable real datasets of numerous features (variables) in which dimension reduction by SVD would hurt the predictive power of trained classification models?
65282
I am working with a time-series dataset that is based on demand-supply dynamics with several variables.  THe sample data for one time period is: &gt; data # A tibble: 19 x 6    Pred_error_A Pred_error_B  Pred_error_C  Pred_error_D  Pred_error_E  Pred_error_Target                  &lt;dbl&gt;         &lt;dbl&gt;          &lt;dbl&gt;      &lt;dbl&gt;      &lt;dbl&gt;           &lt;dbl&gt;  1            -0.45           1.72          -0.170      1.85       -0.01          -0.7    2            -0.45           1.72          -0.170      2.35       -0.01          -1.2    3            -0.45           1.92          -0.170      2.35       -0.01          -1      4            -0.0500         0.37          -0.12       1.65       -0.01          -1.5    5             0             -0.840         -0.12       0.950      -0.01          -1.7    6             0             -0.200         -0.12       1.35       -0.01          -1.5    7             0             -0.460         -0.12       1.35       -0.01          -1.8    8             0             -0.520         -0.12       1.05       -0.01          -1.5    9             0             -0.340         -0.12       1.05       -0.01          -1.4   10             0             -0.340         -0.12       1.35       -0.01          -1.7   11             0             -0.110         -0.12       1.05       -0.01          -1.2   12             0             -0.110         -0.12       0.850      -0.01          -1     13             0              0             -0.12       0.350      -0.01          -0.4   14             0              0             -0.12      -0.15       -0.01           0.100 15             0              0             -0.12      -0.350      -0.01           0.300 16             0              0             -0.02       0          -0.01          -0.1   17             0              0              0          0           0              0     18             0              0              0          0           0              0     19             0              0              0          0           0              0   I was working on a regression for each time period with data and wanted to find the relative importance of regressors in explaining the target variable ( Pred_error_A through E and dependent variable is Pred_error_Target ) using relaimpo package in R. However, I soon realized I can't use regression setting because of near to perfect collinearity and hence no relative importance values. Is there a way to go around this? Or, can I use any other method (instead of regression)? Any suggestions will be useful. The predictors are near to perfect and each one is important. I can't exclude any single one and I need the relative importance of all 5 regressors. I am also trying XGBoost as an alternative. However, I am not sure if it takes care of multicollinearity.
30954
Let $X_{N \times d}$ be the data matrix, where $N$ is the number of samples and $d$ the size of the features space.  Using kernel PCA (kPCA), one first computes a kernel matrix $K_{N \times N}$, and then, after its eigenvectors $E_{N \times N}$ have been computed, it is possible to project the data onto the first $c \leq N$ components as: $$X_\mathrm{projected} = KE_c,$$ where $E_c$ denotes first $c$ columns of $E$. Equivalently, in Matlab notation:  Projected_data = K*E(:,1:c);   The new projected data have now size ${N \times c} $.  I would like to know if it is possible to project an unseen data vector $x_{1 \times d} $ onto the previously computed principal components $E$. If it is possible, what is the correct procedure?
79994
I have a very large dataset and about 5% random values are missing. These variables are correlated with each other. The following example R dataset is just a toy example with dummy correlated data.  set.seed(123)  # matrix of X variable  xmat &lt;- matrix(sample(-1:1, 2000000, replace = TRUE), ncol = 10000) colnames(xmat) &lt;- paste ("M", 1:10000, sep ="") rownames(xmat) &lt;- paste("sample", 1:200, sep = "") #M variables are correlated   N &lt;- 2000000*0.05 # 5% random missing values  inds &lt;- round ( runif(N, 1, length(xmat)) ) xmat[inds] &lt;- NA  &gt; xmat[1:10,1:10]          M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 sample1  -1 -1  1 NA  0 -1  1 -1  0  -1 sample2   1  1 -1  1  0  0  1 -1 -1   1 sample3   0  0  1 -1 -1 -1  0 -1 -1  -1 sample4   1  0  0 -1 -1  1  1  0  1   1 sample5  NA  0  0 -1 -1  1  0 NA  1  NA sample6  -1  1  0  1  1  0  1  1 -1  -1 sample7  NA  0  1 -1  0  1 -1  0  1  NA sample8   1 -1 -1  1  0 -1 -1  1 -1   0 sample9   0 -1  0 -1  1 -1  1 NA  0   1 sample10  0 -1  1  0  1  0  0  1 NA   0   Is there a (best) way to impute missing values in this situation? Is the Random Forest algorithm helpful? Any working solution in R would be much appreciated.  Edits:  (1) Missing values are randomly distributed among the variables and samples.As number of variables is very large (here in the example - 10000), while the number of samples is small here in the above dummy example it is about 200. So when we look at any sample over all the variables (10000), there is high chances that there will be missing value at some variable - due to large number of variables. So just deleting the sample is not option.   (2) The variable can be treated both as quantitative or qualitative (binary) in the process of imputing. The only judgement is how well we can predict it (accuracy). So predictions like 0.98 instead of 1 might be acceptable rather 0 vs 1 or -1 vs 1. I might need to tradeoff between computing time and accuracy.   (3)  The issue I have thinking how overfitting can affect the results as the number of variables are large compared to number of samples.   (4) As the total quantity of missing values is about 5% and is random (not concentrated in any variables or samples as precaution was taken to remove the variables or samples that have very high missing values)   (5) Making data complete for analysis is first objective and accuracy is secondary. So not too sensitive to accuracy.