I am looking to clean up a large data set (~300) with a large number of features (~140).  I would like to explore outliers in the data.  My first thought was to use PCA to reduce these features to a few components that explain most of the variance, and then exclude outliers of these new n factors.  However, there are a few issues with this approach.  The first is that PCA is affected by outliers itself.  Robust PCA may be a good alternative here?  The second issue is that some of the variables I would like to partial out, like age, sex, etc..., but not remove data based on these.  I would like to control for them first, and then apply some outlier detection on the result.  Does it makes sense to partial these out first, and then apply a robust PCA outlier analysis on the results of that?
54012
Suppose I have a large set of multivariate data with at least three variables. How can I find the outliers? Pairwise scatterplots won't work as it is possible for an outlier to exist in 3 dimensions that is not an outlier in any of the 2 dimensional subspaces.  I am not thinking of a regression problem, but of true multivariate data. So answers involving robust regression or computing leverage are not helpful.  One possibility would be to compute the principal component scores and look for an outlier in the bivariate scatterplot of the first two scores. Would that be guaranteed to work? Are there better approaches?
122498
I want to do principal component analysis (factor analysis) on SPSS based on 22 variables. However, some of my variables are very skewed (skewness calculated from SPSS ranges from 2&ndash;80!).   So here are my questions:   Should I keep the skewed variables like that or could I transform the variables on principal component analysis? If yes, how would I interpret factor scores?  What type of transformation should I do? log10 or ln? Originally, my KMO (Kaiser&ndash;Meyer&ndash;Olkin) is 0.413. Much literature recommends a minimum of 0.5. Can I still do factor analysis, or do I need to remove variables to raise my KMO to 0.5?
39903
This post is the continuation of another post related to a . Basically, at this point I'm interested in a robust way to discover the periodicity/seasonality of a generic time series affected by a lot of noise. From a developer point of view, I would like a simple interface such as:  unsigned int discover_period(vector&lt;double&gt; v);  Where v is the array containing the samples, and the return value is the period of the signal. The main point is that, again, I can't make any assumption regarding the analyzed signal. I already tried an approach based on the signal autocorrelation (detecting the peaks of a correlogram), but it's not robust as I would like.
12334
I am running into a problem where an independent variable, which should have no predictive power on the dependent variable based on domain knowledge, comes out with very small p-value because the sample size is very large(~100,000). If I only use &lt; 5000 data points, then the p-value becomes large enough to support the prior that the variable is insignificant. However, I don't think tweaking the sample size to get to the desired conclusion is a good practice. Is there any procedure to adjust for small p-value simply due to huge sample size?
88274
Background: I want to classify the residential areas of a city into groups based on their social-economic characteristics, including housing unit density, population density, green space area, housing price, number of schools / health centers / day care centers, etc. I want to understand how many different groups the residential areas can be divided into, and what are their unique characteristics. This information could facilitate city planning.   Based on some examples (cf., this blog post: ), I figured out the way to do the analysis is:   First do PCA analysis. Determine the number of unique groups (clusters) based on PCA results (e.g., using the "elbow" method, or alternatively, the number of components that explains 80 to 90% of total variance).  After determining the number of clusters, apply k-means clustering to do the classification.       My questions: it seemed the number of PCA components is related to clusters analysis. So is that's true, if, say, we found 5 PCA components explained more than 90% of variation of all features, then we would apply k-means clustering and get 5 clusters. So would the 5 groups exactly corresponding to the 5 components in PCA analysis?   In another words, I guess my question is: What is the connection between the PCA analysis and k-means clustering?    Updates: Thanks to Emre, xeon, and Kirill's inputs. So the current answers:   Doing PCA before clustering analysis is also useful for dimensionality reduction as a feature extractor and visualize / reveal clusters. Doing PCA after clustering can validate the clustering algorithm (reference: ). PCA is sometimes applied to reduce the dimensionality of the dataset prior to clustering. However, Yeung &amp; Ruzzo (2000) showed that clustering with the PC's instead of the original variables does not necessarily improve cluster quality. In particular, the first few PC's (which contain most of the variation in the data) do not necessarily capture most of the cluster structure.   Yeung, Ka Yee, and Walter L. Ruzzo. An empirical study on principal component analysis for clustering gene expression data. Technical report, Department of Computer Science and Engineering, University of Washington, 2000. ()  It seemed PCA is necessary before a . Based on Ibes (2015), in which cluster analysis was run using the factors identified in the PCA.    Ibes, Dorothy C.  Landscape and Urban Planning, Volume 137, May 2015, Pages 122–137.
3058
I'm new to feature selection and I was wondering how you would use PCA to perform feature selection. Does PCA compute a relative score for each input variable that you can use to filter out noninformative input variables? Basically, I want to be able to order the original features in the data by variance or amount of information contained.
26471
Is it possible to use kernel principal component analysis (kPCA) for Latent Semantic Indexing (LSI) in the same way as PCA is used?  I perform LSI in R using the prcomp PCA function and extract the features with highest loadings from the first $k$ components. By that I get the features describing the component best.  I tried to use the kpca function (from the kernlib package) but cannot see how to access the weights of the features to a principal component. Is this possible overall when using kernel methods?
107228
I refer to   which seems to question the importance of the normal distribution of the residuals, arguing that this together with heteroskedasticity could potentially be avoided by using robust standard errors.  I have considered various transformations - roots, logs etc. - and all is proving useless in resolving completely the issue.  Here is a Q-Q plot of my residuals:    Data   Dependent variable: already with logarithmic transformation (fixes outlier issues and a problem with skewness in this data) Independent variables: age of firm, and a number of binary variables (indicators) (Later on I have some counts, for a separate regression as independent variables)   The iqr command (Hamilton) in Stata does not determine any severe outliers which rule out normality, but the graph below suggests otherwise and so does the Shapiro-Wilk test.
123279
The  for cross-validation says the following about using feature-scaling and cross-validation:      Just as it is important to test a predictor on data held-out from training, preprocessing (such as standardization, feature selection, etc.) and similar data transformations similarly should be learnt from a training set and applied to held-out data for prediction   I understand the reason behind this is to prevent information leakage between the training &amp; test sets during cross-validation, which could result in an optimistic estimate of model performance.  I am wondering then, if I wish to use Principal Component Analysis to reduce the size of a feature set before training say a regression model, and PCA requires feature-scaling to be effective, how do I chain feature-scaling to PCA to cross-validated regression, without introducing data leakage between the train-test splits in the cross-validation?
2977
In general, what is meant by saying that the fraction $x$ of the variance in an analysis like PCA is explained by the first principal component? Can someone explain this intuitively but also give a precise mathematical definition of what "variance explained" means in terms of principal component analysis (PCA)?  For simple linear regression, the r-squared of best fit line is always described as the proportion of the variance explained, but I am not sure what to make of that either. Is proportion of variance here just the extend of deviation of points from the best fit line?
10595
It is a common practice to apply PCA (principal component analysis) before a clustering algorithm (such as k-means). It is believed that it improves the clustering results in practice (noise reduction).   However I am interested in a comparative and in-depth study of the relationship between PCA and k-means. For example, Chris Ding and Xiaofeng He, 2004,  showed that "principal components are the continuous solutions to the discrete cluster membership indicators for K-means clustering". However, I have hard time understanding this paper, and Wikipedia actually .  Also, the results of the two methods are somewhat different in the sense that PCA helps to reduce the number of "features" while preserving the variance, whereas clustering reduces the number of "data-points" by summarizing several points by their expectations/means (in the case of k-means). So if the dataset consists in $N$ points with $T$ features each, PCA aims at compressing the $T$ features whereas clustering aims at compressing the $N$ data-points.  I am looking for a layman explanation of the relations between these two techniques + some more technical papers relating the two techniques.
103242
I am looking for a robust estimation of the mean that has a specific property. I have a set of elements for which I want to calculate this statistic. Then, I add new elements one at a time, and for each additional element I would like to recalculate the statistic (also known as an online algorithm). I would like this update calculation to be fast, preferably O(1), i.e. not dependent on the size of the list.  The usual mean has this property that it can be updated efficiently, but is not robust to outliers. Typical robust estimators of the mean, like inter-quartile mean and trimmed mean cannot be updated efficiently (since they require maintaining a sorted list).  I would appreciate any suggestions for robust statistics that can be calculated/updated efficiently.
86995
In my machine learning class these two methods were discussed and mentioned that both should be used. I have a couple questions about this:  1) Can I mix and match these two approaches? e.g. Feature Scale x1 and Mean Normalize x2?  2) How do you determine which of these options to apply? It seems that either could accomplish the task of increasing your convergence rates... I suppose you just need to know your data set to understand which will reliably reduce your values while leaving as few outliers as possible?
66825
I have conducted PCA that has reduced the dimensions of my data from more than 20 to 7 (7 PCAs explain about 85% of the total variation). As a second step, I have to cluster my data based on these new 7 PCAs. My question is: how I should reconstruct/transform my data (cbind/rbind)? As I understand, each PCA is a weighted mix of the original variables. So should I just replace the old variables with the PCAs? Also, how should I interpret the final results? If there was no PCA, each cluster would incorporate all variables in some proportion. But after PCA, how would I describe each cluster? For instance, cluster 1 contains 40% of PCA 1 and PCA 1, in turn, has xxx loading scores? Something like this? Would greatly appreciate any help on interpreting this.
52616
I have a dataset that contains data from $307$ subjects and nine variables for each subject. I would like to run a PCA. My problem is that I get a Kaiser-Meyer-Olkin (KMO) value of $0.06$.  Can it be due to the fact that within every subject the variables add up to $1$? Is there another way to test whether it is okay to run PCA on my data?
18745
What is the motivation for demeaning the data when doing PCA. I've been told to do it, but I've never heard a good and/or intuitive reason for it. Is this a case where doing it just makes the math easier, or is it more like: if you don't do it, you'll get the wrong answer.  This is something I've been wondering about for a while.  One other question: when you demean the data, do you also need to normalize it (i.e., divide by standard deviation)?
1306
In today's pattern recognition class my professor talked about PCA, eigenvectors and eigenvalues.   I understood the mathematics of it. If I'm asked to find eigenvalues etc. I'll do it correctly like a machine. But I didn't understand it. I didn't get the purpose of it. I didn't get the feel of it.        I strongly believe in the following quote:     You do not really understand something unless you can explain it to your grandmother. -- Albert Einstein   Well, I can't explain these concepts to a layman or grandma.   Why PCA, eigenvectors &amp; eigenvalues? What was the need for these concepts? How would you explain these to a layman?