I have a data set and I performed the following operation: from sklearn.decomposition import PCA from sklearn.preprocessing import StandardScaler scaler = StandardScaler() X_scaled = scaler.fit_transform(X_train)  pca_test = PCA(n_components=10) pca_test.fit(X_scaled) #------------Using scaled data cvr = np.cumsum(pca_test.explained_variance_ratio_)  pca_test = PCA(n_components=10) pca_test.fit(X_train) #------------Using non scaled data cvr2 = np.cumsum(pca_test.explained_variance_ratio_)  print(np.round(cvr, 2)) [0.31 0.52 0.69 0.81 0.9  0.96 0.98 0.99 1.   1.  ]  print(np.round(cvr2, 2)) [0.97 0.99 0.99 1.   1.   1.   1.   1.   1.   1.  ]  What does the large explained variance in the two method tell me about data or what I am doing with data rather? Essentially when I scale the data there is no variance reduction. So, does it mean that in my case it is not the best idea to apply standardization ?
4429
What are the main differences between performing principal component analysis (PCA) on the correlation matrix and on the covariance matrix? Do they give the same results?
69363
Provided a sample size "N" that I plan on using to forecast data. What are some of the ways to subdivide the data so that I use some of it to establish a model, and the remainder data to validate the model?  I know there is no black and white answer to this, but it would be interesting to know  some "rules of thumb" or usually used ratios. I know back at university, one of our professors used to say model on 60% and validate on 40%.
8209
This question just occurred to me out of the blue. PCA is a way to reduce dimensions. Another way that is often (perhaps too often) used is to take mean values of two or more variables. This is done a great deal to create "scales" or "ratings" that get published in the popular press about various political issues. Dimensions are naively "averaged" and this cumulative "average" is portrayed as a magical one-number-sums-all figure.  Is this, nevertheless, a very crude relative of PCA? If so, what assumptions does it have? I can think of the following:   Sphericity is perfect or near-perfect. All variables are perfectly orthogonal from all other variables. Variables are all perfectly scaled against each other.   What else would there be?
18662
I'm a newbie to data science, and I do not understand the difference between the fit and fit_transform methods in scikit-learn. Can anybody explain simply why we might need to transform data? What does it mean, fitting a model on training data and transforming to test data? Does it mean, for example, converting categorical variables into numbers in training and transforming the new feature set onto test data?
15315
Principal component analysis (PCA) is usually explained via an eigen-decomposition of the covariance matrix. However, it can also be performed via singular value decomposition (SVD) of the data matrix $\mathbf X$. How does it work? What is the connection between these two approaches? What is the relationship between SVD and PCA?  Or in other words, how to use SVD of the data matrix to perform dimensionality reduction?
44347
I am trying to understand the . When I try to apply it to the following two samples of data I am unable to understand how to interpret the results.  Let's say sample 1 is ${0, 5, 7, 12, 11, 17}$ and sample 2 is ${10 ,15 ,17 ,22 ,21 ,27}$. Here sample 2 $=$ sample 1 $+\ 10$ as you can see.  Both have the same standard deviation $\sigma_{2} = \sigma_{1}= 5.95539$ but $\mu_{2}=18.67$ and $\mu_{1}=8.66667$.  Now the coefficient of variation ${\sigma}/{\mu}$ will be different. For sample 2 it will be less than for sample 1. But how do I interpret that result? In terms of variance both are the same; only their means are different. So what's the use of the coefficient of variation here? It's just misleading me, or maybe I am unable to interpret the results.
101933
I've looked through a lot of literature available online, including this forum without any luck and hoping someone can help a statistical issue I currently face:  I have 5 lists of of ranked data, each containing 10 items ranked from position 1 (best) to position 10 (worst). For sake of context, the 10 items in each lists are the same, but in different ranked orders as the technique used to decide their rank is different.  Example data:              List 1      List 2      List 3     ... etc Item 1     Ranked 1    Ranked 2    Ranked 1      Item 2     Ranked 3    Ranked 1    Ranked 2 Item 3     Ranked 2    Ranked 3    Ranked 3 ... etc   I am looking for a way to interpret and analyse the above data so that I get a final result showing the overall rank of each item based on each test and its position, e.g.  Result Rank 1 = Item 1 Rank 2 = Item 3 Rank 3 = Item 4 ... etc   So far I have attempted to interpret this information from performing Pearson's Correlation, Spearman's Correlation, Kendall Tau's B, and Friedman tests. I have found however, that these results have generally paired my lists (i.e. compared list 1 to list 2, then list 1 to list 3 .. etc), or have produced results such as Chi-Square, P-Values etc about the overall data.   Does anyone know how I can interpret this data in a statistically sound method (at a post graduate / PhD applicable level) so that I can understand the overall ranks signalling the importance of each item in the list across the 5 tests please? Or, if there is another type of technique or statistical test I can look into I would appreciate any hints or guidance.   (It maybe also worth noting, I have also performed the simpler mathematical techniques such as sums, averaging, minimum - maximum tests etc, but do not feel these are statistically important enough at this level).  Any help or advice would be greatly appreciated, thank you for your time.
24206
My understanding is that when some features have different ranges in their values (for example, imagine one feature being the age of a person and another one being their salary in USD) will affect negatively algorithms because the feature with bigger values will take more influence, is it a good practice to simply ALWAYS scale/normalize the data?  It looks to me that if the values are already similar among then, then normalizing them will have little effect, but if the values are very different normalization will help, however it feels too simple to be true :)  Am I missing something? Are there situations/algorithms were actually it is desirable to let some features to deliberately outweigh others?
36884
Hey guys I found one or two papers which use ridge regression (for basketball data). I was always told to standardize my variables if I ran a ridge regression, but I was simply told to do this because ridge was scale variant (ridge regression wasn't really part of our course, so our lecturer skimmed through it).  These papers I read didn't standardize their variables, which I found a bit surprising. They also ended up with large values of lambda (around the 2000-4000 level) via cross-validation, and I was told that this is due to not standardizing the variables.  How exactly does leaving the variable(s) unstandardised lead to high lambda values and also, what are the consequences of not standardizing the variables in general? Is it really such a big deal?  Any help is much appreciated.
115472
I'm running PCA on my dataset which can be found . There are 6497 instances and 12 attributes with 13th column is the class (ranging from 3 - 9) for wine quality.  I've read what PCA is supposed to do. It combines attributes and shows variance in the data. Finally leading to be able to select few dimensions from a high dimension data that have the most variance.   I'm having trouble finding which attributes to pick after running PCA. How can I find out which attributes can I use and still not lose meaning of my data.   I've created the following biplot from the ggbiplot package    How can I interpret from this biplot which attributes to pick?  I've also created a screeplot for variance.     Looking at the variance screeplot I would say that 3 components would represent the data well. Is this a reasonable assessment to make?  Question   How can I decide which attributes to pick after running PCA While doing PCA should I leave out the class variable or keep it?   Code to generate the plots  wine &lt;- read.csv("wine_nocolor.csv") wine1 &lt;- wine[2:13] wine1.pca &lt;- prcomp(wine1, scale. = TRUE) library(devtools) install_github("ggbiplot", "vqv") ggbiplot(wine1.pca, obs.scale = 1, var.scale = 10, groups = wine$q, ellipse = TRUE, circle = TRUE, title="Biplot for Wine Quality (PC1, PC2)", var.axes=TRUE, varname.size=8)
121301
I have a couple of quick questions about PCA:    Does the PCA assume that the dataset is Gaussian?   What happens when I apply a PCA to inherently non-linear data?   Given a dataset, the process is to first mean-normalize, set the variance to 1, take an SVD, reduce rank, and finally map the dataset into the new reduced-rank space. In the new space, each dimension corresponds to a "direction" of maximal variance.     But is the correlation of that dataset in the new space always zero, or is that only true for data that is inherently Gaussian?   Suppose I have two datasets, "A" and "B", where "A" corresponds to randomly sampled points taken from a Gaussian, while "B" corresponds to points randomly sampled from another distribution (say Poisson).    How does the PCA(A) compare to the PCA(B)?   By looking at the points in the new space, how would I determine that the PCA(A) corresponds to points sampled from a Gaussian, while PCA(B) corresponds to points sampled from a Poisson?   Is the correlation of the points in "A" 0?   Is the correlation of points in "B" also 0?   More importantly, am I asking the "right" question?   Should I look at the correlation, or is there another metric that I should consider?
72308
I've been reading over this , trying to make sense of how the mean and variance of a gaussian conditional was derived. I've come to accept that unless I allocate a dozen or so hours to refreshing my linear algebra knowledge, it's out of my reach for the time being. that being said, I'm looking for a conceptual explanation for that these equations represent: $$\mu_{1|2} = \mu_1 + \Sigma_{1,2} * \Sigma^{-1}_{2,2}(x_2 - \mu_2)$$ I read the first as &quot;Take $\mu1$ and augment it by some factor, which is the covariance scaled by the precision (measure of how closely $X_2$ is clustered about $\mu_2$, maybe?) and projected onto the distance of the specific $x_2$ from $mu_2$.&quot; $$\Sigma_{1|2} = \Sigma_{1,1} - \Sigma_{1,2} * \Sigma^{-1}_{2,2} * \Sigma_{1,2}$$ I read the second as, &quot;take the variance about $\mu_1$ and subtract some factor, which is covariance squared scaled by the precision about $x_2$.&quot; In either case, the precision $\Sigma^{-1}_{2,2}$ seems to be playing a really important role. A few questions:  Am I right to treat precision as a measure of how closely observations are clustered about the expectation? Why is the covariance squared in the latter equation? (Is there a geometric interpretation?) So far, I've been treating $\Sigma_{1,2} * \Sigma^{-1}_{2,2}$ as a ratio, (a/b), and so this ratio acts to scale the (second) $\Sigma_{1,2}$, essentially accounting for/damping the effect of the covariance; I don't know if this is valid. Anything else you'd like to add/clarify?
20925
I have some variables and I want to reduce their number for further analysis. I initially thought of combining them using factor analysis. But since the variables are of all kinds (rating, count, ordinal, continuous dollar amount) I am thinking about using CATPCA for it. However, I have some questions about the technique -   Is CATPCA the right approach or Latent Class Factor Analysis would be the better one? By using the different kinds of variables together - how will the object scores be interpreted? For example - The rating question is on a 10 point scale but the dollar amount ranges from 0 to 1000s, so what will the object score obtained after CATPCA represent? And is it comparable to the Factor Score obtained from traditional PCA? For ordinal variables - Do all of them need to be in the same form, e.g. 1 meaning lowest and 10 meaning highest? Or can they be used in whatever form they are? Is it good to categorize the count and continuous variables based on their distribution and then treat then as ordinal/categorical variables?   I would highly appreciate any help you can provide!
31913
I'm reading up on PCA, and I'm understanding most of what's going on in terms of the derivation apart from the assumption that eigenvectors need to be orthogonal and how it relates to the projections (PCA scores) being uncorrelated? I have two explanations provided below which use a link between orthogonality and correlation but fail to really explain it: , .  In the second picture it says that the condition $a_{2}^{T}a_{1}=0$  is imposed to ensure that the projection $y_{2}=Xa_2$ will be uncorrelated with $y_{1}=Xa_1$. Can someone provide an example to show why orthogonal vectors ensure uncorrelated variables?  What would happen in PCA if I chose vectors which are not orthogonal; is this even possible? I have read elsewhere that orthogonality is just a by-product of the covariance matrix being symmetric which would suggest that it's not possible to have non pairwise orthogonal eigenvectors. However in the first picture in search of the most 'suitable' matrix it almost seems as we are choosing $p_{1},\ldots,p_{m}$ to be orthogonal to give us a more convenient matrix $\textbf{P}$ one which has nice properties.  I've read other posts on this topic but have been unsatisfied with incorporation of the intuition with uncorrelated variables. I really appreciate any help in understanding this confusion!!
123279
The  for cross-validation says the following about using feature-scaling and cross-validation:      Just as it is important to test a predictor on data held-out from training, preprocessing (such as standardization, feature selection, etc.) and similar data transformations similarly should be learnt from a training set and applied to held-out data for prediction   I understand the reason behind this is to prevent information leakage between the training &amp; test sets during cross-validation, which could result in an optimistic estimate of model performance.  I am wondering then, if I wish to use Principal Component Analysis to reduce the size of a feature set before training say a regression model, and PCA requires feature-scaling to be effective, how do I chain feature-scaling to PCA to cross-validated regression, without introducing data leakage between the train-test splits in the cross-validation?
72054
What difference does centering (or de-meaning) your data make for PCA? I've heard that it makes the maths easier or that it prevents the first PC from being dominated by the variables' means, but I feel like I haven't been able to firmly grasp the concept yet.   For example, the top answer here  describes how not centering would pull the first PCA through the origin, rather than the main axis of the point cloud. Based on my understanding of how the PC's are obtained from the covariance matrix's eigenvectors, I can't understand why this would happen.  Moreover, my own calculations with and without centering seem to make little sense.  Consider the setosa flowers in the iris dataset in R. I calculated the eigenvectors and eigenvalues of the sample covariance matrix as follows.  data(iris) df &lt;- iris[iris$Species=='setosa',1:4] e &lt;- eigen(cov(df)) &gt; e $values [1] 0.236455690 0.036918732 0.026796399 0.009033261  $vectors             [,1]       [,2]       [,3]        [,4] [1,] -0.66907840  0.5978840  0.4399628 -0.03607712 [2,] -0.73414783 -0.6206734 -0.2746075 -0.01955027 [3,] -0.09654390  0.4900556 -0.8324495 -0.23990129 [4,] -0.06356359  0.1309379 -0.1950675  0.96992969   If I center the dataset first, I get exactly the same results. This seems quite obvious, since centering does not change the covariance matrix at all.   df.centered &lt;- scale(df,scale=F,center=T) e.centered&lt;- eigen(cov(df.centered)) e.centered   The prcomp function results in exactly this eigenvalue-eigenvector combination as well, for both the centered and uncentered dataset.   p&lt;-prcomp(df) p.centered &lt;- prcomp(df.centered) Standard deviations: [1] 0.48626710 0.19214248 0.16369606 0.09504347  Rotation:                      PC1        PC2        PC3         PC4 Sepal.Length -0.66907840  0.5978840  0.4399628 -0.03607712 Sepal.Width  -0.73414783 -0.6206734 -0.2746075 -0.01955027 Petal.Length -0.09654390  0.4900556 -0.8324495 -0.23990129 Petal.Width  -0.06356359  0.1309379 -0.1950675  0.96992969   However, the prcomp function has the default option center = TRUE. Disabling this option results in the following PC's for the uncentered data (p.centered remains the same when center is set to false):  p.uncentered &lt;- prcomp(df,center=F) &gt; p.uncentered Standard deviations: [1] 6.32674700 0.22455945 0.16369617 0.09766703  Rotation:                     PC1         PC2        PC3         PC4 Sepal.Length -0.8010073  0.40303704  0.4410167  0.03811461 Sepal.Width  -0.5498408 -0.78739486 -0.2753323 -0.04331888 Petal.Length -0.2334487  0.46456598 -0.8317440 -0.19463332 Petal.Width  -0.0395488  0.04182015 -0.1946750  0.97917752   Why is this different from my own eigenvector calculations on the covariance matrix of the uncentered data? Does it have to do with the calculation? I've seen mentioned that prcomp uses something called the SVD method rather than the eigenvalue decomposition to calculate the PC's. The function princomp uses the latter, but its results are identical to prcomp. Does my issue relate to the answer I described at the top of this post?  EDIT: Issue was cleared up by the helpful @ttnphns. See his comment below, on this question:  and in this answer: . In short: a covariance matrix implicitly involves centering of the data already. PCA uses either SVD or eigendecomposition of the centered data $\bf X$, and the covariance matrix is then equal to ${\bf X'X}/(n-1)$.
87886
I am trying to predict the outcome of a complex system using neural networks (ANN's). The outcome (dependent) values range between 0 and 10,000. The different input variables have different ranges. All the variables have roughly normal distributions.   I consider different options to scale the data before training. One option is to scale the input (independent) and output (dependent) variables to [0, 1] by  using the mean and standard deviation values of each variable, independently. The problem with this method is that if I use the sigmoid activation function at the output, I will very likely miss extreme data, especially those not seen in the training set  Another option is to use a z-score. In that case I don't have the extreme data problem; however, I'm limited to a linear activation function at the output.   What are other accepted normalization techniques that are in use with ANN's? I tried to look for reviews on this topic, but failed to find anything useful.