I am a beginner. I have a dataset of 1700 samples with 4 features and I have to perform Hierarchical Clustering (the agglomerative version) and I need to decide whether or not to scale the data and perform PCA. If I scale the data (I tried almost all the "scalers") and then perform PCA, the clustering seems to be incorrect, while if I perform PCA without data scaling, it seems correct. Why is this? Or do you think I am probably doing something wrong?
4429
What are the main differences between performing principal component analysis (PCA) on the correlation matrix and on the covariance matrix? Do they give the same results?
56040
In machine learning tasks it is common to shuffle data and normalize it. The purpose of normalization is clear (for having same range of feature values). But, after struggling a lot, I did not find any valuable reason for shuffling data.  I have read this post  discussing when we need to shuffle data, but it is not obvious why we should shuffle the data. Furthermore, I have frequently seen in algorithms such as Adam or SGD where we need batch gradient descent (data should be separated to mini-batches and batch size has to be specified). It is vital according to this  to shuffle data for each epoch to have different data for each batch. So, perhaps the data is shuffled and more importantly changed.  Why do we do this?
39340
It seems that a number of the statistical packages that I use wrap these two concepts together. However, I'm wondering if there are different assumptions or data 'formalities' that must be true to use one over the other. A real example would be incredibly useful.
33149
Is there a rational for the number of observations per cluster in a random effect model? I have a sample size of 1,500 with 700 clusters modeled as exchangeable random effect. I have the option to merge clusters in order to build fewer, but larger clusters. I wonder how can I choose the minimum sample size per cluster as to have meaningful results in predicting the random effect for each cluster? Is there a good paper that explains this?
45073
Normally in principal component analysis (PCA) the first few PCs are used and the low variance PCs are dropped, as they do not explain much of the variation in the data.  However, are there examples where the low variation PCs are useful (i.e. have use in the context of the data, have an intuitive explanation, etc.) and should not be thrown away?
72054
What difference does centering (or de-meaning) your data make for PCA? I've heard that it makes the maths easier or that it prevents the first PC from being dominated by the variables' means, but I feel like I haven't been able to firmly grasp the concept yet.   For example, the top answer here  describes how not centering would pull the first PCA through the origin, rather than the main axis of the point cloud. Based on my understanding of how the PC's are obtained from the covariance matrix's eigenvectors, I can't understand why this would happen.  Moreover, my own calculations with and without centering seem to make little sense.  Consider the setosa flowers in the iris dataset in R. I calculated the eigenvectors and eigenvalues of the sample covariance matrix as follows.  data(iris) df &lt;- iris[iris$Species=='setosa',1:4] e &lt;- eigen(cov(df)) &gt; e $values [1] 0.236455690 0.036918732 0.026796399 0.009033261  $vectors             [,1]       [,2]       [,3]        [,4] [1,] -0.66907840  0.5978840  0.4399628 -0.03607712 [2,] -0.73414783 -0.6206734 -0.2746075 -0.01955027 [3,] -0.09654390  0.4900556 -0.8324495 -0.23990129 [4,] -0.06356359  0.1309379 -0.1950675  0.96992969   If I center the dataset first, I get exactly the same results. This seems quite obvious, since centering does not change the covariance matrix at all.   df.centered &lt;- scale(df,scale=F,center=T) e.centered&lt;- eigen(cov(df.centered)) e.centered   The prcomp function results in exactly this eigenvalue-eigenvector combination as well, for both the centered and uncentered dataset.   p&lt;-prcomp(df) p.centered &lt;- prcomp(df.centered) Standard deviations: [1] 0.48626710 0.19214248 0.16369606 0.09504347  Rotation:                      PC1        PC2        PC3         PC4 Sepal.Length -0.66907840  0.5978840  0.4399628 -0.03607712 Sepal.Width  -0.73414783 -0.6206734 -0.2746075 -0.01955027 Petal.Length -0.09654390  0.4900556 -0.8324495 -0.23990129 Petal.Width  -0.06356359  0.1309379 -0.1950675  0.96992969   However, the prcomp function has the default option center = TRUE. Disabling this option results in the following PC's for the uncentered data (p.centered remains the same when center is set to false):  p.uncentered &lt;- prcomp(df,center=F) &gt; p.uncentered Standard deviations: [1] 6.32674700 0.22455945 0.16369617 0.09766703  Rotation:                     PC1         PC2        PC3         PC4 Sepal.Length -0.8010073  0.40303704  0.4410167  0.03811461 Sepal.Width  -0.5498408 -0.78739486 -0.2753323 -0.04331888 Petal.Length -0.2334487  0.46456598 -0.8317440 -0.19463332 Petal.Width  -0.0395488  0.04182015 -0.1946750  0.97917752   Why is this different from my own eigenvector calculations on the covariance matrix of the uncentered data? Does it have to do with the calculation? I've seen mentioned that prcomp uses something called the SVD method rather than the eigenvalue decomposition to calculate the PC's. The function princomp uses the latter, but its results are identical to prcomp. Does my issue relate to the answer I described at the top of this post?  EDIT: Issue was cleared up by the helpful @ttnphns. See his comment below, on this question:  and in this answer: . In short: a covariance matrix implicitly involves centering of the data already. PCA uses either SVD or eigendecomposition of the centered data $\bf X$, and the covariance matrix is then equal to ${\bf X'X}/(n-1)$.
86650
I've with me 50 MB data from a machine consisting of event logs such as device status, warning and error. I wish to perform text mining on the same to find correlation between errors i.e. one error could trigger another in future and take a prescriptive action on it. I've used hierarchical clustering in R to generate a dendrogram but the result couldn't yeild expected insight. Hence, I've planned to perform Apriori Algorithm on the same.  My queries are:   Is Hierarchical Clustering suggested to find such correlations? Is Apriori an apt algorithm in such situation and why? Or, is there any other approach to solve this problem?   The logs that I've looks like this :
52616
I have a dataset that contains data from $307$ subjects and nine variables for each subject. I would like to run a PCA. My problem is that I get a Kaiser-Meyer-Olkin (KMO) value of $0.06$.  Can it be due to the fact that within every subject the variables add up to $1$? Is there another way to test whether it is okay to run PCA on my data?
86126
Is there any methodical way to compare the directions, magnitudes, etc of PCA results for different samples drawn from the same population?  I'm leaving the nature of the test deliberately vague because I'd like to hear all the various possibilities... e.g. there might be (and I'm speculating here) a test comparing the sizes of the first principal components, or a test comparing the directions of the principal components, or there's some kind of distance measure between PCA outcomes and a test statistic for their equality.  As far as a use case goes, I don't have on in mind. Just out of curiosity, maybe as an exploratory technique.
13209
I am working with many algorithms: RandomForest, DecisionTrees, NaiveBayes, SVM (kernel=linear and rbf), KNN, LDA and XGBoost. All of them were pretty fast except for SVM. That is when I got to know that it needs feature scaling to work faster. Then I started wondering if I should do the same for the other algorithms.
87153
I have ~30 observations of individuals ranking 20 different products from top to bottom (1-20). I would like to have a way to "cluster" these individuals based on their rankings of the different products.  For example, I would like to cluster respondents with similar high rankings of the same products. This can be done looking at only the top 3 products ranked by each respondent, if that's simpler.  What would be an appropriate way to perform this analysis?
76019
Both PCA and autoencoder can do demension reduction, so what are the difference between them? In what situation I should use one over another?
66825
I have conducted PCA that has reduced the dimensions of my data from more than 20 to 7 (7 PCAs explain about 85% of the total variation). As a second step, I have to cluster my data based on these new 7 PCAs. My question is: how I should reconstruct/transform my data (cbind/rbind)? As I understand, each PCA is a weighted mix of the original variables. So should I just replace the old variables with the PCAs? Also, how should I interpret the final results? If there was no PCA, each cluster would incorporate all variables in some proportion. But after PCA, how would I describe each cluster? For instance, cluster 1 contains 40% of PCA 1 and PCA 1, in turn, has xxx loading scores? Something like this? Would greatly appreciate any help on interpreting this.
31913
I'm reading up on PCA, and I'm understanding most of what's going on in terms of the derivation apart from the assumption that eigenvectors need to be orthogonal and how it relates to the projections (PCA scores) being uncorrelated? I have two explanations provided below which use a link between orthogonality and correlation but fail to really explain it: , .  In the second picture it says that the condition $a_{2}^{T}a_{1}=0$  is imposed to ensure that the projection $y_{2}=Xa_2$ will be uncorrelated with $y_{1}=Xa_1$. Can someone provide an example to show why orthogonal vectors ensure uncorrelated variables?  What would happen in PCA if I chose vectors which are not orthogonal; is this even possible? I have read elsewhere that orthogonality is just a by-product of the covariance matrix being symmetric which would suggest that it's not possible to have non pairwise orthogonal eigenvectors. However in the first picture in search of the most 'suitable' matrix it almost seems as we are choosing $p_{1},\ldots,p_{m}$ to be orthogonal to give us a more convenient matrix $\textbf{P}$ one which has nice properties.  I've read other posts on this topic but have been unsatisfied with incorporation of the intuition with uncorrelated variables. I really appreciate any help in understanding this confusion!!
75272
I have a cluster of p-dimensional points and given a new p-dimensional point $x$ I want to determine whether or not it is likely to belong to this cluster.   The cluster is made up of $n$ p-dimensional points, I am making the assumption that these points are drawn from a multivariate Gaussian distribution with sample mean ${\hat{\mu_X}}$ and sample covariance matrix $\hat{\sigma_X}$.  Given a new point $x$ I am trying to decide if it is likely to belong to this cluster using the following threshold on the : $$\frac{n}{(n-1)^2}\left(X_i-\hat{\mu}_X\right)'\hat{\sigma}_X^{-1}\left(X_i-\hat{\mu}_X\right)&gt;B_{0.95}\left(\frac{p}{2},\frac{n-p-1}{2}\right)$$  However, some clusters have very few sample points $n$ in which case calculating the inverse covariance matrix $\hat{\sigma_X}^{-1}$ becomes impossible.  Are there any other equivalent or more appropriate measure I can use in this case?
86995
In my machine learning class these two methods were discussed and mentioned that both should be used. I have a couple questions about this:  1) Can I mix and match these two approaches? e.g. Feature Scale x1 and Mean Normalize x2?  2) How do you determine which of these options to apply? It seems that either could accomplish the task of increasing your convergence rates... I suppose you just need to know your data set to understand which will reliably reduce your values while leaving as few outliers as possible?
90415
Say I generate a dataset $X$: the first $i$ samples follow $x_i\sim N(\mu_1,\Sigma_1^2)$, the next j samples follow $N(\mu_2,\Sigma_2^2)$ and the last $l$ samples from $N(\mu_3,\Sigma_3^2)$. Naturally, I denote the clusters $(1,2,3)$ by the order of creation.  When using kmeans(X,3)$clusters I get the predicted assignment of the samples, but here lies the problem: kmeans might enumerate the clusters differently, for example set $N(\mu_2,\Sigma_2^2)$ as cluster#3 and vice versa. In this situation, I cannot compare the 'original' cluster assignment (say $(1,1,2,2,2,3,3,3,3,3)$) with the predicted one (say $(1,1,3,3,3,2,2,2,2,2)$). this will result with only 20% accuracy, even though the actual accuracy is 100%.  Any ideas of how to map the centroids or measure the accuracy of the kmeans performance?