I am talking about K-means clustering problem.  Theoretically, my understanding is :     If there are K clusters, the K-means algorithm outputs K centroids/means such that it minimizes total sum of SSE (Sum squared error) of all clusters.    Now, I was trying to understand how each iteration works. I referred 'an example' section at  .   This suggests the following procedure for finding the k means:  Make initial guesses for the means m1, m2, ..., mk  Until there are no changes in any mean  Use the estimated means to classify the samples into clusters.  **(we can use a minimum-distance classifier to separate them. That is, we can say that x is in cluster i if || x - mi ||  is the minimum of all the k distances)**      For i from 1 to k     Replace mi with the mean of all of the samples for cluster i     end_for  end_until   I don't see, at any point, this iteration dealing with SUM of SSEs. It just considers each clusters independently.  So, can I say the above iteration process doesn't totally encapsulate the details of K-means algorithm. It is a sub-optimal approach which can work in some cases and NOT in some other cases?
120683
From a conceptual and algorithmic standpoint, I understand how K-means works. However, from a mathematical standpoint, I don't understand why minimizing the WCSS (within-cluster sums of squares) will necessarily maximize the distance between clusters. In other words, can somebody show how this  is equal to maximizing the distance between clusters? It would be helpful to see a derivation that shows all of the steps or please point me to the appropriate reference(s).  Update I found this Witten and Tibshirani  but it isn't obvious how to get from equation 7 to equation 8.
77854
I'm studying 'Introduction to Statistical Learning' by James, Witten, Hastie, Tibshirani.   In page 139, of their book, they began by introducing Bayes' Theorem $p_k(X)=P(Y=k|X=x) = \dfrac{\pi_kf_k(x)}{\sum_{l=1}^k \pi_l f_l(x)}$. $\pi$ is not mathematical constant, but denotes the prior probability. Nothing is strange in this equation.   The book claims that it wants to obtain an estimate for $f_k(x)$ that it can plug into the equation given above. To estimate $f_k(x)$, it assumes that is normal. In the one dimensional setting, $f_k(x)=\dfrac{1}{\sqrt{2\pi}\sigma}\exp(-\dfrac{1}{2\sigma^2}(x-\mu_k)^2)$, where $\mu_k$ and $\sigma^2_k$ are the mean and variance for the $k$th class. It is assumed that $\sigma^2_1 = \sigma^2_2 = \cdots = \sigma^2_K$. (I began to get confused from the last statement.)   Plugging $f_k$ into $p_x$, you have this quite messy equation (1) :   $$p_x(k)=\dfrac{\pi_k \frac{1}{\sqrt{2\pi}\sigma}\exp(-\frac{1}{2\sigma^2}(x-\mu_k)^2)}{\sum_{l=1}^K \pi_l \frac{1}{\sqrt{2\pi}\sigma}\exp(-\frac{1}{2\sigma^2}(x-\mu_l)^2)}. $$  Again, no surprises here since it is just substitution.     The Bayes' Classifier involves assigning an observation to the class for which equation (1) is the largest. Taking the log of Equation (1) and rearranging terms, it is not hard to show that this is equivalent to assigning the observation to the class for which the following is the largest:  $$\delta_k(x)=x \cdot \dfrac{\mu_k}{\sigma^2} - \dfrac{\mu_k^2}{2\sigma^2} + \log(\pi_k) $$  Question: I do not understand where this came from, and what it means. I tried doing the log of equation and it doesn't become this. Are we taking the derivative somewhere here, since this is the largest observation?
23026
The representative point of a cluster or cluster center for the k-means algorithm is the component-wise mean of the points in its cluster. The mean is chosen because it helps to minimize the within cluster variances (which is to say that it is minimizing within cluster squared Euclidean distance, since its the same). Now, when I want to use a different distance metric, like L1 (Manhattan distance), I would be minimizing the within cluster absolute deviations (if I choose to). I was looking at what the centers calculation would be like, because the mean won't reduce the new objective function, and I found this on the Matlab documentation of their kmeans function for the distance parameter     cityblock - Sum of absolute differences, i.e., the L1 distance. Each centroid is the component-wise median of the points in that cluster.   Now, how did they come up with the component-wise median? How does a median minimize absolute deviations?
60350
I know that Random forest uses Bagging, but how does ensemble give a single prediction in sklearn Bagging vs Random forest? Also, I do not understand why in Bagging you not need to choose a tree estimator, while not in RF?  other than this, are the other steps the same? wht are main implementation differences in sklearn Bagging vs Random forest?  bonus points if you have "explain like I'm 5 years old" simple example using sklearn please.
18904
I believe there's a way to find the kth largest element in an unsorted array of length n in O(n).  Or perhaps it's "expected" O(n) or something.  How can we do this?
118484
I am trying to test how well my unsupervised K-Means clustering properly clusters my data.  I have an unsupervised K-Means clustering model output (as shown in the first photo below) and then I clustered my data using the actual classifications.    The photo below are the actual classifications.  I am trying to test, in Python, how well my K-Means classification (above) did against the actual classification.    For my K-Means code, I am using a simple model, as follows:  kmeans = KMeans(n_clusters=4, random_state=0).fit(myData) labels = kmeans.labels_   What would be the best way for me to compare how well my unsupervised KMeans clustering model did against the actual classifications?
50890
In a VAE, the encoder learns to output two vectors:  $$\mathbf{\mu} \in\ \mathbb{R}^{z}$$ $$\mathbf{\sigma} \in\ \mathbb{R}^{z}$$  which are the mean and variances for the latent vector $\mathbf{z}$, the latent vector $\mathbf{z}$ is then calculated by:  $$\mathbf{z} = \mu + \sigma \epsilon$$  where: $\epsilon = N(0, \mathbf{I}_{z \times z})$  The KL divergence loss for a VAE for a single sample is defined as (referenced from  and ):  $$\frac{1}{2} \left[ \left(\sum_{i=1}^{z}\mu_{i}^{2} + \sum_{i=1}^{z}\sigma_{i}^{2} \right) - \sum_{i=1}^{z} \left(log(\sigma_{i}^{2}) + 1 \right) \right]$$  Though, I'm not sure how they got their results, would anyone care to explain or point me to the right resources?
10595
It is a common practice to apply PCA (principal component analysis) before a clustering algorithm (such as k-means). It is believed that it improves the clustering results in practice (noise reduction).   However I am interested in a comparative and in-depth study of the relationship between PCA and k-means. For example, Chris Ding and Xiaofeng He, 2004,  showed that "principal components are the continuous solutions to the discrete cluster membership indicators for K-means clustering". However, I have hard time understanding this paper, and Wikipedia actually .  Also, the results of the two methods are somewhat different in the sense that PCA helps to reduce the number of "features" while preserving the variance, whereas clustering reduces the number of "data-points" by summarizing several points by their expectations/means (in the case of k-means). So if the dataset consists in $N$ points with $T$ features each, PCA aims at compressing the $T$ features whereas clustering aims at compressing the $N$ data-points.  I am looking for a layman explanation of the relations between these two techniques + some more technical papers relating the two techniques.
6474
When performing k-fold cross validation, I understand that you obtain the accuracy metrics by pointing all the folds except one at that one fold and make predictions, and then repeat this process $k$ times.  You can then run accuracy metrics on all of your instances (precision, recall, % classified correctly), which ought to be the same as if you calculated them each time and then averaged the result (correct me if I'm wrong).  The end result you want is a final model.  Do you average the models obtained to make your set of $k$ predictions to end up with the model that has the accuracy metrics obtained by the above method?
96159
Over the past few days I have been trying to understand how Markov Chain Monte Carlo (MCMC) works. In particular I have been trying to understand and implement the Metropolis-Hastings algorithm. So far I think I have an overall understanding of the algorithm but there are a couple of things that are not clear to me yet. I want to use MCMC to fit some models to data. Because of this I will describe my understanding of Metropolis-Hastings algorithm for fitting a straight line $f(x)=ax$ to some observed data $D$:  1) Make an initial guess for $a$. Set this $a$ as our current $a$ ($a_0$). Also add $a$ at the end of the Markov Chain ($C$).  2) Repeat the steps bellow a number of times.  3) Evaluate current likelihood (${\cal L_0}$) given $a_0$ and $D$.  4) Propose a new $a$ ($a_1$) by sampling from a normal distribution with $\mu=a_0$ and $\sigma=stepsize$. For now, $stepsize$ is constant.  5) Evaluate new likelihood (${\cal L_1}$) given $a_1$ and $D$.  6) If ${\cal L_1}$ is bigger than ${\cal L_0}$, accept $a_1$ as the new $a_0$, append it at the end of $C$ and go to step 2.  7) If ${\cal L_1}$ is smaller than ${\cal L_0}$ generate a number ($U$) in range [0,1] from a uniform distribution  8) If $U$ is smaller than the difference between the two likelihoods (${\cal L_1}$ - ${\cal L_0}$), accept $a_1$ as the new $a_0$, append it at the end of $C$ and go to step 2.  9) If $U$ is bigger than the difference between the two likelihoods (${\cal L_1}$ - ${\cal L_0}$), append the $a_0$ at the end of $C$, keep using the same $a_0$, go to step 2.  10) End of Repeat.  11) Remove some elements from the start of $C$ (burn-in phase).  12) Now take the average of the values in $C$. This average is the estimated $a$.  Now I have some questions regarding the above steps:   How do I construct the likelihood function for $f(x)=ax$ but also for any arbitrary function? Is this a correct implementation of Metropolis-Hastings algorithm? How the selection of the random number generation method at Step 7 can change the results? How is this algorithm going to change if I have multiple model parameters? For example, if I had the model $f(x)=ax+b$.   Notes/Credits: The main structure of the algorithm described above is based on the code from an MPIA Python Workshop.
21705
I got this sum, in some work related to another :  $$S_m=\sum_{k=1}^m  \frac{1}{k}{m \choose k} $$  Are there any known results about this (bounds, asymptotics)?
86598
When running hierarchical clustering analysis of a matrix of individuals x samples (e.g., employee performances across different days), there are several possibilities for normalization. If one is clustering the columns (to see whether on certain days individuals perform similarly), one could   z-score normalize across the rows to make each individual employees mean and standard deviation comparable across days, or z-score normalize across the columns to make all employees comparable within a day, or  not normalize at all and cluster the raw values   Could someone explain the relative advantages/disadvantages of each approach here? To clarify, I am using correlation distance.  Methods 1 or 2 in practice give different results but it's not clear that for the task of seeing if days cluster together, whether #1 or #2 are more appropriate if one chooses to normalize.
14035
Say clustering was performed. My concern is to understand what characterizes certain cluster. Which variables differ most for a certain cluster? Are there any methods for performing such kind of analysis?  For example, say I have 50 variables and 10 clusters. The desired result would sound like: for a cluster 1 variables 5, 16, 23, 42 and 49 contribute to the biggest difference from other clusters.
85672
I have a question about cluster analysis. There are 3000 companies, which have to be clustered according to their power usage over 5 years. Each company has values for every hour during 5 years. I would like to find out if some companies have the same pattern in usage power over the time period. The results should be used for daily prediction of power usage. If you have some ideas how to cluster time series in SPSS, please share with me.
59474
After training the kmeans model, how can i obtain the number of classes of the model? i've tried with get(), getInfo() and doesnt work.      Value that must to be extracted in "//var to_print_2 = clusterer.get('nClusters')"  // STEP 1: Import image we wish to classify var l8 = ee.ImageCollection('LANDSAT/LC08/C01/T1')  // Remove the clouds var cloud_free = ee.Algorithms.Landsat.simpleComposite({   collection: l8.filterDate('2018-01-01', '2018-12-31'),   asFloat: true })  // STEP 2: Draw your box which you will sample from!   // Make the training dataset. var training = cloud_free.sample({   region: geometry,   scale: 30,   numPixels: 5000 });  // Step 3: Create the clusterer and train it, play around with // number of classes var clusterer = ee.Clusterer.wekaKMeans(3).train(training);  print('clusterer',clusterer instanceof ee.ComputedObject);  var to_print = clusterer.getInfo();  print('clusterer_info',clusterer.toString());  print(typeof clusterer);  var clusterer_list = ee.List(clusterer);  print('clusterer_list',clusterer_list);  print(typeof clusterer_list);   //var to_print_2 = clusterer.get('nClusters');  //print(to_print_2);  //var to_print_3 = clusterer.Filter.filterMetadata('nClusters','equals',3);  //print(to_print_3);  // Step 4: Classify our image var classified = cloud_free.cluster(clusterer); print(classified); var cluster_1 = classified.eq(0); print('min',cluster_1);   // Display Map.setCenter(-122.31040963183364, 37.782065864682096, 11) Map.addLayer(classified.randomVisualizer(), {}, 'clusters');
54554
I want to produce four groups (high/high, high/low, low/high and low/low) using two continues variables and compare these groups in terms of a few dependent variables. I know that cluster analysis (k-means and hierarchical) is a type of 'end' analysis which is the main goal of the paper to reach such clusters and then discuss the nature of the clusters. However, I am willing to use this technique (k-means by using k=4) as a kind of preliminary analysis to produce the groups. I know that scatter plot and means splits are other alternatives but they have their own disadvantages that cluster analysis may resolve. What do you think?
12370
I understand the difference between k medoid and k means. But can you give me an example with a small data set where the k medoid output is different from k means output.