I'm doing anomaly detection on unsupervised data using k-means I got a result but I don't know how to validate my clustering result. by plotting I can see my anomalies but how should I validate that clustering is accurate. in supervised learning, we use cross-validation like this any method or technique for unsupervised means.
25448
One of the biggest issue with cluster analysis is that we may happen to have to derive different conclusion when base on different clustering methods used  (including different linkage methods in hierarchical clustering).  I would like to know your opinion on this - which method will you select, and how. One might say "the best method of clustering is which gives you the right answer"; but I may question in response that cluster analysis is supposed to be an unsupervised technique - so how do I know which method or linkage is the right answer?  In general: is a clustering alone robust enough to rely on? Or we need a second method and get a shared result to be based on both?  My question is not only about possible ways to validate / evaluate clustering performance, but is broader - on what basis do we select/prefer one clustering method/algorithm over another one. Also, is there common warnings that we should look around when we are selecting a method to cluster our data?  I know that it is very general question and very difficult to answer. I only would like to know if you have any comment or any advise or any suggestion for me to learn more about this.
49678
I am working on a text clustering problem. The data contains several sentences. Is there a good algorithm which reaches high accuracy on short text?  Can you provide good references?   Algorithms such as KMeans, spectral clustering does not work well for this problem.
123758
I'm very confused, when I read on the wikipedia "From a mathematical standpoint, Rand index is related to the accuracy, but is applicable even when class labels are not used."   and   But when I use in R the rand.index function from fossil package and the Accuracy function from MLmetrics it doesn't give the same answer  &gt; Accuracy(predicted, real) [1] 0.8266667  &gt; rand.index(predicted,real) [1] 0.7114989  Please can someone explain to me the difference between these two and which one should I use?  NB:I use the k-medoids clustering algorithm (function pam from cluster package)
123350
There are numerous threads in CrossValidated on the topic of model selection and cross validation. Here are a few:    @DikranMarsupial's  to    However, the answers to those threads are fairly generic and mostly highlight the issues with particular approaches to cross validation and model selection.   To make things as concrete as possible, say for example that we are working with an SVM with an RBF kernel:  $K(x, x' ) = (\gamma \, \vert x - x'\vert)^2$, and that I have a dataset of features X and labels y, and that I want to    Find the best possible values of my model ($\gamma$ and $C$) Train the SVM with my dataset (for final deployment) Estimate the generalization error and the uncertainty (variance) around this error   To do so, I would personally do  a grid search, e.g. I try every possible combination of  $C$ and $\gamma$. For simplicity, we can assume the following ranges:   $C \in \{10, 100, 1000\}$ $\gamma \in \{0.1, 0.2, 0.5, 1.0\}$   More specifically, using my full dataset I do the following:   For every ($C$,$\gamma$) pair, I do repeated iterations (e.g. 100 random repetitions) of $K$-fold cross validation (e.g. $K=10$), on my dataset, i.e. I train my SVM on $K-1$ folds and evaluate the error on the fold left, iterating through all $K$ folds. Overall, I collect 100 x 10 = 1000 test errors. For each such ($C$,$\gamma$) pair, I compute the mean and the variance of those 1000 test errors $\mu_M, \sigma_M$.   Now I want to choose the best model (the best kernel parameters) that I would use to train my final SVM on the full dataset. My understanding is that choosing the model that had the lowest error mean and variance $\mu_M$ and $\sigma_M$ would be the right choice, and that this model's $\mu_M$ are $\sigma_M$ are my best estimates of the model's generalization error bias and variance when training with the full dataset.  BUT, after reading the answers in the threads above, I am getting the impression that this method for choosing the best SVM for deployment and/or for estimating its error (generalization performance), is flawed, and that there are better ways of choosing the best SVM and reporting its error. If so, what are they? I am looking for a concrete answer please.  Sticking to this problem, how specifically can I choose the best model and properly estimate its generalization error?
15149
I am preparing a paper in the field of Computer Science.  In order to report test results, we usually run a number of tests and report the average of those tests.  For each test, we generate random data.  Because of the randomness, at some points, the results may come out not as expected.  For instance, a graph may be like:   Usually, one should explain why on points 8, 11 and 12 there is a decrease on the plot. Probably, it is because of that randomness.  Not hand-crafting all the graph, but just manipulating a few points makes the graph acceptable:   Since three weeks or so, I work my ass off and try to figure out why my resulting graph looks like the first one. Sometimes I feel like yielding to temptation and just modify the raw data before I go crazy.  I believe, at this point the title became misleading, so let me make it clear:  I am not seeking an advice on data manipulation. I will not manipulate my data. However, I ask to myself "how the hell this can be detected?"  And now, I don't only ask to myself, but to whole community. How is this detected? For editors, referees out there, have you ever detected something like this?
27906
If I have a certain dataset, how smart would it be to initialize cluster centers using means of random samples of that dataset?  For example, suppose I want 5 clusters. I take 5 random samples of say, size=20% of the original dataset. Could I then take the mean of each of these 5 random samples and use those means as my 5 initial cluster centers? I don't know where I read this but I wanted to know what you guys think about the idea.    UPDATE: Please see this thread  for the general discussion about the various initialization methods.
23026
The representative point of a cluster or cluster center for the k-means algorithm is the component-wise mean of the points in its cluster. The mean is chosen because it helps to minimize the within cluster variances (which is to say that it is minimizing within cluster squared Euclidean distance, since its the same). Now, when I want to use a different distance metric, like L1 (Manhattan distance), I would be minimizing the within cluster absolute deviations (if I choose to). I was looking at what the centers calculation would be like, because the mean won't reduce the new objective function, and I found this on the Matlab documentation of their kmeans function for the distance parameter     cityblock - Sum of absolute differences, i.e., the L1 distance. Each centroid is the component-wise median of the points in that cluster.   Now, how did they come up with the component-wise median? How does a median minimize absolute deviations?
103046
I'm unclear on the exact process of using the validation data.  Let's say that I fit my neural network model and adjust hyperparameters using the training set and validation set. Do I then evaluate the test set on this model? Or do I recombine the validation and training sets and fit a fresh model with the hyperparameters that I found during the validation phase, and then evaluate on the test data? I have seen a number of different notebooks and examples that do both ways.  Surely, once I've found my hyperparameters, it makes sense to fit a fresh model using the full training set (recombined with validation set), since the validation loss has no effect on the weights.
95184
What is the minimum number of data points required for a kernel density estimation to be considered non-misleading/acceptable/adequate?  Is there a some rule based on how dispersed the data is?  For example, if the inter-quartile range is "small enough", do you need less points than if the inter-quartile range is "larger"?  Alternatively, you may have encountered a situation where a certain size of dataset was "enough" or "not enough" and can share the reasons why KDE worked/didn't work in that specific case?  Criteria to assess accuracy:  If you had two samplesfrom the same population and you do a KDE on each one, then the two resulting densities are very similar.
116707
I trained an SVM to classify images based on some extracted features (using the ISIC dataset). The resulting ROC curve produced by sklearn looks like this:   I have don't quite understand the line for class 0 and 1 (orange and green). I have never seen a ROC curve under the diagonal. What does this tell me now and is this even a valid curve? The other computed evaluation metrices are the following if that helps my question.
85672
I have a question about cluster analysis. There are 3000 companies, which have to be clustered according to their power usage over 5 years. Each company has values for every hour during 5 years. I would like to find out if some companies have the same pattern in usage power over the time period. The results should be used for daily prediction of power usage. If you have some ideas how to cluster time series in SPSS, please share with me.
37196
I looked for a way to remove outliers from a dataset and I found .  In some of the comments and answers to this question, however, people mentioned that it is bad practice to remove outliers from the data.   In my dataset I have several outliers that very likely are just due to measurement errors. Even if some of them are not, I have no way of checking it case by case, because there are too many data points. Is it statistically valid than just to remove the outliers? Or, if not, what could be another solution?   If I just leave those points there, they influence e.g. the mean in a way that does not reflect reality (because most of them are errors anyway).   EDIT: I am working with skin conductance data. Most of the extreme values are due to artifacts like somebody pulling on the wires.   EDIT2: My main interest in analyzing the data is to determine if there is a difference between two groups
71119
I had a job interview for a data science position. During the interview, I was asked what do I do to make sure the model is not overfitting. My first answer was to use cross-validation to assess the performance of the model. However, the interviewer said that even cross-validation cannot identify completely overfitting. Then I mentioned regularization, but the interviewer said that this could help to reduce overfitting (which I agree), but not to detect it. Are there other techniques that can be used to make sure a model is not overfitting?
25196
I am interested in the current state of the art for selecting initial seeds (cluster centers) for K-means.   Googling leads to two popular choices:   random selection of initial seeds, and, using the KMeans++ selection technique:    Are there any other promising methods that anyone here is aware of, which might not be as popular?
16845
What is an appropriate cross-validation technique for time series data?   I have a daily 4 years time series data and fitting a SVM model by MATLAB R2015b:  SVMModel = fitcsvm(Input, binary_output,'KernelFunction','RBF','BoxConstraint',1); CVSVMModel = crossval(SVMModel); z = kfoldLoss(CVSVMModel)   This a binary classification problem. As default I used 10-fold cross validation, but because of the random nature of this method I think this is not suitable for time series data.   Questions:   Is it better to use other techniques like sliding window validation as discussed ?  How we can implement these techniques in MATLAB?  Are there any predefined functions for other proposed techniques?
52616
I have a dataset that contains data from $307$ subjects and nine variables for each subject. I would like to run a PCA. My problem is that I get a Kaiser-Meyer-Olkin (KMO) value of $0.06$.  Can it be due to the fact that within every subject the variables add up to $1$? Is there another way to test whether it is okay to run PCA on my data?
44407
I trained and tested two models on the same dataset in a 10-fold cross validation manner. I want to show that one model is supreme than the other. Therefore, I want to show the better model has a higher test accuracy mean. In best practice, should I use paired t-test, unpaired t-test or other kinds of test? Should I use Welch's test if people are concerned about unequal variances?