I've seen in a  someone who used  before decision tree or other techniques.  I thought it was just for compressing data but he aimed to improve his score.  How can PCA improve score in this case ? Is it because there is less overfitting ?
32859
Suppose I am running a regression $Y \sim X$. Why by selecting top $k$ principle components of $X$, does the model retain its predictive power on $Y$?   I understand that from dimensionality-reduction/feature-selection point of view, if $v_1, v_2, ... v_k$ are the eigenvectors of covariance matrix of $X$ with top $k$ eigenvalues, then $Xv_1, Xv_2 ... Xv_k$ are top $k$ principal components with maximum variances. We can thereby reduce the number of features to $k$ and retain most of the predictive power, as I understand it.   But why do top $k$ components retain the predictive power on $Y$?   If we talk about a general OLS $Y \sim Z$, there is no reason to suggest that if feature $Z_i$ has maximum variance, then $Z_i$ has the most predictive power on $Y$.   Update after seeing comments: I guess I have seen tons of examples of using PCA for dimensionality reduction. I have been assuming that means the dimensions we are left with have the most predictive power. Otherwise what's the point of dimensionality reduction?
21830
I am working on binary classification problem, I try to evaluate the performance of some classification algorithms (LR,Decission Tree , Random forest ...). I am using  a cross validation technique (to avoid over-fitting) with AUC ROC as scoring function to compare the performance of the algorithms, but I am getting a weird results with Random forest and AdbBoost, I have a perfect AUC_ROC score (i.e. =1) despite the fact that the recall(TPR) and FPR of this algorithms are different from 1 and 0 respectively .  def FPR(y_true, y_pred):     tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()     result = fp / (fp+tn)     return result def FNR(y_true, y_pred):     tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()     result = fn / (tp+fn)     return result FPR_scorer = make_scorer(FPR) FNR_scorer = make_scorer(FNR)     def get_CrossValResults2(model,cv_rst,bestIndx):     best=pd.DataFrame.from_dict(cv_rst).iloc[bestIndx]     roc=&quot;{:.12f}&quot;.format(best['mean_test_roc_auc'])     acc =&quot;{:.0%}&quot;.format(best['mean_test_accuracy'])     prec =&quot;{:.0%}&quot;.format(best['mean_test_precision'])     rec =&quot;{:.0%}&quot;.format( best['mean_test_recall'])     f1 =&quot;{:.0%}&quot;.format(best['mean_test_f1'])     r2=&quot;{:.2f}&quot;.format(best['mean_test_r2'])     g_mean=&quot;{:.2f}&quot;.format(best['mean_test_gmean'])     pr_auc=&quot;{:.8f}&quot;.format(best['mean_test_pr'])     fnr=&quot;{:.0%}&quot;.format(best['mean_test_fnr'])     fpr=&quot;{:.0%}&quot;.format(best['mean_test_fpr'])     rst = pd.DataFrame([[ model, acc,prec,rec,fpr,fnr,f1,roc,pr_auc,g_mean,r2]],columns = ['Model', 'Accuracy', 'Precision', 'Recall','FPR','FNR', 'F1-Score','ROC_auc','PR_auc','gmean','r2'])     return rst        cross_val_rst = pd.DataFrame(columns = ['Model', 'Accuracy', 'Precision', 'Recall','FPR','FNR', 'F1-Score','ROC_auc','PR_auc','gmean','r2'])              scoring = {'accuracy':'accuracy','recall':'recall','precision':'precision','fpr':FPR_scorer,'fnr':FNR_scorer,'f1':'f1' ,'roc_auc':'roc_auc','pr':'average_precision','gmean':Gmean_scorer,'r2':'r2'}     param_grid = {'n_estimators': [200],               'max_depth': [80,90],               'min_samples_leaf': [2,3, 4],             'min_samples_split': [2,5,12],             'criterion': [ 'gini'],              'class_weight' : [class_weights], 'n_jobs' : [-1]}  clf = GridSearchCV(RandomForestClassifier(class_weight=class_weights), param_grid, cv=kfold,scoring=scoring,refit=refit)#Fit the model bestmodel = clf.fit(X,Y)       cross_val_rst = cross_val_rst.append(get_CrossValResults2(model='Random Forrest',bestIndx=bestmodel.best_index_,cv_rst=bestmodel.cv_results_),ignore_index=True)
1306
In today's pattern recognition class my professor talked about PCA, eigenvectors and eigenvalues.   I understood the mathematics of it. If I'm asked to find eigenvalues etc. I'll do it correctly like a machine. But I didn't understand it. I didn't get the purpose of it. I didn't get the feel of it.        I strongly believe in the following quote:     You do not really understand something unless you can explain it to your grandmother. -- Albert Einstein   Well, I can't explain these concepts to a layman or grandma.   Why PCA, eigenvectors &amp; eigenvalues? What was the need for these concepts? How would you explain these to a layman?
3949
Imagine a standard machine-learning scenario:     You are confronted with a large multivariate dataset and you have a   pretty blurry understanding of it. What you need to do is to make   predictions about some variable based on what you have. As usual, you   clean the data, look at descriptive statistics, run some models,   cross-validate them etc., but after several attempts, going back and   forth and trying multiple models nothing seems to work and your   results are miserable. You can spend hours, days, or weeks on such a   problem...   The question is: when to stop? How do you know that your data actually is hopeless and all the fancy models wouldn't do you any more good than predicting the average outcome for all cases or some other trivial solution?  Of course, this is a forecastability issue, but as far as I know, it is hard to assess forecastability for multivariate data before trying something on it. Or am I wrong?   Disclaimer: this question was inspired by this one  that did not attract much attention. It would be nice to have detailed answer to such question for reference.
92944
I frequently hear up- or down-sampling of data discussed as a way of dealing with classification of imbalanced data.  I understand that this could be useful if you're working with a binary (as opposed to probabilistic or score-based) classifier and treating it as a black box, so sampling schemes are your only way to tweak its position on the "ROC curve" (in quotes because if your classifier is inherently binary I guess it doesn't have a real ROC curve, but the same concept of trading off false positives and false negatives still applies).  But it seems like the same justification doesn't hold if you actually have access to some kind of score that you're later thresholding to make a decision. In this case, isn't up-sampling just an ad-hoc way of expressing a view about your desired trade-off between false positives and false negatives when you have much better tools available, like actual ROC analysis? It seems like it would be weird in this case to expect up-sampling or down-sampling to do anything but change your classifier's "prior" on each class (i.e. unconditional probability of being that class, the baseline prediction)--I wouldn't expect it to change the classifier's "odds ratio" (how much the classifier adjusts its baseline prediction based on the covariates).  So my question is: if you have a classifier that isn't a binary black box, are there any reasons to expect up- or down-sampling to have a much better effect than adjusting the threshold to your liking? Failing that, are there any empirical studies showing reasonably large effects for up- or down-sampling on reasonable performance metrics (e.g., not accuracy)?
86126
Is there any methodical way to compare the directions, magnitudes, etc of PCA results for different samples drawn from the same population?  I'm leaving the nature of the test deliberately vague because I'd like to hear all the various possibilities... e.g. there might be (and I'm speculating here) a test comparing the sizes of the first principal components, or a test comparing the directions of the principal components, or there's some kind of distance measure between PCA outcomes and a test statistic for their equality.  As far as a use case goes, I don't have on in mind. Just out of curiosity, maybe as an exploratory technique.
116527
I have read about the log-sum-exp trick in many places (e.g. , and ) but have never seen an example of how it is applied specifically to the Naive Bayes classifier (e.g. with discrete features and two classes)  How exactly would one avoid the problem of numerical underflow using this trick?
78311
What are principal component scores (PC scores, PCA scores)?
59890
How does a  work, and what differentiates it from other linear classifiers, such as the , , or ? *  (* I'm thinking in terms of the underlying motivations for the algorithm, optimisation strategies, generalisation capabilities, and run-time complexity)
7623
I have a dataset for which I have multiple sets of binary labels. For each set of labels, I train a classifier, evaluating it by cross-validation. I want to reduce dimensionality using principal component analysis (PCA). My question is:  Is it possible to do the PCA once for the whole dataset and then use the new dataset of lower dimensionality for cross-validation as described above? Or do I need to do a separate PCA for every training set (which would mean doing a separate PCA for every classifier and for every cross-validation fold)?  On one hand, the PCA does not make any use of the labels. On the other hand, it does use the test data to do the transformation, so I am afraid it could bias the results.  I should mention that in addition to saving me some work, doing the PCA once on the whole dataset would allow me to visualize the dataset for all label sets at once. If I have a different PCA for each label set, I would need to visualize each label set separately.
107838
In Decision Trees, when doing classification or regression, do we use only numerical values? Suppose I have a categorical column Wind as a feature. Suppose I am have 5 rows (observations), and the values for Wind are [high,low,high,medium,medium].  Can I feed this categorical data into a decision tree classifier (like scikit-learn), will it take the data? That is, will the splitting of node based on wind happen on categorical values high with count 2, medium with count 2 and low with count 1. Or, will the system converts it into dummy variable?
123279
The  for cross-validation says the following about using feature-scaling and cross-validation:      Just as it is important to test a predictor on data held-out from training, preprocessing (such as standardization, feature selection, etc.) and similar data transformations similarly should be learnt from a training set and applied to held-out data for prediction   I understand the reason behind this is to prevent information leakage between the training &amp; test sets during cross-validation, which could result in an optimistic estimate of model performance.  I am wondering then, if I wish to use Principal Component Analysis to reduce the size of a feature set before training say a regression model, and PCA requires feature-scaling to be effective, how do I chain feature-scaling to PCA to cross-validated regression, without introducing data leakage between the train-test splits in the cross-validation?
32123
Random forests are considered to be black boxes, but recently I was thinking what knowledge can be obtained from a random forest?  The most obvious thing is the importance of the variables, in the simplest variant it can be done just by calculating the number of occurrences of a variable. The second thing I was thinking about are interactions. I think that if the number of trees is sufficiently large then the number of occurrences of pairs of variables can be tested (something like chi square independence).  The third thing are nonlinearities of variables. My first idea was just to look at a chart of a variable  Vs score, but I'm not sure yet whether it makes any sense.    Added 23.01.2012 Motivation    I want to use this knowledge to improve a logit model. I think (or at least I hope) that it is possible to find interactions and nonlinearities that were overlooked.
15315
Principal component analysis (PCA) is usually explained via an eigen-decomposition of the covariance matrix. However, it can also be performed via singular value decomposition (SVD) of the data matrix $\mathbf X$. How does it work? What is the connection between these two approaches? What is the relationship between SVD and PCA?  Or in other words, how to use SVD of the data matrix to perform dimensionality reduction?
105346
I've come across a scenario where I have 10 signals/person for 10 people (so 100 samples) containing 14000 data points (dimensions) that I need to pass to a classifier. I would like to reduce the dimensionality of this data and PCA seems to be the way to do so. However, I've only been able to find examples of PCA where the number of samples is greater than the number of dimensions. I'm using a PCA application that finds the PCs using SVD. When I pass it my 100x14000 dataset there are 101 PCs returned so the vast majority of dimensions are obviously ignored. The program indicates the first 6 PCs contain 90% of the variance.   Is it a reasonable assumption that these 101 PCs contain essentially all the variance and the remaining dimensions are neglectable?  One of the papers I've read claims that, using a similar (though slightly lower quality) dataset than my own, they were able to reduce 4500 dimensions down to 80 retaining 96% of the original information. The paper hand-waves over the details of the PCA technique used, only 3100 samples were available, and I have reason to believe less samples than that were used to actually perform PCA (to remove bias from the classification phase).   Am I missing something or is this really the way that PCA is used with high dimensionality-low sample size dataset? Any feedback would be greatly appreciated.
27733
I have a data set in the form of (features, binary output 0 or 1), but 1 happens pretty rarely, so just by always predicting 0, I get accuracy between 70% and 90% (depending on the particular data I look at). The ML methods give me about the same accuracy, and I feel, there should be some standard methods to apply in this situation, that would improve the  accuracy over the obvious prediction rule.
28107
I am working on the Titanic dataset. So far my submission has 0.78 score using soft majority voting with logistic regression and random forest. As for the features, I used Pclass, Age, SibSp, Parch, Fare, Sex, Embarked.  My question is how to further boost the score for this classification problem?   One thing I tried is to add more classifiers for the majority voting, but it does not help, it even worthens the result. How do I understand this worthening effect?  Thanks for your insight.