Consider a classifier that, given an input vector ${\bf x}$ outputs both a prediction $y'$ whose accuracy ($a \in \{0, 1\}$) can be measured, as well as a predicted accuracy that corresponds to the predicted probability ($p \in [0, 1]$) that the prediction is correct. The accuracy is computed w.r.t. to the known target value $y$, i.e., $a = 1$ if $y = y'$ and $0$ otherwise.  Ideally, the predicted accuracy represents $p(y = y' | {\bf x})$ with $y' = f({\bf x})$ and $f$ the function computed by the classifier.  My question is the following: Given $n$ predictions of the form $(a_i, p_i)$, how can i measure how well the classifier predicted its accuracy?  I can think of two ways to tackle this issue, but I wonder if these approaches are correct and if there are other established approaches out there.   Binning: By sorting predictions based on their $p_i$ value and splitting them into bins $B_j$, the true accuracy can be estimated as: $a_j = \sum_{(a_i, p_i) \in B_j} \frac{a_i}{|B_j|}$ and for every bin the accuracy $a_j$ can be compared to $\sum_{(a_i, p_i) \in B_j} \frac{p_i}{|B_j|}$, the average predicted accuracy. However, quite some information would be lost if not many predictions are available. Squared distance: $\sum_{i = 1}^n \frac{(a_i - p_i)^2}{n}$. By measuring the average squared distance between the accuracy per prediction to the predicted accuracy, a relative measure can be computed that is minimal if the true accuracy per prediction matches the predicted accuracy. It is nice that this method forgoes binning and works by considering predictions standalone but the result is a bit hard to interpret.
82875
I do not study statistics but engineering, but this is a statistics question, and I hope you can lead me to what I need to learn to solve this problem.    I have this situation where I calculate probabilities of 1000's of things happening in like 30 days.  If in 30 days I see what actually happened, how can I test to see how accurately I predicted?  These calculations result in probabilities and in actual values (ft).  What is the method for doing this? Thanks, CP
27477
In machine learning, we let $X$ be a real-valued input vector and $Y$ be a real number output, with joint distribution $P(X,Y)$. We are looking for a function $f(X)$ for predicting $Y$ given the values of $X$. In most of machine learning problems, we give penalty for each time that the prediction is wrong. The loss function is defined as follows:  $L(Y,f(X))=(Y-f(X))^2$.   The expected square prediction error is given as:   $$EPE(f) = E(Y-f(X))^2 = \displaystyle\int [y-f(x)]^2P(dx,dy). $$    The book I'm reading says: By conditioning on $X$, we can write EPE as $EPE(f) = E_XE_{Y|X}((Y-f(X))^2|X)$, and eventually getting a solution that is $f(x) = E(Y|X=x)$.   I'm not sure I follow what happened in these steps. Please help me with the derivation.  Thanks
57682
How to check classification accuracy for principal component analysis (PCA)? In order to compare it with different methods, for example with random forest classification, how can I compute accuracy?  E.g. for random forest the accuracy can be computed as follows:   pred1 &lt;- predict(iris.rf,x,type="response") pr&lt;-table(pred1,testset[,1]) confusionMatrix(pr)   But what about PCA?
41967
In R, I'm running a linear model of the form:  fit &lt;- lm(cbind(Y1,Y2,Y2) ~ X1 + X2 + X3, data=regression_table);   Then I'm using predict to generate predictions;  predicted_vals &lt;- predict(fit, test_data);   Does any one have suggestions on how I should go about evaluating the training error of this multivariate model? I understand the notion of cross validation for univariate multiple models, though I'm not sure how to go about go about doing something similar for a multivariate model.
21830
I am working on binary classification problem, I try to evaluate the performance of some classification algorithms (LR,Decission Tree , Random forest ...). I am using  a cross validation technique (to avoid over-fitting) with AUC ROC as scoring function to compare the performance of the algorithms, but I am getting a weird results with Random forest and AdbBoost, I have a perfect AUC_ROC score (i.e. =1) despite the fact that the recall(TPR) and FPR of this algorithms are different from 1 and 0 respectively .  def FPR(y_true, y_pred):     tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()     result = fp / (fp+tn)     return result def FNR(y_true, y_pred):     tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()     result = fn / (tp+fn)     return result FPR_scorer = make_scorer(FPR) FNR_scorer = make_scorer(FNR)     def get_CrossValResults2(model,cv_rst,bestIndx):     best=pd.DataFrame.from_dict(cv_rst).iloc[bestIndx]     roc=&quot;{:.12f}&quot;.format(best['mean_test_roc_auc'])     acc =&quot;{:.0%}&quot;.format(best['mean_test_accuracy'])     prec =&quot;{:.0%}&quot;.format(best['mean_test_precision'])     rec =&quot;{:.0%}&quot;.format( best['mean_test_recall'])     f1 =&quot;{:.0%}&quot;.format(best['mean_test_f1'])     r2=&quot;{:.2f}&quot;.format(best['mean_test_r2'])     g_mean=&quot;{:.2f}&quot;.format(best['mean_test_gmean'])     pr_auc=&quot;{:.8f}&quot;.format(best['mean_test_pr'])     fnr=&quot;{:.0%}&quot;.format(best['mean_test_fnr'])     fpr=&quot;{:.0%}&quot;.format(best['mean_test_fpr'])     rst = pd.DataFrame([[ model, acc,prec,rec,fpr,fnr,f1,roc,pr_auc,g_mean,r2]],columns = ['Model', 'Accuracy', 'Precision', 'Recall','FPR','FNR', 'F1-Score','ROC_auc','PR_auc','gmean','r2'])     return rst        cross_val_rst = pd.DataFrame(columns = ['Model', 'Accuracy', 'Precision', 'Recall','FPR','FNR', 'F1-Score','ROC_auc','PR_auc','gmean','r2'])              scoring = {'accuracy':'accuracy','recall':'recall','precision':'precision','fpr':FPR_scorer,'fnr':FNR_scorer,'f1':'f1' ,'roc_auc':'roc_auc','pr':'average_precision','gmean':Gmean_scorer,'r2':'r2'}     param_grid = {'n_estimators': [200],               'max_depth': [80,90],               'min_samples_leaf': [2,3, 4],             'min_samples_split': [2,5,12],             'criterion': [ 'gini'],              'class_weight' : [class_weights], 'n_jobs' : [-1]}  clf = GridSearchCV(RandomForestClassifier(class_weight=class_weights), param_grid, cv=kfold,scoring=scoring,refit=refit)#Fit the model bestmodel = clf.fit(X,Y)       cross_val_rst = cross_val_rst.append(get_CrossValResults2(model='Random Forrest',bestIndx=bestmodel.best_index_,cv_rst=bestmodel.cv_results_),ignore_index=True)
53361
I've been trying to perform a binary classification using an SVM classifier (scikit-learn's SVC with RBF kernel). I have a sample size of about 100, with about 70 features each. The features are of approximately the same order of magnitude in their raw form, and the values tend to be already distributed around the 0 (not always though). The distribution of two such features is shown in the histograms below.   I performed a Z-score transformation on all features, as I know this to be considered a good practice when working with multiple features in machine learning. The problem is that when I use the raw data, I always manage to get better accuracy than with the Z-scores (about 2-3%). Bear in mind that the parameters of the SVMs in each case are optimized using a grid-search, so I'm not using exactly identical classifiers.  Does this make sense, getting worse results with Z-scores? I would expect to get the same or better results. What could be the mathematical logic behind this?  Edit  To answer two frequent questions from comments and answers:   My classes are indeed distributed equally (exactly 50%/50%) I also use measures other than accuracy (AUC, F1, etc.), but having worked on this project for some time, accuracy correlates well with what I need.
104906
I have a training set with about 3000 positive instances and 3000 negative instances. But my test data set is pretty much un-balanced. The positive set only has 50 instances and negative has 1500 instances.This causes the precision very low. Are there any approaches to solve this problem? I use SVM to build classifier.
117577
I have a dataset of 140000 examples and 30 features for which I am training several classifiers for a binary classification (SVM, Logistic Regression, Random Forest etc)  In many cases hyperparameter tuning on the whole dataset using either Grid or Random search is too costly time-wise.  I started using the following technique   Sub sample my dataset Use the obtained fraction to tune the hyperparameters on Use the obtained parameters to train a model using the whole dataset   To evaluate each set of parameters on the second step I use sklearn's GridSearchCV with cv=10. To evaluate the final model that I create in the third step I use sklearn's cross_val_predict. In that sense I evaluate my models leaving a 10% percent of data out, I train on the rest and measure the predictive accuracy on the 10%, iteratively 10 times, then taking the average of the scores.  What made me worry is that the prediction accuracy I get from training on my whole dataset, is really close to the evaluation I get when tuning the parameters for the best set of parameters (each tested set of parameters outputs a score obtained from averaging 10-fold-cross validation results).  Most of the times the accuracy that cross_val_predict measured using all the training examples (whole dataset) is a little bit above what the evaluation of the best parameters returned.  To illustrate this here is the evaluation of a set of parameters (on a smaller dataset than what I described above but the effect is the same)  Best parameters set found on development set: {'kernel': 'rbf', 'C': 9, 'gamma': 0.1} Scores for all sets of parameters 0.851 (+/-0.006) for {'kernel': 'rbf', 'C': 3, 'gamma': 0.5} 0.852 (+/-0.006) for {'kernel': 'rbf', 'C': 3, 'gamma': 0.1} 0.829 (+/-0.006) for {'kernel': 'rbf', 'C': 3, 'gamma': 0.001} 0.853 (+/-0.006) for {'kernel': 'rbf', 'C': 9, 'gamma': 0.1} ...   And here are the averaged scores (from cross_val_predict) I got from training on my whole dataset using the best parameters  precision    recall  f1-score   support        0       0.86      0.85      0.86     15417       1       0.86      0.87      0.87     16561  avg / total       0.86      0.86      0.86     31978  acc score: 0.863750078179 roc au score: 0.863370490059 [[13147  2270]  [ 2087 14474]]   As you can see training on the whole dataset improves the results. I have also validated that badly tuned model (e.g. using the default values or random values for C and gamma) leads to much worse prediction accuracy.   Overall I think that tuning the hyperparameters on a subset is not ideal but can potentially lead to relatively good results without having to wait too long. I for example before using that approach used optunity package for tuning the hyperparameter on the whole dataset. This procedure would take 3-5 days to complete and would produce results that either had really good precision or really good recall but not both, so although for each class either the precision or the recall was really high (higher than what any of my other classifiers had achieved) the f1 meassure was really low. In the contrary using the later approach leads to some hours of training and a better f1 meassure.   My concerns are:  Do I limit my classification accuracy? Do I avoid using all the prediction power that my dataset can offer by tuning only on a subset? If such a harm of performance is happening is it somehow limited by some factor?
64162
My stats professor assigned this problem: "Show that the expected prediction error (EPE) for a squared error loss when $Y=f(X)+\varepsilon$ with estimator $\hat{f}(x)$ assuming $X=x$ is fixed and $\varepsilon~(0,\sigma^2)$ can be written as a combination of the bias and variance. In other words, show that   $$EPE(x) = E[(Y-\hat{f}(x))^2] = \sigma^2 + \text{Bias}^2 + \text{Var}(\hat{f}(x)).$$  I've come up with 3 ways to derive this relationship, but all of them depend on the assumption that $Y=f(X)+\varepsilon$ and $\hat{f}(X)$ are independent, or at least that their covariance is zero. For example, this allows me to use $\text{Var}[Y-\hat{f}(X)] = \text{Var}(Y)+\text{Var}(\hat{f}(X))$. This assumption makes intuitive sense to me because there is no necessary connection between $Y$ and the estimated value $\hat{f}(X)$ (the estimator could be a random number generator, after all). But I'm struggling to justify the independence assumption in a rigorous way. Can somebody nudge me toward understanding?
41987
I am studying statistics and I am wondering when it comes to standard error or a sampling if the calculation changes when there are weights added.   I have a weighted mean:  $$\mu_{w} = \dfrac{\sum_{i=1}^Nw_ix_i}{\sum_{i=1}^Nw_i}$$ and a weighted variance calculated by:  $$s^2_{w}=\dfrac{\sum_{i=1}^Nw_i}{(\sum_{i=1}^Nw_i)^2-\sum_{i=1}^Nw_i^2}\cdot \sum_{i=1}^N(x_i-\mu)^2$$  is the sampling error still calculated as  $$\text{SE}=\sqrt{\dfrac{s^2_{w}}{n}}$$
121542
I trained a model with some algorithms like random forest, logistic regression and so on.   My dataset was split into 80% CV train data (so actually 60% of data to train the model and 20 % for testing with cv). I built my model and now I'm using the last 20% (which I didn't used the whole time) but I'm wondering why is the predictive accuracy better with the validation set than with the training set?    specificity validation: 0.62962963 sensitivity validation: 0.94761905   on the left is the mean, on the right is the standard deviation (+/-) with a 10 times cross validation    specificity_train: 0.55  (+/- 0.10)  recall_train: 0.94   (+/- 0.02)   Did I something wrong? Thought this is the normal / best way to test, if the model is overfitted or not.  Or is it OK due to the standard deviation?
94266
I have a data set on predicting solar power generation,the dataset has 20 independent var and 1 dependent. The accuracy of my model is stuck at 60%. I have tried several models but this accuracy is best I could get other sucks even more. Here is my code: data_path = r'drive/My Drive/Proj/S.P.F./solarpowergeneration.csv' dts = pd.read_csv('solarpowergeneration.csv') dts.head() X = dts.iloc[:, :-1].values y = dts.iloc[:, -1].values print(X.shape, y.shape) y = np.reshape(y, (-1,1)) y.shape from sklearn.model_selection import train_test_split X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42) from sklearn.preprocessing import StandardScaler sc= StandardScaler() X_train = sc.fit_transform(X_train) X_test = sc.transform(X_test) y_train = sc.fit_transform(y_train) y_test = sc.transform(y_test)  import keras.backend as K   def calc_accu(y_true, y_pred):   return K.mean(K.equal(K.round(y_true), K.round(y_pred)))  def get_spfnet():   ann = tf.keras.models.Sequential()    ann.add(Dense(X_train.shape[1], activation='relu')) # ann.add(BatchNormalization())   ann.add(Dropout(0.3))   ann.add(Dense(32, activation='relu', kernel_regularizer=regularizers.l2(0.01)))  # ann.add(BatchNormalization())   ann.add(Dropout(0.3))   ann.add(Dense(32, activation='relu', kernel_regularizer=regularizers.l2(0.01)))  # ann.add(BatchNormalization())   ann.add(Dropout(0.3))    ann.add(Dense(1))   ann.compile(loss='mse',               optimizer='adam',               metrics=[tf.keras.metrics.RootMeanSquaredError(), calc_accu])   return ann  spfnet = get_spfnet() #spfnet.summary() hist = spfnet.fit(X_train, y_train, batch_size=32, epochs=250, verbose=2)  the accuracy and loss graphs are plt.plot(hist.history['calc_accu']) plt.title('Model Accuracy') plt.xlabel('Epochs') plt.ylabel('Accuracy') plt.show() plt.plot(hist.history['root_mean_squared_error']) plt.title('Model error') plt.xlabel('Epochs') plt.ylabel('error') plt.show()       after 50 epochs nothing seems to improve, neither curve seems to overfit on the data I tried other models like reducing layers and removing kernel regularizes, using kernel_initlizers='normal' and 'he-normal'    but they perform poorly stuck at 20%. dataset:
16814
Apparently,     the Fisher analysis aims at simultaneously maximising the   between-class separation, while minimising the within-class   dispersion. A useful measure of the discrimination power of a variable   is hence given by the diagonal quantity: $B_{ii}/W_{ii}$.     I understand that the size (p x p) of the Between (B) and Within-Class (W) matrices are given by the number of input variables, p. Given this, how can $B_{ii}/W_{ii}$ be a "useful measure of the discrimination power" of a single variable? At least two variables are required to construct the matrices B and W, so the respective traces would represent more than one variable.  Update: Am I right in thinking that $B_{ii}/W_{ii}$ is not a trace over a trace, where the sum is implied, but the matrix element $B_{ii}$ divided by $W_{ii}$? Currently that is the only way I can reconcile the expression with the concept.
85678
I've been wondering for a while now if there's any deep mathematical or statistical significance to finding the line that minimizes the square of the errors between the line and the data points.   If we use a less common method like LAD, where we just consider the absolute deviation, then outliers make less difference to the final model, while if we take the cube of the error (or any other power higher than 2), then outliers are far more significant than with the least squares model.  I suppose what I'm really asking is mathematically, is raising the error to the power of 2 really that special. Is it say more "accurate" in some sense than raising the error to the power of 1.95 or 2.05???   Thanks!
44026
I just finished reading  discussion. They argue that PR AUC is better than ROC AUC on imbalanced dataset.   For example, we have 10 samples in test dataset. 9 samples are positive and 1 is negative. We have a terrible model which predicts everything positive. Thus, we will have a metric that TP = 9, FP = 1, TN = 0, FN = 0.  Then, Precision = 0.9, Recall = 1.0. The precision and recall are both very high, but we have a poor classifier.  On the other hand, TPR = TP/(TP+FN) = 1.0, FPR = FP/(FP+TN) = 1.0. Because the FPR is very high, we can identify that this is not a good classifier.  Clearly, ROC is better than PR on imbalanced datasets. Can somebody explain why PR is better?
55014
I want to do an , then cross-validate model prediction accuracy with 80% data for training and 20% for validation, and calculate  for predictions.  Consider this dataset:    X     Y ----------  2.3    1  3.1    2  3.5    2  10.0   5  6.8    4  5.0    3  5.4    2  3.2    1   I did this:  x=c(2.3,3.1,3.5,10.0,6.8,5.0,5.4,3.2) y=c(1,2,2,5,4,3,2,1) myData=data.frame(cbind(x,y))  library("MASS") reg=polr(as.factor(myData$y)~myData$x,data=myData,method="probit")   I saw , but I couldn't fully understand. Suppose myValidationData contains 20% of data which I want to use for validation. So, I would do:  fit=predict(reg,type="probs") x=c(5.6, 5.1) y=c(3,3) myValidationData=data.frame(cbind(x,y))   This is how I tried to predict, but is it correct, when I want to cross-validate?  fit=predict(reg,data=myValidationData,type="probs")   How should I measure RMSE? And, how can I plot the prediction?
77854
I'm studying 'Introduction to Statistical Learning' by James, Witten, Hastie, Tibshirani.   In page 139, of their book, they began by introducing Bayes' Theorem $p_k(X)=P(Y=k|X=x) = \dfrac{\pi_kf_k(x)}{\sum_{l=1}^k \pi_l f_l(x)}$. $\pi$ is not mathematical constant, but denotes the prior probability. Nothing is strange in this equation.   The book claims that it wants to obtain an estimate for $f_k(x)$ that it can plug into the equation given above. To estimate $f_k(x)$, it assumes that is normal. In the one dimensional setting, $f_k(x)=\dfrac{1}{\sqrt{2\pi}\sigma}\exp(-\dfrac{1}{2\sigma^2}(x-\mu_k)^2)$, where $\mu_k$ and $\sigma^2_k$ are the mean and variance for the $k$th class. It is assumed that $\sigma^2_1 = \sigma^2_2 = \cdots = \sigma^2_K$. (I began to get confused from the last statement.)   Plugging $f_k$ into $p_x$, you have this quite messy equation (1) :   $$p_x(k)=\dfrac{\pi_k \frac{1}{\sqrt{2\pi}\sigma}\exp(-\frac{1}{2\sigma^2}(x-\mu_k)^2)}{\sum_{l=1}^K \pi_l \frac{1}{\sqrt{2\pi}\sigma}\exp(-\frac{1}{2\sigma^2}(x-\mu_l)^2)}. $$  Again, no surprises here since it is just substitution.     The Bayes' Classifier involves assigning an observation to the class for which equation (1) is the largest. Taking the log of Equation (1) and rearranging terms, it is not hard to show that this is equivalent to assigning the observation to the class for which the following is the largest:  $$\delta_k(x)=x \cdot \dfrac{\mu_k}{\sigma^2} - \dfrac{\mu_k^2}{2\sigma^2} + \log(\pi_k) $$  Question: I do not understand where this came from, and what it means. I tried doing the log of equation and it doesn't become this. Are we taking the derivative somewhere here, since this is the largest observation?