I'm training a CNN model for binary classification. The obtained confusion matrix and some corresponding performance measures on validation set are as follows: ans =         11046         177          561        9306  accuracy:    0.965007 sensitivity: 0.943144 specificity: 0.984229 precision:   0.981335 recall:      0.943144 f_measure:   0.961860 gmean:       0.963467  It seems the results are good. But the training loss seems bad, as shown as follows:  Is this a good model?
4773
I've fitted my model and am trying to understand whether it's any good. I've calculated the recommended metrics to assess it ($R^2$/ AUC / accuracy / prediction error / etc) but do not know how to interpret them. In short, how do I tell if my model is any good based on the metric? Is an $R^2$ of 0.6 (for example) sufficient to let me proceed to draw inferences or base scientific/business decisions?     This question is intentionally broad, to cover a wide variety of situations that members frequently encounter; such questions could be closed as duplicates of this one. Edits to broaden the scope beyond the metrics mentioned here are welcome, as are additional answers - particularly those that offer insight about other classes of metrics.
49384
I'm using Python Keras package for neural network. This is the . Is batch_size equals to number of test samples? From Wikipedia we have  information:     However, in other cases, evaluating the sum-gradient may require   expensive evaluations of the gradients from all summand functions.   When the training set is enormous and no simple formulas exist,   evaluating the sums of gradients becomes very expensive, because   evaluating the gradient requires evaluating all the summand functions'   gradients. To economize on the computational cost at every iteration,   stochastic gradient descent samples a subset of summand functions at   every step. This is very effective in the case of large-scale machine   learning problems.   Above information is describing test data? Is this same as batch_size in keras (Number of samples per gradient update)?
60782
I have an imbalanced dataset with 88 positive samples and 128575 negative samples. I was reluctant to over/undersample the data since it's a biological dataset and I didn't want to introduce synthetic data. I built a Random Forest Classifier with this original dataset. I got an F1 score of 0 for the positive class. Zero precision. Zero recall. I cross-checked the predictions and test data. The model predicts some positives none of which are actually positive. Worst performance.  So, I tried to oversample the positive class. I upsampled the positives to 1000 samples. To my surprise, the F1 score for this dataset was 0.97, for the positive class. Then I tried lesser samples. I was able to achieve an F1 score of 0.83 with 200 positive samples, which is just 2.25 times of the original positive samples.  I would like to know why this occurs. For 88 samples, F1 score is 0.00 (rounded off to two digits). For 200 samples it's 0.83. There is no data leakage. All the features are engineered. I used imbalanced-learn module for oversampling. Can someone explain why is this difference in performance?
84538
I wonder how to compute precision and recall using a confusion matrix for a multi-class classification problem. Specifically, an observation can only be assigned to its most probable class / label. I would like to compute:    Precision = TP / (TP+FP)   Recall = TP / (TP+FN)   for each class, and then compute the micro-averaged F-measure.
98893
I have a trained logistic regression model that I am applying to a testing data set. The dependent variable is binary (boolean). For each sample in the testing data set, I apply the logistic regression model to generates a % probability that the dependent variable will be true. Then I record whether the acutal value was true or false. I'm trying to calculate an $R^2$ or Adjusted $R^2$ figure as in a linear regression model.  This gives me a record for each sample in the testing set like:  prob_value_is_true         acutal_value    .34                          0    .45                          1    .11                          0    .84                          0     ....                        ....             I am wondering how to test the accuracy of the model. My first attempt was to use a contingency table and say "if prob_value_is_true > 0.80, guess that the actual value is true" and then measure the ratio of correct to incorrect classifications. But I don't like that, because it feels more like I'm just evaluating the 0.80 as a boundary, not the accuracy of the model as a whole and at all prob_value_is_true values.  Then I tried to just look at each prob_value_is_true discrete value, as an example, looking at all samples where prob_value_is_true=0.34 and measuring the % of those samples where the acutal value is true (in this case, perfect accuracy would be if the % of samples that was true = 34%). I might create a model accuracy score by summing the difference at each discrete value of prob_value_is_true. But sample sizes are a huge concern here, especially for the extremes (nearing 0% or 100%), such that the averages of the acutal values are not accurate, so using them to measure the model accuracy doesn't seem right.  I even tried creating huge ranges to ensure sufficient sample sizes (0-.25, .25-.50, .50-.75, .75-1.0), but how to measure "goodness" of that % of actual value stumps me. Say all samples where prob_value_is_true is between 0.25 and 0.50 have an average acutal_value of 0.45. Is that good since its in the range? Bad since its not near 37.5% (the center of the range)?  So I'm stuck at what seems like should be an easy question, and hoping someone can point me to a resource or method to calculate an accuracy stastic for a logistic regression model.
27733
I have a data set in the form of (features, binary output 0 or 1), but 1 happens pretty rarely, so just by always predicting 0, I get accuracy between 70% and 90% (depending on the particular data I look at). The ML methods give me about the same accuracy, and I feel, there should be some standard methods to apply in this situation, that would improve the  accuracy over the obvious prediction rule.
50839
In a neural network class I'm taking the error measure is defined as:    If the purpose of squaring the difference of the predicted and target values is to always have a positive value, then why not just use the absolute value of the difference instead?  Secondly, why is the summation halved?
22064
Suppose my training data contains ~100 variables, and each example is tagged as "success" or "failure".  I understand how a neural network can be used to try and predict success vs failure based on the variables.  However I am interested in the neural network outputting the posterior probability rather than success or failure. In fact, I evaluate the efficacy of the NN based on how accurate the probabilities are (eg. AUROC over the entire dataset) rather than % of cases with correct prediction.  Are NN's the right tool for the job here and if so how do you structure the NN to output this?  (NOTE: I'm a ML newbie!)
43359
I am trying recently to optimize models but for some reason, whenever I try to run the optimization the model score in the end is worse than before, so I believe I do something wrong. in order to optimize my model I define param grid and than fit with the train data and then according to the results run again with nre parameters, e.g- #ROUND 1 param_grid={     'max_depth': [3,4,5],     'learning_rate':[0.1,0.01,0.05],     'gamma': [0,0.25,1.0],     'reg_lambda':[0,1.0,10.0],     'scale_pos_weight':[1,3,5] }  grid_search = GridSearchCV(estimator = clf_xgb, param_grid = param_grid,                            cv = 3, n_jobs = -1, verbose = 2) grid_search.fit(X_train,y_train) grid_search.best_params_  &gt;&gt;&gt;.....   (and now based on the result changing the params...) after this step I choose the best hyperparameters and run the model; clf_xgb=xgb.XGBClassifier(seed=42,                          objective='binary:logistic',                          gamma=0,                          learn_rate=0.7,                          max_depth=6,                          reg_lambda=0.8,                          scale_pos_weight=1,                          subsample=0.9,                          cilsample_bytree=0.5)  clf_xgb.fit(X_train,            y_train,            verbose=True,            early_stopping_rounds=10,            eval_metric='aucpr',            eval_set=[(X_test,y_test)])  The problem is that when I check the model score clf_xgb.score(X_test,y_test)  I always get lower score than what I got before the optimization which makes me suspect that I'm missing something in the way doing it/basic principle in this process. Is it possible that after running the optimization my score won't get better (and even worse?) ? Where is my mistake? Are there other parameters that could influence or improve my model?
117473
Heads up: I'm not sure if this is the best place to post this question, so let me know if there is somewhere better suited.  I am trying to train a simple neural network to learn a simple quadratic function of the form: $f(x) = 5 - 3x + 2x^2$  I set up a single-layered network with a single neuron. The input is a 2d array of the form $(x, x^2)$ and I don't use an activation function. I expect that the weights and biases I extract from the network will correspond to the coefficients in the function $f(x)$.  I randomly generate some training points and labels, as well as a validation data set, and train my model using the Keras sequential model called from TensorFlow.   import tensorflow as tf from tensorflow import keras import numpy as np import matplotlib.pyplot as plt  def fTest(x_arg):     return 5 - 3*x_arg + 2*(x_arg)**2   # training data t = np.random.choice(np.arange(-10,10, .01),5000 ) t1 = [] for i in range(len(t)):     t1.append([t[i], t[i]**2]) s = [] for i in range(len(t)):     s.append(fTest(t[i])) t1 = np.array(t1) s = np.array(s)  # validation set v = np.random.choice(np.arange(-10,10, .01),5000 ) v1 = [] for i in range(len(v)):     v1.append([v[i], v[i]**2]) u = [] for i in range(len(v)):     u.append(fTest(v[i])) v1 = np.array(v1) u = np.array(u)  model = keras.Sequential([     keras.layers.Dense(1, input_shape=(2,) , use_bias=True), ])  model.compile(optimizer='adam',                loss='mean_squared_logarithmic_error',               metrics=['mae','accuracy'])  model.fit(t1, s, batch_size=50, epochs=2000, validation_data=(v1,u))    The model seems to train, but very poorly. The 'accuracy' metric is also zero, which I am very confused about.   Epoch 2000/2000 200/200 [==============================] - 0s 23us/step - loss: 0.0018 - mean_absolute_error: 1.0144 - acc: 0.0000e+00 - val_loss: 0.0014 - val_mean_absolute_error: 1.0276 - val_acc: 0.0000e+00   Visually, the predictions of the model seem to be reasonably accurate    I've tried other loss-functions but none of them seem to work any better. I'm fairly new to to using TF/Keras so is there something obvious that I'm missing?  Edit: corrected training output
121830
I want to compare the classification performance of two heuristics: $h_0$ and $h_1$. I have around 700 samples for which I know the output class $c$ and the results of the two heuristics $c_{\textit{h_0}}, c_{\textit{h_1}}$.  That is, for each sample I have $\text{sample}_{\textit{id}}, c, c_{\textit{h_0}}, c_{\textit{h_1}}$.   $h_0$ has around ~71% correct predictions of the output class and $h_1$ has around ~77% correct. What I want to calculate now is whether this performance difference is just by chance or if the prediction performance of $h_1$ is statistical significant better than that of $h_0$.  I have found , but I don't know whether I can just put my data in there like the following matrix to calculate a p-value.  $$   \begin{matrix}                                &amp; h_0 &amp; h_1 \\     \text{correct prediction} &amp; 497 &amp; 539 \\     \text{wrong prediction}   &amp; 203 &amp; 161   \end{matrix} $$  I don't know if I can calculate the p-value like that or if I'm completely on the wrong track. I would really appreciate it if someone could point me in the right direction.
17688
I am beginning to learn how to use scikit-learn and I have a hard time choosing the right model.  Here is my dataset: I have 100 persons. Each person was measured three times: baseline, first event and second event. Each measurement had 100 different markers per person that range from 0.1 to 1000. Additionally I have outcome measurements of each event: outcome can be 0, 1 or 2. My task is to find just a few markers (let’s say 10) that can predict outcome with a good accuracy. If I am right it should be: Supervised learning/Classification problem. What model would be the best?  Thanks for your help!
3456
I followed this  using the Fashion MNIST dataset. The training set contains 60,000 28x28 pixels greyscale images, split into 10 classes (trouser, pullover, shoe, etc...). The tutorial uses a simple model:  model = keras.Sequential([     keras.layers.Flatten(input_shape=(28, 28)),     keras.layers.Dense(128, activation='relu'),     keras.layers.Dense(10) ])   This model reaches 91% accuracy after 10 epochs.  I am now practicing with another dataset called CIFAR-10, which consists of 50,000 32*32 pixels RGB images, also split into 10 classes (frog, horse, boat, etc...).   Considering that both the Fashion MNIST and CIFAR-10 datasets are pretty similar in terms of number of images and image size and that they have the same number of classes, I naively tried training a similar model, simply adjusting the input shape:    model = keras.Sequential([      keras.layers.Flatten(input_shape=(32, 32, 3)),      keras.layers.Dense(128, activation='relu'),      keras.layers.Dense(10)   ])   Alas, after 10 epochs, the model reaches an accuracy of 45%. What am I doing wrong?  I am aware that I have thrice as many samples in an RGB image than in a grayscale image, so I tried increasing the number of epochs as well as the size of the intermediate dense layer, but to no avail.    Below is my full code:  import tensorflow as tf import IPython.display as display from PIL import Image from tensorflow import keras import numpy as np import matplotlib.pyplot as plt import pdb import pathlib import os from tensorflow.keras import layers #Needed to make the model from tensorflow.keras import datasets, layers, models  (train_images, train_labels), (test_images, test_labels) = datasets.cifar10.load_data()  IMG_HEIGHT = 32 IMG_WIDTH = 32  class_names = ['airplane', 'automobile', 'bird', 'cat', 'deer',                'dog', 'frog', 'horse', 'ship', 'truck']   train_images = train_images / 255.0 test_images = test_images / 255.0  def make_model():       model = keras.Sequential([          keras.layers.Flatten(input_shape=(IMG_HEIGHT, IMG_WIDTH, 3)),          keras.layers.Dense(512, activation='relu'),          keras.layers.Dense(10)       ])       model.compile(optimizer='adam',                    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),                    metrics=['accuracy'])       return model  model=make_model() history = model.fit(train_images, train_labels, epochs=10)
21830
I am working on binary classification problem, I try to evaluate the performance of some classification algorithms (LR,Decission Tree , Random forest ...). I am using  a cross validation technique (to avoid over-fitting) with AUC ROC as scoring function to compare the performance of the algorithms, but I am getting a weird results with Random forest and AdbBoost, I have a perfect AUC_ROC score (i.e. =1) despite the fact that the recall(TPR) and FPR of this algorithms are different from 1 and 0 respectively .  def FPR(y_true, y_pred):     tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()     result = fp / (fp+tn)     return result def FNR(y_true, y_pred):     tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()     result = fn / (tp+fn)     return result FPR_scorer = make_scorer(FPR) FNR_scorer = make_scorer(FNR)     def get_CrossValResults2(model,cv_rst,bestIndx):     best=pd.DataFrame.from_dict(cv_rst).iloc[bestIndx]     roc=&quot;{:.12f}&quot;.format(best['mean_test_roc_auc'])     acc =&quot;{:.0%}&quot;.format(best['mean_test_accuracy'])     prec =&quot;{:.0%}&quot;.format(best['mean_test_precision'])     rec =&quot;{:.0%}&quot;.format( best['mean_test_recall'])     f1 =&quot;{:.0%}&quot;.format(best['mean_test_f1'])     r2=&quot;{:.2f}&quot;.format(best['mean_test_r2'])     g_mean=&quot;{:.2f}&quot;.format(best['mean_test_gmean'])     pr_auc=&quot;{:.8f}&quot;.format(best['mean_test_pr'])     fnr=&quot;{:.0%}&quot;.format(best['mean_test_fnr'])     fpr=&quot;{:.0%}&quot;.format(best['mean_test_fpr'])     rst = pd.DataFrame([[ model, acc,prec,rec,fpr,fnr,f1,roc,pr_auc,g_mean,r2]],columns = ['Model', 'Accuracy', 'Precision', 'Recall','FPR','FNR', 'F1-Score','ROC_auc','PR_auc','gmean','r2'])     return rst        cross_val_rst = pd.DataFrame(columns = ['Model', 'Accuracy', 'Precision', 'Recall','FPR','FNR', 'F1-Score','ROC_auc','PR_auc','gmean','r2'])              scoring = {'accuracy':'accuracy','recall':'recall','precision':'precision','fpr':FPR_scorer,'fnr':FNR_scorer,'f1':'f1' ,'roc_auc':'roc_auc','pr':'average_precision','gmean':Gmean_scorer,'r2':'r2'}     param_grid = {'n_estimators': [200],               'max_depth': [80,90],               'min_samples_leaf': [2,3, 4],             'min_samples_split': [2,5,12],             'criterion': [ 'gini'],              'class_weight' : [class_weights], 'n_jobs' : [-1]}  clf = GridSearchCV(RandomForestClassifier(class_weight=class_weights), param_grid, cv=kfold,scoring=scoring,refit=refit)#Fit the model bestmodel = clf.fit(X,Y)       cross_val_rst = cross_val_rst.append(get_CrossValResults2(model='Random Forrest',bestIndx=bestmodel.best_index_,cv_rst=bestmodel.cv_results_),ignore_index=True)
99247
Is there an existing methodology for applying a supervised learning model to a uncertain dataset? For example, say we have a dataset with classes A and B:  +----------+----------+-------+-----------+ | FeatureA | FeatureB | Label | Certainty | +----------+----------+-------+-----------+ |        2 |        3 | A     | 50%       | |        3 |        1 | B     | 80%       | |        1 |        1 | A     | 100%      | +----------+----------+-------+-----------+   How could we train a machine learning model on this? Thanks.
110157
I'm new to data mining and I'm trying to train a decision tree against a data set which is highly unbalanced. However, I'm having problems with poor predictive accuracy.  The data consists of students studying courses, and the class variable is the course status which has two values - Withdrawn or Current.   Age Ethnicity Gender Course  ... Course Status   In the data set there are many more instances which are Current than Withdrawn. Withdrawn instances only accounting for 2% of the total instances.  I want to be able to build a model which can predict the probability that a person will withdraw in the future. However when testing the model against the training data, the accuracy of the model is terrible.  I've had similar issues with decision trees where the data is dominated by one or two classes.  What approach can I use to solve this problem and build a more accurate classifier?
39830
In the last layer of  'CNNs'   it is common to use softmax  activation functions for multi-class classification.I would like to know if is it necessary using a softmax activation function when creating a CNN for image classification task, and does it have nothing to do with the optimizer used to train the model ?