I have been trying to set up a ConvNet to classify some data. This data should be classified to either 1 (being what I need to get from the image) and 0 for everything that is irrelevant. I have successfully extracted 50k samples (positive) but I am having a hard time of getting negative samples. What would happen if I trained my net with 15k positive samples and lets say 5k negative ? I have read that this could be a problem for statistical algorithms ... is that also relevant to convolutional neural networks ?
124238
I deal with a fraud detection (credit-scoring-like) problem. As such there is a highly imbalanced relation between fraudulent and non-fraudulent observations.   provides a great overview of different classification metrics. Precision and Recall or kappa both seem to be a good choice:     One way to justify the results of such classifiers is by comparing them to those of baseline classifiers and showing that they are indeed better than random chance predictions.    As far as I understand, kappa could be the slightly better choice here, as random chance is taken into account. From  I understand that kappa deals with the concept of information gain:     [...] an Observed Accuracy of 80% is a lot less impressive with an Expected Accuracy of 75% versus an Expected Accuracy of 50% [...]   Therefore, my questions would be:    Is it correct to assume kappa to be a better-suited classification metric for this problem?  Does simply using kappa prevent the negative effects of imbalance on the classification algorithm? Is re-(down/up)-sampling or cost-based learning (see ) still required?
95495
I am trying to balance 324 features for a 10 class problems, but it just seem humanly impossible,   The problem i am studying consist of recognizing character , each observation consist of 324 features in the form pixels extracted from a 18x18 grid.  Should each pixel be seen as different  feature, or could one just turn the 324 feature into one feature, and thus easily compute the things -  Or would that increase error.   So as the title states. am I recognition class based on multiple features, class based on the pattern of one feature?  It should be said that the range of each pixels 0 -1.
104906
I have a training set with about 3000 positive instances and 3000 negative instances. But my test data set is pretty much un-balanced. The positive set only has 50 instances and negative has 1500 instances.This causes the precision very low. Are there any approaches to solve this problem? I use SVM to build classifier.
111848
I always read in books that when we do classification or machine learning tasks it's always better to normalize the features so to make them in one range like 0-1. Today I used weka to play with Iris dataset. First I just built a J48 classifier without normalizing the values, and the it made perfect performance. However when I normalized all the features to be in the range 0-1, the classifier made so much mistakes. Why is that? Shouldn't normalization be used always?
92944
I frequently hear up- or down-sampling of data discussed as a way of dealing with classification of imbalanced data.  I understand that this could be useful if you're working with a binary (as opposed to probabilistic or score-based) classifier and treating it as a black box, so sampling schemes are your only way to tweak its position on the "ROC curve" (in quotes because if your classifier is inherently binary I guess it doesn't have a real ROC curve, but the same concept of trading off false positives and false negatives still applies).  But it seems like the same justification doesn't hold if you actually have access to some kind of score that you're later thresholding to make a decision. In this case, isn't up-sampling just an ad-hoc way of expressing a view about your desired trade-off between false positives and false negatives when you have much better tools available, like actual ROC analysis? It seems like it would be weird in this case to expect up-sampling or down-sampling to do anything but change your classifier's "prior" on each class (i.e. unconditional probability of being that class, the baseline prediction)--I wouldn't expect it to change the classifier's "odds ratio" (how much the classifier adjusts its baseline prediction based on the covariates).  So my question is: if you have a classifier that isn't a binary black box, are there any reasons to expect up- or down-sampling to have a much better effect than adjusting the threshold to your liking? Failing that, are there any empirical studies showing reasonably large effects for up- or down-sampling on reasonable performance metrics (e.g., not accuracy)?
72915
Suppose I build a neural network for classification. The last layer is a dense layer with Softmax activation. I have five different classes to classify. Suppose for a single training example, the true label is [1 0 0 0 0] while the predictions be [0.1 0.5 0.1 0.1 0.2]. How would I calculate the cross entropy loss for this example?
16634
I'm training a neural network and the training loss decreases, but the validation loss doesn't, or it decreases much less than what I would expect, based on references or experiments with very similar architectures and data. How can I fix this?    As for question        to which this question is inspired, the question is intentionally left general so that other questions about how to reduce the generalization error of a neural network down to a level which has been proved to be attainable, can be closed as duplicates of this one.  See also dedicated thread on Meta:
6212
Assume I want to do binary classification (something belongs to class A or class B). There are some possibilities to do this in the output layer of a neural network:   Use 1 output node. Output 0 (&lt;0.5) is considered class A and 1 (>=0.5) is considered class B (in case of sigmoid) Use 2 output nodes. The input belongs to the class of the node with the highest value/probability (argmax).    Are there any papers written which (also) discuss this? What are specific keywords to search on?  This question is already asked before on this site  with no real answers. I need to make a choice (Master Thesis), so I want to get insight in the pro/cons/limitations of each solution.
86140
I have a very unbalanced sample set, e.g. 99% true and 1% false.   Is it reasonable to select a balanced subset with a 50/50-distribution for neural network training? The reason for this is, that I guess training on the original data set may induce a bias on the true-samples.  Can you suggest me some literature that covers this topic especially for neural networks?
81294
I'm implementing a neural network with Keras, but the Sequential model returns nan as loss value. I have sigmoid activation function in the output layer to squeeze output between 0 and 1, but maybe doesn't work properly.  This is the code:  def data_generator(batch_count, training_dataset, training_dataset_labels):   while True:     start_range = 0     for batch in batch_count:       end_range = (start_range + batch[1])       batch_dataset = training_dataset[start_range:end_range]       batch_labels = training_dataset_labels[start_range:end_range]       start_range = end_range       yield batch_dataset, batch_dataset  mlp = keras.models.Sequential()  # add input layer mlp.add(     keras.layers.Input(         shape = (training_dataset.shape[1], )     ) ) # add hidden layer mlp.add(     keras.layers.Dense(         units=training_dataset.shape[1] + 10,         input_shape = (training_dataset.shape[1] + 10,),         kernel_initializer='random_uniform',         bias_initializer='zeros',         activation='relu')     ) # add output layer mlp.add(     keras.layers.Dense(         units=1,         input_shape = (1, ),         kernel_initializer='glorot_uniform',         bias_initializer='zeros',         activation='sigmoid')     )  print('Compiling model...\n')  mlp.compile(     optimizer='adam',     loss=listnet_loss )  mlp.summary() # print model settings   # Training with tf.device('/GPU:0'):   print('Start training')   #mlp.fit(training_dataset, training_dataset_labels, epochs=50, verbose=2, batch_size=3, workers=10)   mlp.fit_generator(data_generator(groups_id_count, training_dataset, training_dataset_labels),                     steps_per_epoch=len(training_dataset), epochs=50, verbose=2, workers=10, use_multiprocessing=True)   How can I do?
85814
When training a neural network, what difference does it make to set:   batch size to $a$ and number of iterations to $b$ vs. batch size to $c$ and number of iterations to $d$   where $ ab = cd $?  To put it otherwise, assuming that we train the neural network with the same amount of training examples, how to set the optimal batch size and number of iterations? (where batch size * number of iterations = number of training examples shown to the neural network, with the same training example being potentially shown several times)  I am aware that the higher the batch size, the more memory space one needs, and it often makes computations faster. But in terms of performance of the trained network, what difference does it make?
76036
I have to deal with a text classification problem. A web crawler crawls webpages of a certain domain and for each webpage I want to find out whether it belongs to only one specific class or not. That is, if I call this class Positive, each crawled webpage belongs either to class Positive or to class Non-Positive.  I already have a large training set of webpages for class Positive. But how to create a training set for class Non-Positive which is as representative as possible? I mean, I could basically use each and everything for that class. Can I just collect some arbitrary pages that definitely do not belong to class Positive? I'm sure the performance of a text classification algorithm (I prefer to make use of a Naive Bayes algorithm) highly depends on which webpages I choose for class Non-Positive.  So what shall I do? Can somebody please give me an advice? Thank you very much!
30589
I used sklearn to compute roc_auc_score for a dataset of 72 instances. The accuracy was at 97% (2 misclassifications), but the ROC AUC score was 1.0. How is this possible? I would think that even one misclassification should have dropped the score to slightly below 1.0.  # Python 3.6.4 # numpy==1.14.3 # scikit-learn==0.19.1 # scipy==1.1.0  from sklearn import metrics import numpy as np  y_true = np.array([0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0]) y_prob = np.array([0.0, 0.1, 0.0, 0.0, 0.0, 0.0, 0.1, 0.0, 0.1, 0.0, 0.0, 0.0, 0.0, 0.7, 0.5, 0.0, 0.0, 0.0, 0.0, 0.0, 0.1, 0.1, 0.0, 0.0, 0.9, 0.0, 0.0, 0.0, 0.0, 0.1, 0.0, 0.0, 0.0, 0.1, 0.1, 0.0, 0.0, 0.7, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.1, 0.0, 0.0, 0.0, 0.0, 0.1, 0.0, 0.0, 0.1, 0.1, 0.0, 0.9, 0.0, 0.0, 0.4, 0.0, 0.0, 0.0])  # Show which actuals do not match their expected probabilities for index, (actual, predicted_prob) in enumerate(zip(y_true, y_prob)):   if (actual == 1 and predicted_prob &lt;= 0.5) or (actual == 0 and predicted_prob &gt; 0.5):     print (f'Mismatch at index {index}. Actual={actual}, predicted_prob={predicted_prob}')   rocauc = metrics.roc_auc_score(y_true, y_prob)     print (f'ROCAUC: {rocauc}')  # Outputs: # Mismatch at index 14. Actual=1.0, predicted_prob=0.5 # Mismatch at index 68. Actual=1.0, predicted_prob=0.4 # ROCAUC: 1.0   Then I debugged the score computation itself and looked at the coordinate ROC output.  # In sklearn/metrics/ranking.py, line 271: --&gt; 271         fpr, tpr, tresholds = roc_curve(y_true, y_score,     272                                         sample_weight=sample_weight)  ipdb&gt; fpr array([0.        , 0.        , 0.        , 0.18181818, 1.        ]) ipdb&gt; tpr array([0.33333333, 0.66666667, 1.        , 1.        , 1.        ]) ipdb&gt; tresholds array([0.9, 0.7, 0.4, 0.1, 0. ])  # coords = [] # for x, y in zip(fpr, tpr): #   coords.append((x, y))  ipdb&gt; pp coords [(0.0, 0.3333333333333333),  (0.0, 0.6666666666666666),  (0.0, 1.0),  (0.18181818181818182, 1.0),  (1.0, 1.0)]   All of those coordinates are on x=0 or y=1, meaning ROCAUC is showing 1.0. The only plausible explanation I can think of is that if the alg had more fpr/tpr points, it would show a very tight curve that never reaches x,y = (0, 1), and the ROCAUC would be close to 1, but not exactly 1. Is that a reasonable interpretation or am I missing something?
88722
I found two questions  and  about this issue but there is no obvious answer or explanation yet.I enforce the same problem where the validation error is less than training error in my Convolution Neural Network. What does that mean?
55894
I am performing a classification task, to try to detect an object. A picture of the environment is taken, candidates are generated of this possible object using vision algorithms, and once isolated, these candidates will be passed through a CNN for the final decision on whether the object has been detected or not. I am attempting to use transfer learning on InceptionV3 but am having difficulty training it, as I only have one set/class of images. The dilemma is that I only have one class of data and when I pass it through the network, I get a 100% accuracy (because there is nothing to compare it to). How should I overcome this? Should I find more categories online to add to my dataset? What should these categories be? Just to clarify, as an example, I have class &quot;cat&quot;. Not &quot;cat&quot; and &quot;dog&quot;. Not &quot;cat&quot; and &quot;no cat&quot;. Just &quot;cat&quot;. That is what my dataset consists of at the moment.
56697
What is your opinion about oversampling in classification in general, and the SMOTE algorithm in particular? Why would we not just apply a cost/penalty to adjust for imbalance in class data and any unbalanced cost of errors? For my purposes, accuracy of prediction to a future set of experimental units is the ultimate measure.  For reference, the SMOTE paper:
48866
I have a classification problem which is multi-label with N labels. I would like to know which method would be the better choice? Training N classifiers (1 for each label) or a single classifier which use K-hot encoding. I'm using a neural network for this with a softmax at the last layer. (Is there any chance that the softmax would cause a problem?) I'm using tensorflow for the model.