We have a VGG16 network trained from scratch with a Sigmoid output function. We have 6 classes and the usual output looks like this:  scores': [6.494849458249519e-08, 1.8738395510808914e-06, 3.010111981893715e-07, 0.0, 0.0, 0.8633317947387695]   The problem is that the output value is very low in each class, I would like to have a normalized output that sums to 1.0 Thanks
113895
What decides the choice of function ( Softmax vs Sigmoid ) in a Logistic classifier ?  Suppose there are 4 output classes . Each of the above function gives the probabilities of each class being the correct output . So which one to take for a classifier ?
69474
Most of the advanced deep learning models like VGG, ResNet, etc. require square images as input, usually with a pixel size of $224x224$.  Is there a reason why the input has to be of equal shape, or can I build a convnet model with say $100x200$ as well (if I want to do facIAL recognition for example and I have portrait images)?  Is there increased benefit with a larger pixel size, say $512x512$?
25691
I have a handful of these strange normal maps I need to use in my scene that store X in the Alpha channel, Y in the green channel, and require you to calculate Z with the following formula:  Z = sqrt(1 - (X * X + Y * Y))  Red is always 100% and Blue is 0%, but these channels are not used.  Despite figuring this much out, I haven't been able to produce a node setup that can correctly convert this into the desired purpley tangent space normal map.  Any ideas?
117473
Heads up: I'm not sure if this is the best place to post this question, so let me know if there is somewhere better suited.  I am trying to train a simple neural network to learn a simple quadratic function of the form: $f(x) = 5 - 3x + 2x^2$  I set up a single-layered network with a single neuron. The input is a 2d array of the form $(x, x^2)$ and I don't use an activation function. I expect that the weights and biases I extract from the network will correspond to the coefficients in the function $f(x)$.  I randomly generate some training points and labels, as well as a validation data set, and train my model using the Keras sequential model called from TensorFlow.   import tensorflow as tf from tensorflow import keras import numpy as np import matplotlib.pyplot as plt  def fTest(x_arg):     return 5 - 3*x_arg + 2*(x_arg)**2   # training data t = np.random.choice(np.arange(-10,10, .01),5000 ) t1 = [] for i in range(len(t)):     t1.append([t[i], t[i]**2]) s = [] for i in range(len(t)):     s.append(fTest(t[i])) t1 = np.array(t1) s = np.array(s)  # validation set v = np.random.choice(np.arange(-10,10, .01),5000 ) v1 = [] for i in range(len(v)):     v1.append([v[i], v[i]**2]) u = [] for i in range(len(v)):     u.append(fTest(v[i])) v1 = np.array(v1) u = np.array(u)  model = keras.Sequential([     keras.layers.Dense(1, input_shape=(2,) , use_bias=True), ])  model.compile(optimizer='adam',                loss='mean_squared_logarithmic_error',               metrics=['mae','accuracy'])  model.fit(t1, s, batch_size=50, epochs=2000, validation_data=(v1,u))    The model seems to train, but very poorly. The 'accuracy' metric is also zero, which I am very confused about.   Epoch 2000/2000 200/200 [==============================] - 0s 23us/step - loss: 0.0018 - mean_absolute_error: 1.0144 - acc: 0.0000e+00 - val_loss: 0.0014 - val_mean_absolute_error: 1.0276 - val_acc: 0.0000e+00   Visually, the predictions of the model seem to be reasonably accurate    I've tried other loss-functions but none of them seem to work any better. I'm fairly new to to using TF/Keras so is there something obvious that I'm missing?  Edit: corrected training output
101892
I've got this problem where basically:  $$ Z = f(X) \sim Normal(0, 1) $$  $f$ is pretty non-linear and I don't have its inverse function. In practice (using Stan's MCMC samplers or Tensorflow's optimisers or by using literally any method), is it possible to draw samples from X?    My limited progress:  I don't really know how I'd specify such a model in Stan, or if it's even possible ($f$ is non-linear, almost certainly non-monotonic so I can't add a Jacobian adjustment and treat X as a parameter).  I dunno if treating X as a variable and constructing an objective that applies $f$ to X and optimises in the direction that makes the sample "more normal" is the way to go here. I don't think that such an approach would scale well, with increasing dimensionality of X and Z.  I'm looking for any other ideas on how to sample from X. Is it possible to use the KL divergence here somehow?
63630
Suppose weights of NN are Gaussian spread initialization then forward pass for all the inputs will evaluate to zero which computes to 0.69 ($-\log_{e}0.5 \approx 0.69 $, since sigmoid (0) = 0.5) average loss at least in the first pass. For three classes it would be $1.0986$ ($-\log_{e}\frac{1}{3}$) and so on.   I understand when weights are zero-initialized, but why does this happen when weights are Normally distributed to start with?
59099
I am working in a classification problem in which I use RBF with a single hidden layer. I want to use SoftMax activation function for the output layer. I already read some documents about the activation function of the output layer but can't find my answer. I try to write the equation for softmax and linear activation function and compare them. in my opinion yes they are different, but after I saw some documents I become confused so to become sure about I decide to ask it here.   is there any difference between linear and nonlinear activation function in a single hidden network?
69251
I have a classification problem where pixels will be labeled with soft labels (which denote probabilities) rather than hard 0,1 labels. Earlier with hard 0,1 pixel labeling the cross entropy loss function (sigmoidCross entropyLossLayer from Caffe) was giving decent results. Is it okay to use the sigmoid cross entropy loss layer (from Caffe) for this soft classification problem?
53361
I've been trying to perform a binary classification using an SVM classifier (scikit-learn's SVC with RBF kernel). I have a sample size of about 100, with about 70 features each. The features are of approximately the same order of magnitude in their raw form, and the values tend to be already distributed around the 0 (not always though). The distribution of two such features is shown in the histograms below.   I performed a Z-score transformation on all features, as I know this to be considered a good practice when working with multiple features in machine learning. The problem is that when I use the raw data, I always manage to get better accuracy than with the Z-scores (about 2-3%). Bear in mind that the parameters of the SVMs in each case are optimized using a grid-search, so I'm not using exactly identical classifiers.  Does this make sense, getting worse results with Z-scores? I would expect to get the same or better results. What could be the mathematical logic behind this?  Edit  To answer two frequent questions from comments and answers:   My classes are indeed distributed equally (exactly 50%/50%) I also use measures other than accuracy (AUC, F1, etc.), but having worked on this project for some time, accuracy correlates well with what I need.
92757
When performing the linear SVM classification, it is often helpful to  normalize the training data, for example by subtracting the mean and  dividing by the standard deviation, and afterwards scale the test data  with the mean and standard deviation of training data. Why this process  changes dramatically the classification performance?
437
I got a slight confusion on the  algorithm used in  (MLP).  The error is adjusted by the cost function. In backpropagation, we are trying to adjust the weight of the hidden layers. The output error I can understand, that is, e = d - y [Without the subscripts].  The questions are:   How does one get the error of hidden layer? How does one calculate it? If I backpropagate it, should I use it as a cost function of an adaptive filter or should I use a pointer (in C/C++) programming sense, to update the weight?
88160
Why is the de-facto standard sigmoid function, $\frac{1}{1+e^{-x}}$, so popular in (non-deep) neural-networks and logistic regression?   Why don't we use many of the other derivable functions, with faster computation time or slower decay (so vanishing gradient occurs less). Few examples are . One of my favorites with slow decay and fast calculation is $\frac{x}{1+|x|}$.  EDIT  The question is different to  as I'm only interested in the 'why' and only for the sigmoid.
81664
I'm aware that there are many ways of dealing with datasets where there is a strong class imbalance in the target variable: downsampling the more prevalent and less important class, over-weighting the valuable minority class, etc. Are there any loss functions that are ideally suited for these types of problems? I've heard of AUC, but AUC sounds like a means of evaluating a model once its training is complete (correct me if I'm wrong). Log loss and squared loss penalize a model's over confidence, but overconfidence and class imbalance seem unrelated. Is that true?
107088
I was wondering why binary crossentropy can be used as the loss function in autoencoders trained on (normalized) images, e.g.  or ? I know that binary crossentropy can be used in binray classification problems where the ground-truth labels (i.e. $y$) are either 0 or 1 and therefore when predictions (i.e. $p$) are correct, in both cases, the loss value would be zero:  $$ BCE(y,p) = -y.log(p) - (1-y).\log{(1-p)} $$  $$ BCE(0,0) = 0, BCE(1,1) = 0 $$  However, binary crossentropy does not have a value of zero when neither of its arguments are both zero or one, which is the case for an autoencoder with ground-truth labels in range $[0,1]$ (i.e. assuming the input data has been normalized in this range). I thought a regression loss function such as mean squared error or mean absolute error must be used instead, which have a value of zero when labels and predictions are the same. What am I missing here?
116263
I'm applying standard 0-1 scaling of features before SVM classification for financial data but the results are worse. This is the results before scaling      NORMAL DATA AVERAGE RESULTS       Profit           PF         avMC         avPP         avRC        totTP        totFP         PF&gt;1     algosnum           SS          SSl   4389060.90         6.85        -0.00        60.69         0.50        16086        10973            5            8            1            5   and this is after scaling  NORMAL DATA AVERAGE RESULTS       Profit           PF         avMC         avPP         avRC        totTP        totFP         PF&gt;1     algosnum           SS          SSl   2256204.80      2044.51        -0.07        52.53         0.46        14577        12220            4            8            1            5   Scaling is performed in 0-1 range, test data is scaled according to scaling factor of train data. From the above results you can see that precision went down (avPP) from 60.69 to 52.53, average Mathew Correlation Index from 0 to -0.07 number of true positives went down from 16086 to 14577 and number of false positives grown from 10973 to 12220. The presented result is an outcome of 80 classifications on different financial instruments data for 80 data sets 20000x200 so i think result is quite significant.  So my question is: In such situation how I should proceed? Shall I stick to scaling? Or maybe I should generate different data set to check if this behavior is consistent? What sort of analysis of my features I can make? My data set is a mix of binary and continuous features in different scales.
35720
I am reading the Pattern Recognition and Machine Learning book by Bishops. I have some difficulties proving the (6.21) equation. Can someone help me?   N.B: If someone has a better title for this question, i'm opened to it as i couldn't find the good words.
95495
I am trying to balance 324 features for a 10 class problems, but it just seem humanly impossible,   The problem i am studying consist of recognizing character , each observation consist of 324 features in the form pixels extracted from a 18x18 grid.  Should each pixel be seen as different  feature, or could one just turn the 324 feature into one feature, and thus easily compute the things -  Or would that increase error.   So as the title states. am I recognition class based on multiple features, class based on the pattern of one feature?  It should be said that the range of each pixels 0 -1.