I am relatively new to Deep Learning and wanted to implement a Variational Autoencoder for images. For data preprocessing I only rescaled the pixel values from the range of [0..255] to [0...1]. The training looked good. The reconstructions of the VAE were pretty good and the loss-curves also looked good. Now I read that many people tell you to normalize the data, so that the image data for example have the mean = 0 and std = 1. Is this really necessary for training and crucially changes the results from training or is rescaling the images enough?
104288
I'm learning Neural Networks and I grasped the algebra behind them. I'm now interested in understanding how normalization and centering of the input data affect them. In my personal learning project (Regression with NN) I've transformed my input variables to a range between 0 and 1 using the following function:  normalize &lt;- function(x) {return((x - min(x))/ (max(x) - min(x)))}   The NN model fits well and has an acceptable out-of-sample prediction error.   However, I read in other  that scaling the inputs to have mean 0 and a variance of 1 is advised for NN. I don't fully understand:   how this transformation works better for NN against the min-max normalization between 0 and 1. how can I assess which transformation to apply in my data?
91979
I would like to use a neural network for image classification.  I'll start with pre-trained CaffeNet and train it for my application.    How should I prepare the input images?  In this case, all the images are of the same object but with variations (think: quality control).  They are at somewhat different scales/resolutions/distances/lighting conditions (and in many cases I don't know the scale).  Also, in each image there is an area (known) around the object of interest that should be ignored by the network.  I could (for example) crop the center of each image, which is guaranteed to contain a portion of the object of interest and none of the ignored area; but that seems like it would throw away information, and also the results wouldn't be really the same scale (maybe 1.5x variation).  Dataset augmentation  I've heard of creating more training data by random crop/mirror/etc, is there a standard method for this?  Any results on how much improvement it produces to classifier accuracy?
69305
I am trying to understand some descriptions of PCA (the first two are from Wikipedia), emphasis added:     Principal components are guaranteed to be independent only if the data set is jointly normally distributed.   Is the independence of principal components very important? How can I understand this description?     PCA is sensitive to the relative scaling of the original variables.     What does 'scaling' mean there? Normalization of different dimensions?     The transformation is defined in such a way that the first principal component has the largest possible variance and each succeeding component in turn has the highest variance under the constraint that it be orthogonal to the preceding components.   Can you explain this constraint?
43228
Recently, I have been studying autoencoders. If I understood correctly, an autoencoder is a neural network where the input layer is identical to the output layer. So, the neural network tries to predict the output using the input as golden standard.  What is the usefulness of this model? What are the benefits of trying to reconstruct some output elements, making them as equal as possible to the input elements? Why should one use all this machinery to get to the same starting point?
81294
I'm implementing a neural network with Keras, but the Sequential model returns nan as loss value. I have sigmoid activation function in the output layer to squeeze output between 0 and 1, but maybe doesn't work properly.  This is the code:  def data_generator(batch_count, training_dataset, training_dataset_labels):   while True:     start_range = 0     for batch in batch_count:       end_range = (start_range + batch[1])       batch_dataset = training_dataset[start_range:end_range]       batch_labels = training_dataset_labels[start_range:end_range]       start_range = end_range       yield batch_dataset, batch_dataset  mlp = keras.models.Sequential()  # add input layer mlp.add(     keras.layers.Input(         shape = (training_dataset.shape[1], )     ) ) # add hidden layer mlp.add(     keras.layers.Dense(         units=training_dataset.shape[1] + 10,         input_shape = (training_dataset.shape[1] + 10,),         kernel_initializer='random_uniform',         bias_initializer='zeros',         activation='relu')     ) # add output layer mlp.add(     keras.layers.Dense(         units=1,         input_shape = (1, ),         kernel_initializer='glorot_uniform',         bias_initializer='zeros',         activation='sigmoid')     )  print('Compiling model...\n')  mlp.compile(     optimizer='adam',     loss=listnet_loss )  mlp.summary() # print model settings   # Training with tf.device('/GPU:0'):   print('Start training')   #mlp.fit(training_dataset, training_dataset_labels, epochs=50, verbose=2, batch_size=3, workers=10)   mlp.fit_generator(data_generator(groups_id_count, training_dataset, training_dataset_labels),                     steps_per_epoch=len(training_dataset), epochs=50, verbose=2, workers=10, use_multiprocessing=True)   How can I do?
22612
Recently I was coaching a developer who's new to making pixel art games, and realized there's some "conventional wisdom" around scaling pixel art that's not obvious to a newcomer, and might not be well documented. So, here's an attempt to fix that....     I've heard I should only scale pixel art by 200%, 300% etc. not in-between sizes like 120%   Why is that? Shouldn't scaling "just work" the way it does with other images? What if I really need to make my character just a fraction larger or smaller to get the right gameplay or look? How do I fix this?  I'm told it's "bad" to use pixel art at different scales in one game - why?    For example, if I scaled up my character so each sprite texel is 4x4 screen pixels, while my background art is scaled up so each sprite texel is 6x6 screen pixels. Isn't a pixel a pixel at the end of the day?
123905
I created a model for two different datasets that have different scales. When checking which one performed better, I am struggling with figuring out the best methodology. My top choices right now are R^2, RMSE, and variance. However, R^2 seems to be the only option that works best in my opinion because it doesn't seem to take into account the scale of the data but wondering if there are options such as normalizing the numbers etc.   Below is a sample situation:        Date      Group      Actual_Values       Predicted_Values 2020-04-24          A              0.181                  0.140 2020-04-27          A              0.190                  0.127 2020-04-28          A              0.188                  0.126 2020-04-29          A              0.188                  0.123 2020-04-30          A              0.185                  0.122 2020-05-01          A              0.178                  0.124 2020-05-04          A              0.182                  0.118 2020-05-05          A              0.180                  0.120 2020-05-06          A              0.178                  0.120 2020-05-07          A              0.180                  0.121 2020-05-08          A              0.184                  0.124  2020-04-24          B              86.24                  86.25 2020-04-27          B              94.00                  86.65 2020-04-28          B              98.44                  86.88 2020-04-29          B              107.3                  87.05 2020-04-30          B              94.26                  87.27 2020-05-01          B              96.71                  87.42 2020-05-04          B              92.28                  87.59 2020-05-05          B              90.97                  87.73 2020-05-06          B              87.47                  87.80 2020-05-07          B              88.93                  87.90 2020-05-08          B              92.41                  87.93   From this dataset I tested the following code:  library(ModelMetrics) with_rmse&lt;- historical_results %&gt;%   group_by(Group) %&gt;%   summarise(     RMSE = rmse(Predicted_Values, Actual_Values)     ,R2 = cor(Predicted_Values, Actual_Values)^2     ,variance = abs(var(Predicted_Values, Actual_Values))   )  Group           RMSE            R2      variance     A     0.05929241    0.02067078  3.548883e-06        B       8.461625    0.02991999     0.5624356   So for example, the above results like 8.4 for RMSE may actually be good but looks bad just because the original values were higher than A. Any help would be great, thanks! This is just an example - disregard the bad R2 :)
36884
Hey guys I found one or two papers which use ridge regression (for basketball data). I was always told to standardize my variables if I ran a ridge regression, but I was simply told to do this because ridge was scale variant (ridge regression wasn't really part of our course, so our lecturer skimmed through it).  These papers I read didn't standardize their variables, which I found a bit surprising. They also ended up with large values of lambda (around the 2000-4000 level) via cross-validation, and I was told that this is due to not standardizing the variables.  How exactly does leaving the variable(s) unstandardised lead to high lambda values and also, what are the consequences of not standardizing the variables in general? Is it really such a big deal?  Any help is much appreciated.
3877
Do I transform all my data or folds (if CV is applied) at the same time? e.g.  (allData - mean(allData)) / sd(allData) Do I transform trainset and testset separately? e.g.  (trainData - mean(trainData)) / sd(trainData)  (testData - mean(testData)) / sd(testData) Or do I transform trainset and use calculations on the testset? e.g.  (trainData - mean(trainData)) / sd(trainData)  (testData - mean(trainData)) / sd(trainData)   I believe 3 is the right way. If 3 is correct do I have to worry about the mean not being 0 or the range not being between [0; 1] or [-1; 1] (normalization) of the testset?
17719
I still don't understand why we force the distribution of the hidden representation of a Variational Autoencoder (VAE) to follow a multivariate normal distribution. Why this specific distribution and not another one ?   This is maybe linked with another question : Why is the weights distribution in a neural network following a Gaussian Distribution ? Is it just the application of the Central Limit theorem that tells you that many independent inputs will generate many independent errors, and the observed weights are the results of these multiple back-propagated signals... ?
49431
I have 40 different images (1, ..40),( each one has 10 repetition " 40x10"),i should use one hot encoding for each image (the 40),  for example image (1) the one hote encoding will be encoding will be: [ 1 0 0 0 0 0 0.....40(0)],  for images 2 the one hot encoding will be : [ 0 1 0 0 0 0 0 ...40(0)],  since each image has 10 repetition, for example; for the 10 images of 1 ( they are for same object but with slight difference among them),  should  I use same code vector for the 10 repetition!!
54253
My input variables have different dimensions. Some variables are decimal while some are hundreds. Is it essential to center (subtract mean) or scale (divide by standard deviation) these input variables in order to make the data dimensionless when using random forest?
108157
Most of real world datasets have features with missing values. Replacing missing values with an appropriate value such as its mean, is considered as a good step in feature engineering. Some times we also standardize/normalize feature columns before feeding it to train an model. Before modelling  we also split our dataset to training and testing sets. My first query is how do we do feature engineering in this splitted dataset? Do we use a global mean of the unsplitted features  to replace the missing value of those features in both training and testing set or should we use local means of those sets? Like the above question how do we do normalization to a train, test dataset? The last but an important question, in productions we mostly get feature values one at a time (think a row of features), how do we feature engineer such data rows?
39206
I have dataset of 15 features and the goal is to estimate a best fitting curve (regression). Now I want to use a deep learning technique for this purpose. Now I see that there are many architectures that can be used such as CNN, DBM or auto-encoder, etc.. Which one to go for. And I could not find any examples for how to use the same. A sample code would be of great help. Thanks  P.S. Some of the features are categorical. How do we handle that as well?
6266
If I have a dataset with $n$ observations and $p$ variables (dimensions), and generally $n$ is small ($n=12-16$), and $p$ may range from small ($p = 4-10$) to perhaps much larger ($p= 30-50$).  I remember learning that $n$ should be much larger than $p$ in order to run principal component analysis (PCA) or factor analysis (FA), but it seems like this may not be so in my data. Note that for my purposes I am rarely interested in any principal components past PC2.   Questions:   What are the rules of thumb for minimum sample size when PCA is OK to use, and when it is not? Is it ever OK to use the first few PCs even if $n=p$ or $n&lt;p$?  Are there any references on this? Does it matter if your main goal is to use PC1 and possibly PC2 either:    simply graphically, or  as synthetic variable then used in regression?
50890
In a VAE, the encoder learns to output two vectors:  $$\mathbf{\mu} \in\ \mathbb{R}^{z}$$ $$\mathbf{\sigma} \in\ \mathbb{R}^{z}$$  which are the mean and variances for the latent vector $\mathbf{z}$, the latent vector $\mathbf{z}$ is then calculated by:  $$\mathbf{z} = \mu + \sigma \epsilon$$  where: $\epsilon = N(0, \mathbf{I}_{z \times z})$  The KL divergence loss for a VAE for a single sample is defined as (referenced from  and ):  $$\frac{1}{2} \left[ \left(\sum_{i=1}^{z}\mu_{i}^{2} + \sum_{i=1}^{z}\sigma_{i}^{2} \right) - \sum_{i=1}^{z} \left(log(\sigma_{i}^{2}) + 1 \right) \right]$$  Though, I'm not sure how they got their results, would anyone care to explain or point me to the right resources?
11359
I found rectified linear unit (ReLU) praised at several places as a solution to the  for neural networks. That is, one uses max(0,x) as activation function. When the activation is positive, it is obvious that this is better than, say, the sigmoid activation function, since its derivation is always 1 instead of an arbitrarily small value for large x. On the other hand, the derivation is exactly 0 when x is smaller than 0. In the worst case, when a unit is never activated, the weights for this unit would also never change anymore, and the unit would be forever useless - which seems much worse than even vanishingly small gradients. How do learning algorithms deal with that problem when they use ReLU?