Lets say that I have 3,200 observations that I want to use for training a neural network model, and I want to set the batch size to 32. The number of minibatches used in every epoch for updating the weights is therefore 100. How will my data be divided during minibatching? Assuming I have datapoints $x_{1}, x_{2}, ... , x_{3200}$ -  will minibatch slice the data as it is, and use the same slices in every epoch? e.g. $$ Epoch 1: minibatch1: [x_{1}, x_{2}, ... , x_{32}], minibatch2: [x_{33}, x_{34}, ... , x_{64}] \\ Epoch 2: minibatch1: [x_{1}, x_{2}, ... , x_{32}], minibatch2: [x_{33}, x_{34}, ... , x_{64}]\\ Epoch 40:  minibatch1: [x_{1}, x_{2}, ... , x_{32}], minibatch2: [x_{33}, x_{34}, ... , x_{64}] $$ Or are the slices shuffled in every epoch, such that there are different combinations of observations in every minibatch in every training epoch? e.g. $$ Epoch 1: minibatch1: [x_{67}, x_{2891}, ... , x_{930}], minibatch2: [x_{102}, x_{7}, ... , x_{1241}] \\ Epoch 2: minibatch1: [x_{3174}, x_{15}, ... , x_{412}], minibatch2: [x_{753}, x_{2447}, ... , x_{1630}]\\ Epoch 40:  minibatch1: [x_{456}, x_{73}, ... , x_{1984}], minibatch2: [x_{29}, x_{675}, ... , x_{2354}] $$
122689
I saw some related question regarding to the fact is one should use sampling with resampling or without when using minibatch.  However my question is different.  Let's assume that I use sampling witouth replacement. Meaning that for each epoch one training example can only belong one 'batch size' group.  I want to know if one should only resample the training data (into the different batch size) only 1 time or before to start to update the parameters before each epoch.  For instance suppose I have 20 'batch size' group. Suppose at the first iteration I have sampled picture A in group 2 and picture B in group 4.  If so should picture A always stay in group 2 and picture B in group 4. Or should I resample it before to start a new epoch (e.g. at epoch 2 put picture B in group 6 and picture B in group 20).
17691
I want to quantify the relationship between two variables, A and B, using mutual information. The way to compute it is by binning the observations (see example Python code below). However, what factors determines what number of bins is reasonable? I need the computation to be fast so I cannot simply use a lot of bins to be on the safe side.  from sklearn.metrics import mutual_info_score  def calc_MI(x, y, bins):     c_xy = np.histogram2d(x, y, bins)[0]     mi = mutual_info_score(None, None, contingency=c_xy)     return mi
97488
I am using single observation to compute losses using neural network implementation in PyTorch. I am confused in a small detail of SGD. If I compute loss and do loss.backward(), I am accumulating gradients.  If I do this on 100 observations and then run optimizer.step(), should I average out the gradients?  This is what I am doing as of now:  def compute_loss(training_data):   for data in training_data:     loss = F.mse_loss(data[0], data[1])     loss.backward()  def optimize(sample):   optimizer.zero_grad()   compute_loss(sample)   optimizer.step()   Should it be rather:  def compute_loss(training_data):   for data in training_data:     loss = F.mse_loss(data[0], data[1])     loss.backward(torch.Tensor([1.0/len(training_data)]))
6455
I am trying to classify images to more then a 100 classes, of different sizes ranged from 300 to 4000 (mean size 1500 with std 600). I am using a pretty standard CNN where the last layer outputs a vector of length number of classes, and using pytorch's loss function CrossEntropyLoss.  I tried to use $weights = \frac{max(sizes)}{sizes}$ for the cross entropy loss which improved the unweighted version, not by much.  I also thought about duplicating images such that all classes ends up to be of the same size as the larges one.  Is there any standard way of handling this sort of imbalance?
31606
I was hoping I could receive some clarification into the the four cases:   Placing labeled balls in unlabeled boxes with repetition. Placing labeled balls in unlabeled boxes without repetition. Placing unlabeled balls in unlabeled boxes with repetition. Placing unlabeled balls in unlabeled boxes without repetition.   I just want an intuitive understanding of the four cases.  How many ways are there to arrange, say, 4 balls into 9 boxes or 9 balls into 4 boxes in each case above, and why?
53185
Assume that you found the optimal hyperparameters of a neural network (e.g. a multi layer feed forward NN) with k-fold cross validation in a grid search. Lets assume you have varied the number of epochs and got the optimal epoch number as a result.  Now you want to train the network for the optimal number of epochs on the whole data used for CV. In my opinion following problem arises: When using a constant mini batch size, number of iterations varies for datasets of different size - so your optimal epoch number found for the smaller training split in the CV can not be applied to training with the whole CV data. Am I missing something here? Right at the moment I can not find a proper solution for finding the optimal training epochs for my NN. I searched the net for some time now, but could not find an answer.  P.S. I only described the problem after the inner loop of nested CV, but the same problem arises after the outer loop (training the model on ALL data, not only on CV data for putting the model to production).
109448
I must clarify immediately that I am a practicing software developer, not a statistician, and that my college stats class was a very long time ago…  That said, I would like to know if there is a method for accumulating a set of descriptive statistics that could then be used to produce a boxplot, that does not entail storing a bunch of individual samples?  What I am trying to do is produce a graphical summary of queue service times within a complex multi-queue process. I have in the past used a package called tnftools that allowed large samples to be accumulated and then post-processed into a nice graph of response times and outliers… But tnftools are not available for my current platform.  Ideally I would like to be able to accumulate a set of descriptive statistics "on the fly" as the process runs, and then extract the data for analysis on demand. But I cannot simply have the process accumulate samples as the memory / IO involved in doing so would have an unacceptable impact on the performance of the system.
56145
So, for fun, I am taking some of the data of calls from the call center I work at and trying to do some hypothesis testing on them, specifically the number of calls received in a week, and using a Poisson distribution to fit it. Due to the subject matter of my job, there are two types of weeks, let's call one of them on-weeks where I hypothesize there are more calls, and off-weeks where I hypothesize there are fewer.  I have a theory that the $\lambda$ from on-weeks (let's call it $\lambda_1$) is larger than that of the one from off-weeks (let's call it $\lambda_2$)  So the hypothesis I want to test is $H_0: \lambda_1 &gt; \lambda_2, H_1: \lambda_1 \leq \lambda_2 $  I know how to test for one parameter (say  $H_0: \lambda_1 &gt; 1, H_1: \lambda_1 \leq 1 $) but not so sure how to go about doing 2 given a data set. Let's say I take two week's worth of data from each one $X_1 = 2$ and $X_2 = 3$ for the off-week and $Y_1 = 2$ and $Y_2=6$ for the on-week. Can someone help walk me though this simpler version such that I can apply it to a larger data set? Any help is appreciated, thank you.
39206
I have dataset of 15 features and the goal is to estimate a best fitting curve (regression). Now I want to use a deep learning technique for this purpose. Now I see that there are many architectures that can be used such as CNN, DBM or auto-encoder, etc.. Which one to go for. And I could not find any examples for how to use the same. A sample code would be of great help. Thanks  P.S. Some of the features are categorical. How do we handle that as well?
3193
I'm interested in finding as optimal of a method as I can for determining how many bins I should use in a histogram. My data should range from 30 to 350 objects at most, and in particular I'm trying to apply thresholding (like Otsu's method) where "good" objects, which I should have fewer of and should be more spread out, are separated from "bad" objects, which should be more dense in value. A concrete value would have a score of 1-10 for each object. I'd had 5-10 objects with scores 6-10, and 20-25 objects with scores 1-4. I'd like to find a histogram binning pattern that generally allows something like Otsu's method to threshold off the low scoring objects. However, in the implementation of Otsu's I've seen, the bin size was 256, and often I have many fewer data points that 256, which to me suggests that 256 is not a good bin number. With so few data, what approaches should I take to calculating the number of bins to use?
45382
I have just heard, that it's a good idea to choose initial weights of a neural network from the range $(\frac{-1}{\sqrt d} , \frac{1}{\sqrt d})$, where $d$ is the number of inputs to a given neuron. It is assumed, that the sets are normalized - mean 0, variance 1 (don't know if this matters).  Why is this a good idea?
26173
Is there a standard and accepted method for selecting the number of layers, and the number of nodes in each layer, in a feed-forward neural network? I'm interested in automated ways of building neural networks.
73299
When you are trying to fit models to a large dataset, the common advice is to partition the data into three parts: the training, validation, and test dataset.  This is because the models usually have three "levels" of parameters: the first "parameter" is the model class (e.g. SVM, neural network, random forest), the second set of parameters are the "regularization" parameters or "hyperparameters" (e.g. lasso penalty coefficient, choice of kernel, neural network structure) and the third set are what are usually considered the "parameters" (e.g. coefficients for the covariates.)  Given a model class and a choice of hyperparameters, one selects the parameters by choosing the parameters which minimize error on the training set.  Given a model class, one tunes the hyperparameters by minimizing error on the validation set.  One selects the model class by performance on the test set.  But why not more partitions?  Often one can split the hyperparameters into two groups, and use a "validation 1" to fit the first and "validation 2" to fit the second.  Or one could even treat the size of the training data/validation data split as a hyperparameter to be tuned.  Is this already a common practice in some applications?  Is there any theoretical work on the optimal partitioning of data?
113630
We have a series of observations of different properties (such as heart rate or blood sugar level and others as well) across different days from different people from different geographical regions. The number of days on which the observations are made is the same for all regions; but there could be different number of participants in each region. The result of the data collection will be as follows: DATA We have a matrix, $M_{i,j}$, for each region $i$ and property $j$, with $T$ observations across time (rows) and $N_{i,j}$ (columns) for number of participants from region $i$ and property $j$. Here, we consider all properties together as representing a region. OBJECTIVE The goal is to identify people from which regions have a more similar constitution. QUESTION How can we compute the Bhattacharya Distance across the various matrices $M_{i,j}$ for all the properties combined for a region (corresponding to Case 2 of the data collection)? Dimension of Matrix: $Dim(M_{i,j})=(T,N_{i,j})$ There are two cases here. 1) $N_{i,j}=N_{i,k}$ $,\forall j,k$ That is the number of participants in a region are the same for all properties being measured. 2) $N_{i,j} \neq N_{i,k}$ $,\forall j,k$ That is the number of participants in a region could be different for some of the properties being measured. But if helpful we can assume the simple case 1) $N_{i,j}=N_{i,k}$. Please let me know if anything is not clear and if you need any further information. Related Case When we have a matrix, $M_i$, for each region $i$, with $T$ observations across time (rows) and $N_i$ (columns) for number of participants from that region. Here, we consider each property separately. How can we compute the Bhattacharya Distance across the various matrices $M_i$ (in pairs or combined, whichever is simpler) separately for each property (corresponding to Case 1 of the data collection)? Answer:
2778
I'm building an app that will require user input. Currently, on the training set, I run the following code, in which data is a pandas dataframe with a combination of categorical and numerical columns.  dummified_data = data.get_dummies() train_data = dummified_data[:10000] test_data = dummified_data[10000:12000]   Currently, I have a hand-written function that takes user-inputted data and transforms it into a format like dummy data. This doesn't seem sustainable as the number of columns/the size of my categorical variables grows.  Is there a way to dummify training data and production data consistently?
113
I'm training a neural network for a particular problem which can be predicted with 100% accuracy. However, the problem is that results tend to vary between 99% and 100% even if I train 10 different networks and take the average. I was reading around and found that it was due to random weights initialized every time the program ran. Is it okay to set the random seed to 1 and then train 10 different networks and take the average?  This will result in less variation and easy model selection.
103046
I'm unclear on the exact process of using the validation data.  Let's say that I fit my neural network model and adjust hyperparameters using the training set and validation set. Do I then evaluate the test set on this model? Or do I recombine the validation and training sets and fit a fresh model with the hyperparameters that I found during the validation phase, and then evaluate on the test data? I have seen a number of different notebooks and examples that do both ways.  Surely, once I've found my hyperparameters, it makes sense to fit a fresh model using the full training set (recombined with validation set), since the validation loss has no effect on the weights.