I understand that the basic premise of bootstrapping a dataset with N data points is:  Sample with replacement from N a number of draws, D, equal to N; Repeat step 1 K times - that is, as much as you can stand computationally (slash when the estimated parameters/distribution is relatively normal and stable with minimal gain from increasing k, etc.)  Why does the number of draws D equal N? I can understand intuitively that it wouldn't be great to select fewer draws than N, since fewer draws will limit your ability to correctly estimate the mean. I also saw  suggesting that there are cases when it might be permissible to draw a D smaller than N. But since you're sampling with replacement, what is stopping you from drawing a D twice as large as N? Or three times? Is it because it would &quot;over-weight&quot; more data points with more frequent representation in the dataset relative to rarer datapoints? Thank you for your help!
122058
I want to predict future returns over a 20 days horizon using an ARMA-GARCH model fitted to my data. The goal is to estimate different risk measures like VaR or CVar. In particular say I use AR(1) GARCH(1,1). The smaple I use for estimation has 500 observation of daily logreturns. That's what I usually do:   Estimate AR, ARCH and GARCH coefficients Calculate standardized residual by dividing residuals by estimated conditional variances The standardized residuals constitute my INVARIANTS that is the i.i.d. series from which I extract bootstrap samples to generate scenarios.   The bootstrap samples are extracted by simulating a uniform between 1 and the sample size (500 in this case) and then taking the value corresponding to that position in the vector of the standardized residuals.  The problem is that I have only 500 standardized residuals and I think 500 is the maximum size of bootstrap samples I can extract.   My colleague instead extracts 100000 observations out of the original sample of $N=500$ observations.  I feel that this is somehow incorrect conceptually. Simulating only one step forward would produce exactly the same scenario as the initial one, but with repeted values that add no information.  My colleague claims that if he wants to project over a longer period, e.g. 20 days horizon, the 100000 extractions from the original sample of N=500 obs. would produce many different scenarios at the final horizon, providing a CDF that is smooth.  Actually thit is true because, although the values are simply repeated in the first step, after that they can sum up in many different ways.  That being said I don't feel this is right. I proposed an alternative that is:   From the standardized residuals create a smoothed empirical CDF, say kernel Exctract uniforms between 0 and 1 and feed it to the empirical smoothed CDF, i.e. inverse transform.   This way I feel more confortable to say that I can generate a bootstrab sample of size greater than the original one, but still I am not sure.  I am studying bootstrap theory on a book from Efron      Efron, Tibshirani - An Introduction to the Bootstrap - Springer US (1993)   but there are many concepts that I  don't understand yet.  My question are:   Would you give your opinion on the problem I just showed below? Would you suggest any valid matherial for studying bootstrap other than the book I mentioned? I think that this application of bootstrap is somehow different to the one explained in Efron's bookm, that is evaluating confidence intervals for estimated paramenters. What do you think about it?   Any comment would be much appreciated  I apologize for the length of the post but I tried to be as much concise as I could. Thank you
103534
I'm trying to solve  from the UCI database repository. Unfortunately (or fortunately), I've noticed that my dataset is imbalanced. I've structured the data as 5 classes, according to the final mark reached by the pupil, like so:   If student gets a mark from 0 to 7 => class 1 [FAIL(E)] If student gets a mark from 8 to 9 => class 2 [SUFFICIENT(D)] If student gets a mark from 10 to 11 => class 3 [GOOD(C)] If student gets a mark from 12 to 15 => class 4 [NOTABLE(B)] If student gets a mark from 16 to 19 => class 5 [OUTSTANDING(A)]   My problem is that, as I said, data are imbalanced, so I want to balance it.  I've thought about applying some kind of undersampling method, but my dataset has only 649 instances so I think removing some of them is not the best idea. Then I thought about doing some oversampling, in order to replicate some of the minority class examples and then get classes balanced, but I'm still unsure if that could work.   I would be very grateful if you could give me a hand with this. Its the first time I face a real problem with imbalanced data.
44208
Suppose you have a series of n trials, where the probability of success  in each trial is p.  The distribution of the number of successful trials follows a Binomial distribution with parameters (n, p).  The mean is given by np whereas the variance is np(1-p).  So far so good: this is pretty mundane Stats 101 stuff.  But suppose now that I only knew about the successful trials, and had no knowledge of the total number n of trials, which is the variable I am interested in estimating.  For example, I knew I had 100 successful trials, where each trial had a 0.1 chance of success.  Is there a known probability distribution that describes the probable outcomes for n, the total number of trials?  Estimating the mean is easy: if m is the number of successes, then it's just m/p.  But what about variance and other measures?  What if each success had a different (but known) chance of success?  Suppose I had the following records:   success1 (with p=0.1) success2 (with p=0.1) success3 (with p=0.2)   Again, a good estimation of the total number of trials can be obtained by simply summing 1/p for each successful trial.  In this case that number  is 10+10+5=25.  But what about variance and other measures?
122689
I saw some related question regarding to the fact is one should use sampling with resampling or without when using minibatch.  However my question is different.  Let's assume that I use sampling witouth replacement. Meaning that for each epoch one training example can only belong one 'batch size' group.  I want to know if one should only resample the training data (into the different batch size) only 1 time or before to start to update the parameters before each epoch.  For instance suppose I have 20 'batch size' group. Suppose at the first iteration I have sampled picture A in group 2 and picture B in group 4.  If so should picture A always stay in group 2 and picture B in group 4. Or should I resample it before to start a new epoch (e.g. at epoch 2 put picture B in group 6 and picture B in group 20).
28001
I have found many answers to the question of what an epoch is, but none to this:  Why do we use multiple epochs when training a neural net? How does this not lead to overfitting?  From my understanding 2 epochs means we training with each data point using SGD at least twice.   Why do we not do this when training regression models?
42640
I am currently reading an introductory machine learning book by Daumé () and when discussing the mapping of categorical features with "n" possible values into "n" binary indicator features, the following question is proposed:     Is it a good idea to map a categorical feature with n values to log2(n) binary features?   Why wouldn't that be the case, seeing as how much resources could be spared by working with fewer features? Does this approach depend on the model that is being used?
9388
I have taken a sample of $n$ data points from a population. Each of these points has a true value (known from ground truth) and an estimated value. I then calculate the error for each sampled point and then calculate the RMSE of the sample.  How can I then infer some sort of confidence interval around this RMSE, based upon the sample size $n$?  If I was using the mean, rather than the RMSE, then I wouldn't have a problem doing this as I can use the standard equation  $ m = \frac{Z \sigma}{\sqrt{n}} $  but I don't know whether this is valid for RMSE rather than the mean. Is there some way that I can adapt this?  (I have seen , but I don't have issues with whether my population is normally-distributed, which is what the answer there deals with)
69363
Provided a sample size "N" that I plan on using to forecast data. What are some of the ways to subdivide the data so that I use some of it to establish a model, and the remainder data to validate the model?  I know there is no black and white answer to this, but it would be interesting to know  some "rules of thumb" or usually used ratios. I know back at university, one of our professors used to say model on 60% and validate on 40%.
99507
Does it make sense to compute confidence intervals and to test hypotheses when data from the whole population is available? In my opinion, the answer is no, since we can accurately compute the true values of the parameters. But then, what is the maximal proportion of data from the original population that allows us to use the aforementioned techniques?
93757
Given a $5 \times 2$ dataset $\mathbf{X} =\left( \begin{array}{rr}-0.9&amp;0.2\\2.4&amp;0.7\\-1.4&amp;1.0\\2.9&amp;-0.5\\2.0&amp;-1.0 \end{array} \right)$. Assume that $X\sim N_2(\mu, \Sigma)$.   Using $\hat{\mu}_{MLE}$ and $\hat{\Sigma}_{MLE}$, I need to calculate the eigenvalues and eigenvectors and draw an 80% probability ellipse. In the statistical books I've seen, it is usually stated that to draw such an ellipse, you use $\mathbf{S} = \frac{1}{n-1}\sum(\mathbf{x_j} - \mathbf{\bar{x}})(\mathbf{x_j} - \mathbf{\bar{x}})'$ and calculate the eigenvalues and eigenvectors for this matrix. Then you can draw the ellipse, using these eigenvectors and the calculated $\mathbf{\bar{x}}$.  My Question  Now I now that $\hat{\Sigma}_{MLE} = \frac{n-1}{n}\mathbf{S}$. So what do I need to do here? Is it correct to use  $\hat{\Sigma}_{MLE}$ to calculate the eigenvalues and -vectors and use them to find the 80% confidence interval, or do you always need to use $\mathbf{S}$?
45382
I have just heard, that it's a good idea to choose initial weights of a neural network from the range $(\frac{-1}{\sqrt d} , \frac{1}{\sqrt d})$, where $d$ is the number of inputs to a given neuron. It is assumed, that the sets are normalized - mean 0, variance 1 (don't know if this matters).  Why is this a good idea?
89176
Is stochastic gradient descent basically the name given to mini-batch training where batch size = 1 and selecting random training rows? i.e. it is the same as 'normal' gradient descent, it's just the manner in which the training data is supplied that makes the difference?  One thing that confuses me is I've seen people say that even with SGD you can supply more than 1 data point, and have larger batches, so won't that just make it 'normal' mini-batch gradient descent?
31384
I am trying to understand difference between different  (Monte Carlo simulation, parametric bootstrapping, non-parametric bootstrapping, jackknifing, cross-validation, randomization tests, and permutation tests) and their implementation in my own context using R.  Say I have the following situation – I want to perform ANOVA with a Y variable (Yvar) and X variable (Xvar). Xvar is categorical. I am interested in the following things:  (1) Significance of p-values – false discovery rate  (2) effect size of Xvar levels    Yvar &lt;- c(8,9,10,13,12, 14,18,12,8,9,   1,3,2,3,4) Xvar &lt;- c(rep("A", 5),  rep("B", 5),    rep("C", 5)) mydf &lt;- data.frame (Yvar, Xvar)   Could you gel me to explain the  sampling differences with explicit worked examples how these resampling method work?  Edits: Here are my attempts:  Bootstrap  10 bootstrap samples, sample number of samples with replacement, means that samples can be repeated     boot.samples &lt;- list() for(i in 1:10) {    t.xvar &lt;- Xvar[ sample(length(Xvar), length(Xvar), replace=TRUE) ]    t.yvar &lt;- Yvar[ sample(length(Yvar), length(Yvar), replace=TRUE) ]    b.df &lt;- data.frame (t.xvar, t.yvar)     boot.samples[[i]] &lt;- b.df  } str(boot.samples)  boot.samples[1]   Permutation: 10 permutation samples, sample number of samples without replacement   permt.samples &lt;- list()     for(i in 1:10) {        t.xvar &lt;- Xvar[ sample(length(Xvar), length(Xvar), replace=FALSE) ]        t.yvar &lt;- Yvar[ sample(length(Yvar), length(Yvar), replace=FALSE) ]        b.df &lt;- data.frame (t.xvar, t.yvar)         permt.samples[[i]] &lt;- b.df      }     str(permt.samples)     permt.samples[1]   Monte Caro Simulation     I am not sure about all above terms and whether my above edits are correct. I did find some information on  but I could not tame it to my situation.
50007
I've read that "If the error distribution is significantly non-normal, confidence intervals may be too wide or too narrow" (). So, can anyone elaborate on this? When are the confidence intervals narrow and when are they wide? Does it have anything to do with skewness?
90415
Say I generate a dataset $X$: the first $i$ samples follow $x_i\sim N(\mu_1,\Sigma_1^2)$, the next j samples follow $N(\mu_2,\Sigma_2^2)$ and the last $l$ samples from $N(\mu_3,\Sigma_3^2)$. Naturally, I denote the clusters $(1,2,3)$ by the order of creation.  When using kmeans(X,3)$clusters I get the predicted assignment of the samples, but here lies the problem: kmeans might enumerate the clusters differently, for example set $N(\mu_2,\Sigma_2^2)$ as cluster#3 and vice versa. In this situation, I cannot compare the 'original' cluster assignment (say $(1,1,2,2,2,3,3,3,3,3)$) with the predicted one (say $(1,1,3,3,3,2,2,2,2,2)$). this will result with only 20% accuracy, even though the actual accuracy is 100%.  Any ideas of how to map the centroids or measure the accuracy of the kmeans performance?
121214
An issue I've seen frequently brought up in the context of Neural Networks in general, and Deep Neural Networks in particular, is that they're "data hungry" - that is they don't perform well unless we have a large data set with which to train the network.  My understanding is that this is due to the fact that NNets, especially Deep NNets, have a large number of degrees of freedom. So as a model, a NNet has a very large number of parameters, and if the number of parameters of the model is large relative to the number of training data points, there is an increased tendency to over fit.   But why isn't this issue solved by regularization? As far as I know NNets can use L1 and L2 regularization and also have their own regularization methods like dropout which can reduce the number of parameters in the network.   Can we choose our regularizations methods such that they enforce parsimony and limit the size of the network?     To clarify my thinking: Say we are using a large Deep NNet to try to model our data, but the data set is small and could actually be modeled by a linear model. Then why don't the network weights converge in such a way that one neuron simulates the linear regression and all the others converge to zeros? Why doesn't regularization help with this?
84149
I work in the field of data mining and have had very little formal schooling in statistics. Lately I have been reading a lot of work  that focuses on Bayesian paradigms for learning and mining, which I find very interesting.   My question is (in several parts), given a problem is there a general framework by which it is possible to construct a statistical model? What are the first things you do when given a dataset of which you'd like to model the underlying process? Are there good books/tutorials out there that explain this process or is it a matter of experience? Is inference in the forefront of your mind when constructing your model or do you first aim to describe the data before you worry about how to use it to compute?  Any insight would be greatly appreciated! Thanks.