library(lme4) library(merTools) data(sleepstudy) fm1 &lt;- lmer(Reaction ~ Days + (Days|Subject), data=sleepstudy) PI &lt;- predictInterval(merMod = fm1, newdata = sleepstudy,                   level = 0.95, n.sims = 1000,                   stat = "median", type="linear.prediction",                   include.resid.var = TRUE)   My question is: is there a rule of thumb of what the value of n.sims should be?  EDIT  My data contains 20686 rows (i.e. 20686 response variables) and 20 predictors. For such dataset, how many bootstrap samples are required? Is there any plots or papers that I can refer to that explain the number of bootstrap samples as a function of data size?
30986
I wonder if someone knows any general rules of thumb regarding the number of bootstrap samples one should use, based on characteristics of the data (number of observations, etc.) and/or the variables included?
69363
Provided a sample size "N" that I plan on using to forecast data. What are some of the ways to subdivide the data so that I use some of it to establish a model, and the remainder data to validate the model?  I know there is no black and white answer to this, but it would be interesting to know  some "rules of thumb" or usually used ratios. I know back at university, one of our professors used to say model on 60% and validate on 40%.
72778
I've been using the  in R to build predictive models for classification and regression.  Caret provides a unified interface to tune model hyper-parameters by cross validation or boot strapping.  For example, if you are building a simple 'nearest neighbors' model for classification, how many neighbors should you use?  2? 10? 100? Caret helps you answer this question by re-sampling your data, trying different parameters, and then aggregating the results to decide which yield the best predictive accuracy.  I like this approach because it is provides a robust methodology for choosing model hyper-parameters, and once you've chosen the final hyper-parameters it provides a cross-validated estimate of how 'good' the model is, using accuracy for classification models and RMSE for regression models.  I now have some time-series data that I want to build a regression model for, probably using a random forest. What is a good technique to assess the predictive accuracy of my model, given the nature of the data? If random forests don't really apply to time series data, what's the best way to build an accurate ensemble model for time series analysis?
71336
For a prediction interval in linear regression you still use $\hat{E}[Y|x] = \hat{\beta_0}+\hat{\beta}_{1}x$ to generate the interval. You also use this to generate a confidence interval of $E[Y|x_0]$. What's the difference between the two?
8749
Inspired by Peter Donnelly's talk at , in which he discusses how long it would take for a certain pattern to appear in a series of coin tosses, I created the following script in R. Given two patterns 'hth' and 'htt', it calculates how long it takes (i.e. how many coin tosses) on average before you hit one of these patterns.  coin &lt;- c('h','t')  hit &lt;- function(seq) {     miss &lt;- TRUE     fail &lt;- 3     trp  &lt;- sample(coin,3,replace=T)     while (miss) {         if (all(seq == trp)) {             miss &lt;- FALSE         }         else {             trp &lt;- c(trp[2],trp[3],sample(coin,1,T))             fail &lt;- fail + 1         }     }     return(fail) }  n &lt;- 5000 trials &lt;- data.frame("hth"=rep(NA,n),"htt"=rep(NA,n))  hth &lt;- c('h','t','h') htt &lt;- c('h','t','t')  set.seed(4321) for (i in 1:n) {     trials[i,] &lt;- c(hit(hth),hit(htt))     } summary(trials)   The summary statistics are as follows,        hth             htt          Min.   : 3.00   Min.   : 3.000    1st Qu.: 4.00   1st Qu.: 5.000    Median : 8.00   Median : 7.000    Mean   :10.08   Mean   : 8.014    3rd Qu.:13.00   3rd Qu.:10.000    Max.   :70.00   Max.   :42.000    In the talk it is explained that the average number of coin tosses would be different for the two patterns; as can be seen from my simulation. Despite watching the talk a few times I'm still not quite getting why this would be the case. I understand that 'hth' overlaps itself and intuitively I would think that you would hit 'hth' sooner than 'htt', but this is not the case. I would really appreciate it if someone could explain this to me.
115472
I'm running PCA on my dataset which can be found . There are 6497 instances and 12 attributes with 13th column is the class (ranging from 3 - 9) for wine quality.  I've read what PCA is supposed to do. It combines attributes and shows variance in the data. Finally leading to be able to select few dimensions from a high dimension data that have the most variance.   I'm having trouble finding which attributes to pick after running PCA. How can I find out which attributes can I use and still not lose meaning of my data.   I've created the following biplot from the ggbiplot package    How can I interpret from this biplot which attributes to pick?  I've also created a screeplot for variance.     Looking at the variance screeplot I would say that 3 components would represent the data well. Is this a reasonable assessment to make?  Question   How can I decide which attributes to pick after running PCA While doing PCA should I leave out the class variable or keep it?   Code to generate the plots  wine &lt;- read.csv("wine_nocolor.csv") wine1 &lt;- wine[2:13] wine1.pca &lt;- prcomp(wine1, scale. = TRUE) library(devtools) install_github("ggbiplot", "vqv") ggbiplot(wine1.pca, obs.scale = 1, var.scale = 10, groups = wine$q, ellipse = TRUE, circle = TRUE, title="Biplot for Wine Quality (PC1, PC2)", var.axes=TRUE, varname.size=8)
103304
I have a training set of 87 samples and 9480 variables. My predictors are continuous and my response variable is binary.  I'd like to use the caret package in R to tune a neural network classification model on my data.  In order to do this, I first have to reduce the size of the predictor-set with feature selection to make it computationally feasible, correct?  Lets assume that is correct.  Then the question becomes "which feature selection method?".  If I were to do RF-RFE using the rfe function in caret, for example, I would get an optimal feature set specific to random forests and not necessarily optimal for neural networks.  Any advice?
66581
I am a user more familiar with R, and have been trying to estimate random slopes (selection coefficients) for about 35 individuals over 5 years for four habitat variables. The response variable is whether a location was "used" (1) or "available" (0) habitat ("use" below).  I am using a Windows 64-bit computer.  In R version 3.1.0, I use the data and expression below. PS, TH, RS, and HW are fixed effects (standardized, measured distance to habitat types). lme4 V 1.1-7.   str(dat) 'data.frame':   359756 obs. of  7 variables:  $ use     : num  1 1 1 1 1 1 1 1 1 1 ...  $ Year    : Factor w/ 5 levels "1","2","3","4",..: 4 4 4 4 4 4 4 4 3 4 ...  $ ID      : num  306 306 306 306 306 306 306 306 162 306 ...  $ PS: num  -0.32 -0.317 -0.317 -0.318 -0.317 ...  $ TH: num  -0.211 -0.211 -0.211 -0.213 -0.22 ...  $ RS: num  -0.337 -0.337 -0.337 -0.337 -0.337 ...  $ HW: num  -0.0258 -0.19 -0.19 -0.19 -0.4561 ...  glmer(use ~  PS + TH + RS + HW +      (1 + PS + TH + RS + HW |ID/Year),      family = binomial, data = dat, control=glmerControl(optimizer="bobyqa"))   glmer gives me parameter estimates for the fixed effects that make sense to me, and the random slopes (which I interpret as selection coefficients to each habitat type) also make sense when I investigate the data qualitatively. The log-likelihood for the model is -3050.8.  However, most research in animal ecology do not use R because with animal location data, spatial autocorrelation can make the standard errors prone to type I error. While R uses model-based standard errors, empirical (also Huber-white or sandwich) standard errors are preferred.   While R does not currently offer this option (to my knowledge - PLEASE, correct me if I am wrong), SAS does - although I do not have access to SAS, a colleague agreed to let me borrow his computer to determine if the standard errors change significantly when the empirical method is used.  First, we wished to ensure that when using model-based standard errors, SAS would produce similar estimates to R - to be certain that the model is specified the same way in both programs. I don't care if they are exactly the same - just similar. I tried (SAS V 9.2):  proc glimmix data=dat method=laplace;    class year id;    model use =  PS TH RS HW / dist=bin solution ddfm=betwithin;    random intercept PS TH RS HW / subject = year(id) solution type=UN; run;title;   I also tried various other forms, such as adding lines  random intercept / subject = year(id) solution type=UN; random intercept PS TH RS HW / subject = id solution type=UN;   I tried without specifying the   solution type = UN,   or commenting out  ddfm=betwithin;   No matter how we specify the model (and we have tried many ways), I cannot get the random slopes in SAS to remotely resemble those output from R - even though the fixed effects are similar enough. And when I mean different, I mean that not even the signs are the same. The -2 Log Likelihood in SAS was 71344.94.   I can't upload my full dataset; so I made a toy dataset with only the records from three individuals. SAS gives me output in a few minutes; in R it takes over an hour. Weird. With this toy dataset I'm now getting different estimates for the fixed effects.   My question: Can anyone shed light on why the random slopes estimates might be so different between R and SAS? Is there anything I can do in R, or SAS, to modify my code so that the calls produce similar results? I'd rather change the code in SAS, since I "believe" my R estimates more.   I'm really concerned with these differences and want to get to the bottom of this problem!  My output from a toy dataset that uses only three of the 35 individuals in the full dataset for R and SAS are included as jpegs.         EDIT AND UPDATE:  As @JakeWestfall helped discover, the slopes in SAS do not include the fixed effects. When I add the fixed effects, here is the result - comparing R slopes to SAS slopes for one fixed effect, "PS", between programs: (Selection coefficient = random slope). Note the increased variation in SAS.
106437
I am trying to predict values using arima(0,1,1). After doing predict(mod,n.ahead=5) (in R) am getting the same value for all the predictions:   5947.681 5947.681 5947.681 5947.681 5947.681    Is it correct?
85608
Prediction intervals seem to be talked about most in the context of regression, but I want to reduce it to one random variable to understand the reasoning. Assume you are sampling from a normal distribution $N(\mu ,\sigma ^{2})$. Wikipedia says the prediction interval for a new observation $X_{n+1}$ will be $\overline {X}_{n}+s_{n}{\sqrt  {1+1/n}}\cdot T^{{n-1}}$. I am wondering specifically about the $s_{n}{\sqrt  {1+1/n}}$ part of the equation. If you square it to get the variance, it's $s_{n}^{2}{({1+1/n})}$. Why is the variance $s_{n}^{2}{({1+1/n})}$ instead of just $s_{n}^{2}$? Isn't $s_{n}^{2}$ supposed to be an unbiased estimator of $\sigma ^{2}$ in $N(\mu ,\sigma ^{2})$, from which all the samples (including a hypothetical $X_{n+1}$) are drawn? So why wouldn't a new data point $X_{n+1}$ also have variance of $s_{n}^{2}$? If I had to guess, it's something to do with the uncertainty around $\overline {X}_{n}$, hence the extra $s_{n}^{2}/n$ term. Intuitively, it doesn't make sense to me that there is more uncertainty around a new data point i.e. variance of $s_{n}^{2}{({1+1/n})}$ when you already have same sample data to go off, compared to if you just blindly drew a new data point without any prior sampling i.e. variance of $s_{n}^{2}$. Would appreciate corrections to my thinking and reasoning about this.
41103
When obtaining MCMC samples to make inference on a particular parameter, what are good guides for the minimum number of effective samples that one should aim for?  And, does this advice change as the model becomes more or less complex?
60685
So I am using LASSO in order to reduce the amount of independent variables (~150) I have for my logistic model (n=1200). However, when doing so, the end result (i.e. number of predictors) it chooses varies depending on the seed I provide. 2-3 predictors are consistent, but then sometimes others are added. I guess this is due to the cross-validation for the best lambda parameter often used for LASSO. However, it just seem so inconsistent.  I mean I am using bootstrap in order to validate my results. So in the end I end up with meaningful standard errors, frequency of predictors, corrected statistics etc., but that doesn't change the fact, that the main model, defined on the original data, is kind of random depending on the seed.  How does one "solve" this ? Run the "main model" (i.e. original data) with 100-1000 different seeds to see which predictors are most frequently chosen, and then chose the model that actually has these predictors, or is there anything else that could be recommended ?
11877
I'm trying to construct several GLMMs with my thesis data and I can't solve this question. I have data on pollinator abundance and fruit set from plant individuals (with different treatments - related to a pest infection -) sampled at six study sites (spatial replicas) and during two consecutive years (time replicas). The problem is that not all individuals (plants) were sampled in both years (because of lack of bloom). My question is: How should i use &quot;Year&quot;, as random effect or as fixed effect? I constructed three candidate models and compared the AIC: a) Response variable ~Predictor variables + (1 | year/plant) + (1 | site/plant) b) Response variable ~Predictor variables + (1 | year) + (1 | site/plant)  c) Response variable ~Predictor variables * year + (1 | site/plant) Option a) had the lowest AIC and the highest R2, but I read in some papers that is not correct use a variable with less than five levels as random effect ... so, anyone could help me?
56402
Input Data  Multiple Tests Beds generate measurement data of various complexity.   In it's most basic form, not considering any meta-data, one measurement on a Test Bed will be a small (1 - a few thousand samples) time-series with a couple of dozen channels/signals/attributes per sample.  Measurements across time and Test Beds will have a similar set of signals, but not always the same as sensors are added and removed for the test setups.  Data volume  Currently we estimate our data rate at 6 testbeds x 4 test per hour x 12 hours a day x 4000 samples per test == 1,152,000 samples per day x 365 == 420,480,000 samples per year  _ x 48 columns per sample (currently 32 bit floats, mostly) ~~~ 75 GB per year  (columns in this case refers to channel/signal)  If/When more testbeds are added the data volume might increase accordingly.   Data Input  The test beds generate the data locally and the data is then imported asynchronously into the db. (A few thousand samples might be generated in a the time of one second, then reviewed locally and then either scratched or imported.)  Queries  We expect queries to be mostly on aggregates of the single measurements. I.e., you like to find all measurements (each having 4k samples) where e.g. the mean of channel_output_voltage is within a certain range.  Database layout?  What is a good way to set up tables for this? What factors have to be taken into account?  Theoretically I could go with one table per measurement generating 100,000 tables per year, but it doesn't strike me as a good idea.  Or I could stick everything into one big table (with hundreds of columns) that has room for all channels and channels get added as needed: One row per sample. Unused channels remain NULL.  MEASUREMENTS ------------ measurement_id } PK time_stamp     } channel_1 (may be NULL for a certain measurement_id ...) channel_2 ... channel_n(+1)   Or I could go with an approach of having one table for the samples (timestamps) and one table containing all the values: (one row per sample in MEASUREMENTS table and n rows per sample in the SAMPLE_VALUES table)  SAMPLES            SAMPLE_VALUES ------------       ------------- measurement_id     sample_id time_stamp         channel_id (links to a channels table where there is a name etc.) sample_id          channel_value   What other options are there? How to further investigate which option we should choose?  Database products  Due to customer constraints we would like to put this in MS SQLS or Oracle.    From :     Don't store the raw data, only store aggregates. Seriously.   This assumes that there is a meaningful way to determine ex antes what queries the customer is going to want to run against their data. No way :-)
95184
What is the minimum number of data points required for a kernel density estimation to be considered non-misleading/acceptable/adequate?  Is there a some rule based on how dispersed the data is?  For example, if the inter-quartile range is "small enough", do you need less points than if the inter-quartile range is "larger"?  Alternatively, you may have encountered a situation where a certain size of dataset was "enough" or "not enough" and can share the reasons why KDE worked/didn't work in that specific case?  Criteria to assess accuracy:  If you had two samplesfrom the same population and you do a KDE on each one, then the two resulting densities are very similar.
60192
I have used R to form a 95% prediction interval for the number of endemic species on an island. My lower bound is negative – is that mathematically sound?  In the linear model used in the prediction interval, the data used are:  Area Surface area of island, hectares DiscSC Distance from Santa Cruz, kilometres Elevation Elevation of higher point in metres and it is coded as such:   selected.model &lt;- lm(ES ~ Area + Elevation + DistSC + I(Elevation^2)                      + (Elevation:DistSC) + (A‌​rea:Elevation))    and stepwise regression was performed to find this "best" model   I'm not exactly sure how a prediction interval works. I just want to make sure it is OK. Obviously a negative number of species is incorrect, but I know it takes into account the uncertainty of the mean as well as data scatter.
100442
According to linear regression formula, assuming the predictors are nonrandom, $${\rm var}[\hat{\beta}]=MSE *(X'X)^{-1}$$ where MSE is an unbiased estimator of the population variance.  Intuitively, when the number of predictors (p + 1) is greater than the number of data, the regression model is overfitting the data, variance of $\hat{\beta}$ should be big.  However, according to the formula, MSE = 0 and we have $${\rm var}[\hat{\beta}]=MSE *(X'X)^{-1} = 0$$  What goes wrong? There must be some implicit assumption with this formula that failed in this scenario.