I posted this question on stack.exchange but was recommended to move here. I am developing model predictions using pls and h2o packages which lead to 2 models: pls.model and h2o.model. The R-square (square of pearson correlation) and RMSE for each round of cross-validation are shown below:  R2:  i     R2.PLS     R2.H2O 1   1 0.4415108 0.6232292 2   2 0.3754088 0.6056992 3   3 0.4267580 0.6204750 4   4 0.3505282 0.6062691 5   5 0.2870766 0.5344183 6   6 0.3858786 0.5794828 7   7 0.3449946 0.5692314 8   8 0.2974582 0.5522208 9   9 0.3446449 0.5694339 10 10 0.3987684 0.5561757   RMSE:  i  rmse.pls rmse.h2o 1   1  8.839967 40.99896 2   2  9.347349 29.94260 3   3  4.240366 14.75890 4   4 17.901563 29.89181 5   5  4.686803 66.04993 6   6 31.717909 10.28799 7   7  2.066342 32.74828 8   8 15.979214 21.05928 9   9 19.454079 10.88551 10 10 27.039400 68.27017   I am unable to explain why pls.model has lower R2 but lower error while h2o.model has higher R2 but high error. I checked the scatter plot but no non-linear pattern appear. Would you have any thought of this? And in this case, what should be the better model?  Thanks Phuong
121083
Why in literature usually the common accuracy measures like MAD, MSE, RMSE, MAPE ... are used. Why not using the $R^2$ (coefficient of determination)?  I was thinking about the difference: By using the MSE i can compare the average of the forecast. And when using $R^2$ I'll get a information about the variance.  Why the comparison of averages is mostly commonly used? Can anybody give me an hint?
67239
This section of Introduction to Statistical Learning in R (page 19 in v6, statement 2.3) is motivating the difference between reducible and irreducible error (that is noted by $\epsilon$ and has mean zero).  Consider a given estimate $\hat{f}$ and a set of predictors $X$, which yields the prediction $\hat{Y} = \hat{f}(X)$. Assume for a moment that both $\hat{f}$ and $\hat{X}$ are fixed. Then, it is easy to show that $E(Y - \hat{Y})^2 = E[f(X) + \epsilon - \hat{f}(X)]^2$ $= [f(X) - \hat{f}(X)]^2 + Var(\epsilon)$  I'm having trouble getting the intermediate steps here. I understand that the expectation of the error in prediction should involve the variance of $\epsilon$ but I'd like to understand the proof. I expanded to $E(Y - \hat{Y})^2 = E[f(X) - \hat{f}(X)]^2 + 2E(\epsilon)E[f(X) - \hat{f}(X)] + E(\epsilon)^2$ and I see that I have $E(\epsilon)$ and $E(\epsilon)^2$ terms that could lead to $Var(\epsilon)$, but I'm stuck trying to fit some of the basic expected value and variance manipulations to it.
72778
I've been using the  in R to build predictive models for classification and regression.  Caret provides a unified interface to tune model hyper-parameters by cross validation or boot strapping.  For example, if you are building a simple 'nearest neighbors' model for classification, how many neighbors should you use?  2? 10? 100? Caret helps you answer this question by re-sampling your data, trying different parameters, and then aggregating the results to decide which yield the best predictive accuracy.  I like this approach because it is provides a robust methodology for choosing model hyper-parameters, and once you've chosen the final hyper-parameters it provides a cross-validated estimate of how 'good' the model is, using accuracy for classification models and RMSE for regression models.  I now have some time-series data that I want to build a regression model for, probably using a random forest. What is a good technique to assess the predictive accuracy of my model, given the nature of the data? If random forests don't really apply to time series data, what's the best way to build an accurate ensemble model for time series analysis?
11367
I performed principal component analysis (PCA) with R using two different functions (prcomp and princomp) and observed that the PCA scores differed in sign. How can it be?  Consider this:  set.seed(999) prcomp(data.frame(1:10,rnorm(10)))$x              PC1        PC2  [1,] -4.508620 -0.2567655  [2,] -3.373772 -1.1369417  [3,] -2.679669  1.0903445  [4,] -1.615837  0.7108631  [5,] -0.548879  0.3093389  [6,]  0.481756  0.1639112  [7,]  1.656178 -0.9952875  [8,]  2.560345 -0.2490548  [9,]  3.508442  0.1874520 [10,]  4.520055  0.1761397  set.seed(999) princomp(data.frame(1:10,rnorm(10)))$scores          Comp.1     Comp.2  [1,]  4.508620  0.2567655  [2,]  3.373772  1.1369417  [3,]  2.679669 -1.0903445  [4,]  1.615837 -0.7108631  [5,]  0.548879 -0.3093389  [6,] -0.481756 -0.1639112  [7,] -1.656178  0.9952875  [8,] -2.560345  0.2490548  [9,] -3.508442 -0.1874520 [10,] -4.520055 -0.1761397   Why do the signs (+/-) differ for the two analyses? If I was then using principal components PC1 and PC2 as predictors in a regression, i.e. lm(y ~ PC1 + PC2), this would completely change my understanding of the effect of the two variables on y depending on which method I used! How could I then say that PC1 has e.g. a positive effect on y and PC2 has e.g. a negative effect on y?    In addition: If the sign of PCA components is meaningless, is this true for factor analysis (FA) as well? Is it acceptable to flip (reverse) the sign of individual PCA/FA component scores (or of loadings, as a column of loading matrix)?
43054
I saw an example on a blog where a k nearest neighbors algorithm was run to predict responses on a test data set.The MSE(mean squared error) of the predictions was larger than the variance of the response variable in the test set. Shouldn't the MSE always be less than the variance of the response variable as SSTO(total sum of squares) = RSS(model sum of squares) + SSE(error sum of squares).
24222
I have created a multiple linear regression model to predict prices and about 17% of the predicted prices have come out to be negative.  Is there a way to correct for this error, or does it mean that the independent variables are not good predictors?  The R^2 coefficient is 0.865.
7003
I am observing strange patterns in residuals for my data:   [EDIT] Here are the partial regression plots for the two variables:      [EDIT2] Added the PP Plot   The distribution  seems to be doing fine (see below) but I have no clue where this straight line might be coming from. Any ideas?   [UPDATE 31.07]  It turns out you were absolutely right, I had cases where the retweet count was indeed 0 and these ~ 15 cases resulted in those strange residual patterns.   The residuals look much better now:   I've also included the partial regressions with a loess line.
8286
Given three variables, y and x, which are positive continuous, and z, which is categorical, I have two candidate models given by:  fit.me &lt;- lmer( y ~ 1 + x + ( 1 + x | factor(z) ) )   and  fit.fe &lt;- lm( y ~ 1 + x )   I hope to compare these models to determine which model is more appropriate. It seems to me that in some sense fit.fe is nested within fit.me. Typically, when this general scenario holds, a chi-squared test can be performed. In R, we can perform this test with the following command,  anova(fit.fe,fit.me)   When both models contain random-effects (generated by lmer from the lme4 package), the anova() command works fine. Owing to boundary parameters, it is normally advisable to test the resulting Chi-Square statistic via simulation, nonetheless, we can still use the statistic in the simulation procedure.  When both models contain only fixed-effects, this approach---and, the associated anova() command---work fine.  However, when one model contains random effects and the reduced model contains only fixed-effects, as in the above scenario, the anova() command doesn't work.  More specifically, I get the following error:   &gt; anova(fit.fe, fit.me)  Error: $ operator not defined for this S4 class   Is there anything wrong with using the Chi-Square approach from above (with simulation)? Or is this simply a problem of anova() not knowing how to deal with linear models generated by different functions?  In other words, would it be appropriate to manually generate the Chi-Square statistic derived from the models? If so, what are the appropriate degrees of freedom for comparing these models? By my reckoning:  $$ F = \frac{\left((SSE_{reduced}-SSE_{full})/(p-k)\right)}{\left((SSE_{full})/(n-p-1)\right)} \sim F_{p-k,n-p-1} $$  We are estimating two parameters in the fixed effects model (slope and intercept) and two more parameters (variance parameters for the random slope and random intercept) in the mixed-effects model. Typically, the intercept parameter isn't counted in the degrees of freedom computation, so that implies that $k=1$ and $p=k+2=3$; having said that I'm not sure if the variance parameters for the random-effects parameters should be included in the degrees of freedom computation; the variance estimates for fixed-effect parameters are not considered, but I believe that to be because the parameter estimates for fixed effects are assumed to be unknown constants whilst they are considered to be unknowable random variables for mixed effects. I would appreciate some assistance on this issue.  Finally, does anybody have a more appropriate (R-based) solution to comparing these models?
120636
I seem to have confused myself trying to understand if a $r$-squared value also has a $p$-value.  As I understand it, in linear correlation with a set of data points $r$ can have a value ranging from $-1$ to $1$ and this value, whatever it is, can have a $p$-value which shows if $r$ is significantly different from $0$ (i.e., if there is a linear correlation between the two variables).  Moving on to linear regression, a function can be fitted to the data, described by the equation $Y = a + bX$. $a$ and $b$  (intercept and slope) also have $p$-values to show if they are significantly different from $0$.  Assuming I so far have understood everything correct, are the $p$-value for $r$ and the $p$-value for $b$ just the same thing? Is it then correct to say that it is not $r$-squared that has a $p$-value but rather $r$ or $b$ that does?
117823
I've been doing a machine learning competition where they use RMSLE (Root Mean Squared Logarithmic Error) to evaluate the performance predicting the sale price of a category of equipment. The problem is I'm not sure how to interpret the success of my final result.   For example if I achieved a RMSLE of $1.052$ could I raise it the the exponential power $e$ and interpret it like rmse? (ie. $e^{1.052}=2.863=RMSE$)?  Could I then say that my predictions were $\pm \$2.863$ on average from the the actual prices? Or is there a better way to interpret the metric? Or can the metric even be interpreted at all with the exception of comparing to the other RMSLEs of other models?
117720
I have build regression model focused on association between physical activity and fat mass. The model is as follows: model &lt;- lm(FMI_W ~ IG_WD + SEX + MOTHER_BMI_W + MOTHER_EDUCATION_CATEGORIES, data = df_adolescent)  Where FMI_W = Fat mass index winsorized, IG_WD = novel metric from accelerometers which is reflecting negative drop in intensity of physical activity as time increase and is always negative. However this model is violating the assumptions of normality and homogeneity of residuals. As result I decided to log transform the respons variable as it improves residuals of the model. But now I am having problems with interpretation. The model and results are following: &gt; model &lt;- lm(log(FMI_W) ~ IG_WD + SEX + MOTHER_BMI_W + MOTHER_EDUCATION_CATEGORIES, data = df_adolescent) &gt; summary(model)  Call: lm(formula = log(FMI_W) ~ IG_WD + SEX + MOTHER_BMI_W + MOTHER_EDUCATION_CATEGORIES,      data = df_adolescent)  Residuals:      Min       1Q   Median       3Q      Max  -1.18164 -0.32438 -0.01599  0.28784  1.20763   Coefficients:                              Estimate Std. Error t value Pr(&gt;|t|)     (Intercept)                 -0.601001   0.326940  -1.838   0.0669 .   IG_WD                       -0.414468   0.129461  -3.201   0.0015 **  SEX                          0.520450   0.049994  10.410  &lt; 2e-16 *** MOTHER_BMI_W                 0.030605   0.006954   4.401 1.45e-05 *** MOTHER_EDUCATION_CATEGORIES  0.010488   0.050510   0.208   0.8356     --- Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1  Residual standard error: 0.4389 on 333 degrees of freedom Multiple R-squared:  0.3521,    Adjusted R-squared:  0.3443  F-statistic: 45.23 on 4 and 333 DF,  p-value: &lt; 2.2e-16  Could you please help me with back transformation of estimate of IG_WD. For me it doesn't make sence to use exp(-0.414468) = 0.6606917 as the exponential result is positive value but according to what we know about physical activity it should decrease fat mass. If I use the model with violating the assumptions so without the log transformation I get estimate around -1.9 for IG_WD. Also I have tried BoxCox transformation as is in this post   Actualization: The FMI_W is a shortcut for fat mass index which is equvivalent of $FMI = fat mass(kg)/height^2(m)$. _W stands for winsorzed data to reduce outliers. Summary of FMI_W: &gt; summary(df_adolescent$FMI_W)    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    1.532   2.648   4.111   4.634   6.202   9.875   Distrubution of FMI_W can be seen on next plot:  As suggested in the comments. Would be the glm with gamma family link = &quot;identical&quot; appropriate?
80211
I repeatedly read that one cannot use an F-test to compare two models that are not nested. Usually I have to nested models $$ m: y=X\beta+Z\gamma+\epsilon \text{, and } m_r:y=X\beta+u $$ The test-statistic usually looks something like $$ F(y)=\frac{RSS_r-RSS}{RSS}\cdot\frac{L}{N-K-L} $$ where $rRSS$ and $RSS$ is the residual sum of squares of the restricted and ful model respectively. $N,K,L$ are the number of samples, variables in X and variables in Z.  However, I don't see why I couldn't use the residual sum of squares of any two models $m$ and $m'$ and compute $$ F(y)=\frac{RSS_{m'}-RSS_m}{RSS_m}\cdot\frac{variables(m)-variables(m')}{N-variables(m)}, $$ assuming $variables(m)\geq variables(m')$ w.l.o.g.  Would be great to hear your thoughts on this! Thanks in advance
2647
What is the $R^2$  value given in the summary of a coxph model in R? For example,  Rsquare= 0.186   (max possible= 0.991 )   I foolishly included it a manuscript as an $R^2$ value and the reviewer jumped on it saying he wasn't aware of an analogue of the $R^2$  statistic from the classic linear regression being developed for the Cox model and if there was one please provide a reference. Any help would be great!
82703
Could someone please explain the differences between the 3 fitting methods, method = c("CSS-ML", "ML", "CSS"), in Arima?  If I run the code below I get an error message, but if I specify method="ML" in Arima it runs fine.  So I was curious what the difference was between the 3 fitting methods.  Code with error:  library("fpp")  tsTrain &lt;- window(hsales,end=1989.99)  pvar&lt;-1:10 dvar&lt;-1:2 qvar&lt;-1:7  OrderGrid&lt;-expand.grid(pvar,dvar,qvar)  n &lt;- function(a,b,c) {Arima(tsTrain, order=c(a,b,c))} ModFit &lt;- do.call(Vectorize(n, SIMPLIFY=FALSE), unname(OrderGrid))   Fixed Code:  library("fpp")  tsTrain &lt;- window(hsales,end=1989.99)  pvar&lt;-1:10 dvar&lt;-1:2 qvar&lt;-1:7  OrderGrid&lt;-expand.grid(pvar,dvar,qvar)  n &lt;- function(a,b,c) {Arima(tsTrain, order=c(a,b,c),method="ML")} ModFit &lt;- do.call(Vectorize(n, SIMPLIFY=FALSE), unname(OrderGrid))
96392
I have a Cox proportional hazards model. Judging by Schoenfeld residual vs. time plots and corresponding tests for zero slope, there is clear violation of the PH assumption for several of the variables (too many to stratify).   If we take the single-covariate case for simplicity, the original model looks like:  $$ \hat\lambda(t) = \lambda_0(t)*\exp(X_1\hat\beta_{X_1}) $$  Given the violation of the PH assumption, I want to try fitting time-dependent coefficients for the problem variables. It seems intuitive to fit this:  $$ \hat\lambda(t) = \lambda_0(t)*\exp(X_1\hat\beta_{X_1}+X_1t*\hat\beta_{X_1t}) $$  which simplifies to  $$ \hat\lambda(t) = \lambda_0(t)*\exp(X_1(\hat\beta_{X_1}+t*\hat\beta_{X_1t})) $$  So, effectively, this would parametrically fit a coefficient $\hat\beta^\prime_{X_1}$ that is a linear function of time: $$ \hat\beta^\prime_{X_1} = \hat\beta_{X_1}+t*\hat\beta_{X_1t} $$  Before I could happily dance off into the sunset with this model, I learned it's invalid ( and ). But WHY is it invalid? Why do we have to construct a dataset with start-stop times instead?
111149
My understanding is that $R^2$ cannot be negative as it is the square of R. However I ran a simple linear regression in SPSS with a single independent variable and a dependent variable. My SPSS output give me a negative value for $R^2$. If I was to calculate this by hand from R then $R^2$ would be  positive. What has SPSS done to calculate this as negative?   R=-.395 R squared =-.156 B (un-standardized)=-1261.611   Code I've used:  DATASET ACTIVATE DataSet1.  REGRESSION /MISSING LISTWISE /STATISTICS COEFF OUTS R ANOVA             /CRITERIA=PIN(.05) POUT(.10) /NOORIGIN             /DEPENDENT valueP /METHOD=ENTER ageP   I get a negative value. Can anyone explain what this means?
33750
Source |       SS       df       MS              Number of obs =     300 -------------+------------------------------           F(  5,   294) =16580.50        Model |  1.1499e+10     5  2.2998e+09           Prob &gt; F      =  0.0000     Residual |  40778600.2   294  138702.722           R-squared     =  0.9965 -------------+------------------------------           Adj R-squared =  0.9964        Total |  1.1540e+10   299  38593905.2           Root MSE      =  372.43   Should I be worried about this- does your F statistic size matter?  How do i find out what the p value of my model is - is it the p value attached to _cons in the bottom of my coefficients?  what do i test this F stat against and what warrants significance?