I created a model for two different datasets that have different scales. When checking which one performed better, I am struggling with figuring out the best methodology. My top choices right now are R^2, RMSE, and variance. However, R^2 seems to be the only option that works best in my opinion because it doesn't seem to take into account the scale of the data but wondering if there are options such as normalizing the numbers etc.   Below is a sample situation:        Date      Group      Actual_Values       Predicted_Values 2020-04-24          A              0.181                  0.140 2020-04-27          A              0.190                  0.127 2020-04-28          A              0.188                  0.126 2020-04-29          A              0.188                  0.123 2020-04-30          A              0.185                  0.122 2020-05-01          A              0.178                  0.124 2020-05-04          A              0.182                  0.118 2020-05-05          A              0.180                  0.120 2020-05-06          A              0.178                  0.120 2020-05-07          A              0.180                  0.121 2020-05-08          A              0.184                  0.124  2020-04-24          B              86.24                  86.25 2020-04-27          B              94.00                  86.65 2020-04-28          B              98.44                  86.88 2020-04-29          B              107.3                  87.05 2020-04-30          B              94.26                  87.27 2020-05-01          B              96.71                  87.42 2020-05-04          B              92.28                  87.59 2020-05-05          B              90.97                  87.73 2020-05-06          B              87.47                  87.80 2020-05-07          B              88.93                  87.90 2020-05-08          B              92.41                  87.93   From this dataset I tested the following code:  library(ModelMetrics) with_rmse&lt;- historical_results %&gt;%   group_by(Group) %&gt;%   summarise(     RMSE = rmse(Predicted_Values, Actual_Values)     ,R2 = cor(Predicted_Values, Actual_Values)^2     ,variance = abs(var(Predicted_Values, Actual_Values))   )  Group           RMSE            R2      variance     A     0.05929241    0.02067078  3.548883e-06        B       8.461625    0.02991999     0.5624356   So for example, the above results like 8.4 for RMSE may actually be good but looks bad just because the original values were higher than A. Any help would be great, thanks!
123905
I created a model for two different datasets that have different scales. When checking which one performed better, I am struggling with figuring out the best methodology. My top choices right now are R^2, RMSE, and variance. However, R^2 seems to be the only option that works best in my opinion because it doesn't seem to take into account the scale of the data but wondering if there are options such as normalizing the numbers etc.   Below is a sample situation:        Date      Group      Actual_Values       Predicted_Values 2020-04-24          A              0.181                  0.140 2020-04-27          A              0.190                  0.127 2020-04-28          A              0.188                  0.126 2020-04-29          A              0.188                  0.123 2020-04-30          A              0.185                  0.122 2020-05-01          A              0.178                  0.124 2020-05-04          A              0.182                  0.118 2020-05-05          A              0.180                  0.120 2020-05-06          A              0.178                  0.120 2020-05-07          A              0.180                  0.121 2020-05-08          A              0.184                  0.124  2020-04-24          B              86.24                  86.25 2020-04-27          B              94.00                  86.65 2020-04-28          B              98.44                  86.88 2020-04-29          B              107.3                  87.05 2020-04-30          B              94.26                  87.27 2020-05-01          B              96.71                  87.42 2020-05-04          B              92.28                  87.59 2020-05-05          B              90.97                  87.73 2020-05-06          B              87.47                  87.80 2020-05-07          B              88.93                  87.90 2020-05-08          B              92.41                  87.93   From this dataset I tested the following code:  library(ModelMetrics) with_rmse&lt;- historical_results %&gt;%   group_by(Group) %&gt;%   summarise(     RMSE = rmse(Predicted_Values, Actual_Values)     ,R2 = cor(Predicted_Values, Actual_Values)^2     ,variance = abs(var(Predicted_Values, Actual_Values))   )  Group           RMSE            R2      variance     A     0.05929241    0.02067078  3.548883e-06        B       8.461625    0.02991999     0.5624356   So for example, the above results like 8.4 for RMSE may actually be good but looks bad just because the original values were higher than A. Any help would be great, thanks! This is just an example - disregard the bad R2 :)
117823
I've been doing a machine learning competition where they use RMSLE (Root Mean Squared Logarithmic Error) to evaluate the performance predicting the sale price of a category of equipment. The problem is I'm not sure how to interpret the success of my final result.   For example if I achieved a RMSLE of $1.052$ could I raise it the the exponential power $e$ and interpret it like rmse? (ie. $e^{1.052}=2.863=RMSE$)?  Could I then say that my predictions were $\pm \$2.863$ on average from the the actual prices? Or is there a better way to interpret the metric? Or can the metric even be interpreted at all with the exception of comparing to the other RMSLEs of other models?
2647
What is the $R^2$  value given in the summary of a coxph model in R? For example,  Rsquare= 0.186   (max possible= 0.991 )   I foolishly included it a manuscript as an $R^2$ value and the reviewer jumped on it saying he wasn't aware of an analogue of the $R^2$  statistic from the classic linear regression being developed for the Cox model and if there was one please provide a reference. Any help would be great!
49454
If adjusted R squared is superior to R squared, then why do statistical software continue to report the latter? Is there any kind of situation when a researcher may prefer to use R squared instead of adjusted R squared?
48259
RMSE (Root mean square error) and SD (Standard deviation) have similar formulas.   says     The only difference is that you divide by  $n$ and not  $n−1$ since   you are not subtracting the sample mean here. The RMSE would then   correspond to  $\sigma$ . Therefore, the population RMSE is  $\sigma$   and you want a CI for that.   So I want to know whether RMSE and SD are the same. Also, I want reference about it.
76018
I'm working on a classification problem and I have a very high F1 baseline of 85%. I have trained three classification models and I want to know which one is the best. How can I do so?   I tried two ways:   To compare each model against the baseline using paired t-test. So I have tests like:    baseline vs. model 1 baseline vs. model 2 baseline vs. model 3   That tells me that only model 1 is significantly higher than the baseline and so I concluded that model 1 is the best. Is this a valid methodology given that usually classification models are compared against baselines?  To compare all models in one fell swoop with one-way ANOVA. So I entered the information of models 1-3 and the baseline which gave me a p-value of 0.02 indicating that there is a difference in means. Yet, with pairwise post-hoc tests, there is no significance between any of the pairs.   Which method is the correct one?
72778
I've been using the  in R to build predictive models for classification and regression.  Caret provides a unified interface to tune model hyper-parameters by cross validation or boot strapping.  For example, if you are building a simple 'nearest neighbors' model for classification, how many neighbors should you use?  2? 10? 100? Caret helps you answer this question by re-sampling your data, trying different parameters, and then aggregating the results to decide which yield the best predictive accuracy.  I like this approach because it is provides a robust methodology for choosing model hyper-parameters, and once you've chosen the final hyper-parameters it provides a cross-validated estimate of how 'good' the model is, using accuracy for classification models and RMSE for regression models.  I now have some time-series data that I want to build a regression model for, probably using a random forest. What is a good technique to assess the predictive accuracy of my model, given the nature of the data? If random forests don't really apply to time series data, what's the best way to build an accurate ensemble model for time series analysis?
116242
I want to measure the accuracy in neural network that performs regression. I have two outputs. Is the root-mean-square deviation ( ) the right way to go?   From wikipedia     RMSD is a good measure of accuracy[...]   How do I use it? I want to show how the accuracy is changing over epochs.  In classification measuring accuracy is simple. $$ accuracy = \frac{TP+FN}{TP+FP+TN+FN} $$
29192
I'm currently facing some issue in determining what statistical test might be best to adopt to determine whether or not my three datasets are significantly different from each other. I have the following datasets of probabilities observed within my Experiment: prob355 = [0.06672, 0.07473, 0.14813, 0.25516]; prob532 = [0.07226, 0.08718, 0.09631, 0.14276]; prob1064 = [0.05979, 0.09020, 0.10891, 0.11307]; Within these datasets, each entry corresponds to a probability obtained under certain experimental conditions such that within the dataset the conditions for prob355(1) ≠ prob355(2) prob355(3) ≠ prob355(4). Between the datasets they have been ordered such that prob355(1) = prob532(1) = prob1064(1). These datasets are accompanied by the following errors/standarddeviation/noise in the observed probabilities: model_error_355 = [0.03255, 0.03986, 0.01321, 0.01928]; model_error_532 = [0.01370, 0.00880, 0.00856, 0.04257]; model_error_1064 = [0.03150, 0.02441, 0.02307, 0.02059]; Within these datasets, each entry corresponds to the same entry as in the prob355, prob532 or prob1064. Basically, what I'm looking for is to compare each of the overall datasets (whilst taking the standard deviation in each value into account) amongst each other and determine if overall these datasets are statistically different from each other. Or if possible I would like to find out which of the values are significantly different from each other. The goal of this comparison for the overall datasets would be prove the hypothesis that these values are not significantly different from one another. Secondly, the goal would be to determine if for each of the individual conditions (eg. prob355(1) = prob532(1) = prob1064(1)) the values are significantly different or not. Currently, I'm using Matlab to run my analysis but I'm not bound to this software to run this analysis as long as the alternative is open acces :) So, If anyone knows a good way of making this comparison I would be very happy to hear it. Thanks!
118926
I'm doing a project in predicting default risk for SBA loans using R. My data has 187,986 positives and only 17796 negatives (9.46%). Clearly this is an imbalanced dataset. That's why when I run accuracy rate checking for my model using probability cutoff value=0.5, the accuracy result is way too low as below,  &gt; mytestset$LoanStatus_PIF&lt;-ifelse(mytestset$LoanStatus_PIF=="PIF",1,0) &gt; pred&lt;-predict(final,newdata=mytestset,type="response") &gt; y_pred_num&lt;-ifelse(pred&gt;0.5,1,0) &gt; mean(y_pred_num==mytestset$LoanStatus_PIF)   Result: [1] 0.009682553  When I change the cutoff value to 0.998 and up, the result is way much impressive:  &gt; y_pred_num&lt;-ifelse(pred&gt;0.5,1,0) &gt; mean(y_pred_num==mytestset$LoanStatus_PIF) Result:  [1] 0.9997752   Question: 1. Will my model be rejected because I manually choose the cut-off value?   Does 0.999 make sense for a cutoff value? As common cut-off value is usually 0.5. Thanks !
43054
I saw an example on a blog where a k nearest neighbors algorithm was run to predict responses on a test data set.The MSE(mean squared error) of the predictions was larger than the variance of the response variable in the test set. Shouldn't the MSE always be less than the variance of the response variable as SSTO(total sum of squares) = RSS(model sum of squares) + SSE(error sum of squares).
5647
Let's have some linear model, for example just simple ANOVA:  # data generation set.seed(1.234)                       Ng &lt;- c(41, 37, 42)                     data &lt;- rnorm(sum(Ng), mean = rep(c(-1, 0, 1), Ng), sd = 1)       fact &lt;- as.factor(rep(LETTERS[1:3], Ng))   m1 = lm(data ~ 0 + fact) summary(m1)   Result is as follows:  Call: lm(formula = data ~ 0 + fact)  Residuals:      Min       1Q   Median       3Q      Max  -2.30047 -0.60414 -0.04078  0.54316  2.25323   Coefficients:       Estimate Std. Error t value Pr(&gt;|t|)     factA  -0.9142     0.1388  -6.588 1.34e-09 *** factB   0.1484     0.1461   1.016    0.312     factC   1.0990     0.1371   8.015 9.25e-13 *** --- Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1   Residual standard error: 0.8886 on 117 degrees of freedom Multiple R-squared: 0.4816,     Adjusted R-squared: 0.4683  F-statistic: 36.23 on 3 and 117 DF,  p-value: &lt; 2.2e-16    Now I try two different methods to estimate confidence interval of these parameters  c = coef(summary(m1))  # 1st method: CI limits from SE, assuming normal distribution cbind(low = c[,1] - qnorm(p = 0.975) * c[,2],      high = c[,1] + qnorm(p = 0.975) * c[,2])  # 2nd method confint(m1)   Questions:   What is the distribution of estimated linear regression coefficients? Normal or $t$? Why do both methods yield different results? Assuming normal distribution and correct SE, I'd expect both methods to have the same result.   Thank you very much!  data ~ 0 + fact  EDIT after an answer:  The answer is exact, this will give exactly the same result as confint(m1)!  # 3rd method cbind(low = c[,1] - qt(p = 0.975, df = sum(Ng) - 3) * c[,2],      high = c[,1] + qt(p = 0.975, df = sum(Ng) - 3) * c[,2])
75064
I'm trying to identify the best model to predict the prices of automobiles, using the prices and features available on automobile classified advertisement sites.  For this I used a couple of models from the scikit-learn library and neural network models from pybrain and neurolab. The approach I used so far is to run a fixed amount of data through some models(machine learning algorithms) and compare there $R^2$ values which was calculated with the scikit-learn metrics module.   Is $R^2$ a good method to compare the performance of different models? Although I got quite acceptable results for models such as Elastic net and Random forests I got very poor $R^2$ values for Neural network models,so is $R^2$ an appropriate method for evaluating neural networks (or non-linear methods)?
117574
I want to regress a variable $y$ onto $x,x^2,\ldots,x^5$. Should I do this using raw or orthogonal polynomials? I looked at the question on the site that deals with these, but I don't really understand what's the difference between using them. Why can't I just do a &quot;normal&quot; regression to get the coefficients $\beta_i$ of $y=\sum_{i=0}^5 \beta_i x^i$ (along with p-values and all the other nice stuff) and instead have to worry whether using raw or orthogonal polynomials? This choice seems to me to be outside the scope of what I want to do. In the stat book I'm currently reading (ISLR by Tibshirani et al) these things weren't mentioned. Actually, they were downplayed in a way. The reason is, AFAIK, that in the lm() function in R, using y ~ poly(x, 2) amounts to using orthogonal polynomials and using y ~ x + I(x^2) amounts to using raw ones. But on pp. 116 the authors say that we use the first option because the latter is &quot;cumbersome&quot; which leaves no indication that these commands actually do two completely different things (and have different outputs as a consequence). (third question) Why would the authors of ISLR confuse their readers like that?
104888
Let's say I have a data set (with a binary DV) which I used a generalized linear mixed-effects model on. I am currently trying to run a simulation-based power analysis to determine the sample size I should try to collect for a replication attempt.   I know about the  function for fitted lme4 model objects, which will create simulated responses that follow the size and structure of the actual dataset.  My question is, how can I use this fitted model object to run an a-priori power analysis for my replication?. In other words, is there an easy way to extract the necessary information from the simulate function, vary the sample size, then to simulate responses?  EDIT:  Here is a simple example with ten subjects that should give me a good starting place. The data here are totally fake so the effects are not close to what I have in the pilot data.  The two variables of interest are age in months (between subjects) and whether the stimulus is "similar" or "dissimilar" (within subjects). Each individual is exposed to three items within each similarity group, for a total of six binary responses for each subject. As seen in the model, I am interested in a possible interaction between age and similarity and include a random slope of similarity to do this.   Let me know if I need to modify anything/provide more information to make things easier.   set.seed(28)  fakeData &lt;- data.frame(ID = rep(1:10, each = 6),                        months = rep(round(runif(10, min = 60, max = 80)), each = 18),                        similarity = rep(c("Sim", "Dissim"), each = 3),                        response = sample(c(0, 1), 60, replace = TRUE))  fakeModel &lt;- glmer(response ~ months * similarity + (similarity | ID), family = binomial, data = fakeData) summary(fakeModel)
35464
Let us look at some sample data for 5 hypothetical subjects.  library(emmeans) library(lme4)  # generate some sample data # condition (Placebo, Treatment) # type (some factor, e.g. two different medications) # value (outcome of interest) # subject  data &lt;- data.frame(condition = as.factor(rep(c(0,1), times = 10)), value = rnorm(20), type = rep(c("A","B"), each = 10), subject = as.factor(rep(rep(1:5, each = 2), times = 2)))   Now we like to conduct a mixed model and post-hoc tests with emmeans, where we test conditions per type in a pairwise fashion:  # caluculate a mixed model fit_data &lt;- lmer(value ~ condition*type + (1|subject), data = data)  # emmeans post-hoc test emmeans(fit_data, pairwise ~ condition | type)   So for each type (A or B), I have paired data (condition 0 has 5 data points and condition 1 has 5 data points). If I would do a paired t-test like so:  # paired t-test t.test(data[data$condition==0 &amp; data$type=="A","value"], data[data$condition==1 &amp; data$type=="A","value"], paired = TRUE) t.test(data[data$condition==0 &amp; data$type=="B","value"], data[data$condition==1 &amp; data$type=="B","value"], paired = TRUE)   I would get degrees of freedom of 4 for the paired t-test, but emmeans says the degrees of freedom are 12. Why is there this huge difference? If the emmeans package also would use df = 4, then the p-values would also be more comparable.
18489
Let's say I perform linear regression on some data that produces the following $R^2$: $\text{RSS} = 1966815.13$ $\text{TSS} = 2145213.91$ $R^2 =  0.083$ Now let's say I bucket (take the average of successive equal size windows) the data and perform the exact same linear regression. This time the $R^2$ is: $\text{RSS} = 1187.56$ $\text{TSS} = 4758.32$ $R^2 =  0.75$ What have I just done here? I clearly haven't improved the explanatory power of the model over the original data, yet I've hacked the $R^2$ by averaging. What's the bias / variance argument of why this occurs? A graphical representation:  Edit: I've reduced the total variance of the data, but the bias remains unaffected. When (if ever) is this a valid statistical approach?