Defining sMAPE as $$sMAPE=\frac{2}{h}\Sigma_{t=n+1}^{n+h}\frac{\left\lvert Y_t-\hat{Y_t}\right\rvert}{\left\lvert Y_t\right\rvert+\left\lvert\hat{Y_t}\right\rvert}*100$$ where $Y_t$ is the value of the time series at point $t$, $\hat{Y_t}$ the estimated forecast, $h$ the forecasting horizon and $n$ is the number of the data points available in-sample. My question is regarding what is the value/range of sMAPE where the developed model would be acceptable. I know for MAPE it is said to be proper model when the measure is less than 20%. Is 20% the threshold for sMPAE as well? Thank you in advance.
69362
I have built a forecasting model for a company. Since it is dedicated to practical usage, I prefer to use the relative error parameter (like MAPE, SMAPE, &amp; MASE) as a measurement for my model performance and display it on the dashboard (the percentage error is easier to understand by user). I found an interesting reference about the criteria of a good forecast result based on MAPE from Lewis (1982):  ...but in my case, I can't use MAPE for some dataset because there is some zero actual demand so it can't be used as denominator in MAPE. So, I use SMAPE &amp; MASE instead, but I can't find any good references about good forecast result criteria based on SMAPE &amp; MASE, nothing like above. So, I'd really appreciate any help/suggestion about it :), Thank You! Reference: Lewis, C. D., 1982. Industrial and Business Forecasting Methods. London: Butterworths.
9024
I have a question related to modeling short time-series. It is not a question , but how. What method would you recommend for modeling (very) short time-series (say of length $T \leq 20$)? By "best" I mean here the most robust one, that is the least prone to errors due the fact of limited numbers of observations. With short series single observations could influence the forecast, so the method should provide a cautious estimate of errors and possible variability connected to the forecast. I am generally interested in univariate time-series but it would be also interesting to know about other methods.
96392
I have a Cox proportional hazards model. Judging by Schoenfeld residual vs. time plots and corresponding tests for zero slope, there is clear violation of the PH assumption for several of the variables (too many to stratify).   If we take the single-covariate case for simplicity, the original model looks like:  $$ \hat\lambda(t) = \lambda_0(t)*\exp(X_1\hat\beta_{X_1}) $$  Given the violation of the PH assumption, I want to try fitting time-dependent coefficients for the problem variables. It seems intuitive to fit this:  $$ \hat\lambda(t) = \lambda_0(t)*\exp(X_1\hat\beta_{X_1}+X_1t*\hat\beta_{X_1t}) $$  which simplifies to  $$ \hat\lambda(t) = \lambda_0(t)*\exp(X_1(\hat\beta_{X_1}+t*\hat\beta_{X_1t})) $$  So, effectively, this would parametrically fit a coefficient $\hat\beta^\prime_{X_1}$ that is a linear function of time: $$ \hat\beta^\prime_{X_1} = \hat\beta_{X_1}+t*\hat\beta_{X_1t} $$  Before I could happily dance off into the sunset with this model, I learned it's invalid ( and ). But WHY is it invalid? Why do we have to construct a dataset with start-stop times instead?
116280
Suppose that you have a logistic model and the predicted probabilities of a $1$ all are in $[0.2,0.3]$. So the predictions would be "0" based on a threshold of $0.5$. What would be a good threshold to choose?
87295
I'm interested in a good reference for results concerning asymptotic properties of maximum likelihood estimators. Consider a model $\{f_n(\cdot \mid \theta): \theta \in \Theta, n \in \mathbb N\}$ where $f_n(\mathbf x \mid \theta)$ is an $n$-dimensional density, and $\hat \theta_n$ is the MLE based on a sample $X_1, \ldots, X_n$ from $f_n(\cdot \mid \theta_0)$ where $\theta_0$ is the "true" value of $\theta$. There are two irregularities I'm interested in.    The data $X_1, \ldots, X_n$ are not iid and, as a result, the Fisher information about $\theta$ accrues at a rate slower than $n$.  $\Theta$ is a bounded set, and with positive probability $\hat \theta_n$ lies on the boundary. The boundary corresponds to a "simpler" model, and so there is particular interest in whether or not $\theta_0$ lies on the boundary.    My particular questions are   Letting $J_n(\theta)$ denote the observed Fisher information corresponding to $\theta$, and suppose $\theta_0$ lies in the interior of $\Theta$. Under what conditions is $$\left[J_n(\hat \theta_n)\right]^{1/2}(\hat \theta_n - \theta_0)$$ asymptotically normal as $n \to \infty$? In particular, are the regularity conditions similar to the usual ones, with the relevant modification being $J_n(\hat \theta_n) \to \infty$ in some sense? Suppose instead that $\theta_0$ is on the boundary, and again recall that $\hat \theta_n = \theta_0$ happens with positive probability - for concreteness, in a mixed effects model $Y_{ij} = \mu + \beta_i + \epsilon_{ij}$ we can have $\hat \sigma_{\beta}^2 = 0$. Under what conditions does $\hat \theta_n \to \theta_0$ (almost surely or in probability) and under what conditions will $\hat \theta_n = \theta_0$ eventually (this probably fails for the mixed effects model, but corresponds to "oracle" properties for the LASSO and related estimators, so perhaps it is too much to ask for general results)?   Again, just a pointer to a text with results at this level of generality would be greatly appreciated.
86916
A probably very easy computational question, but I don't really understand how it's done:  I try to compute the point estimates of a normal distribution as $N \sim (\beta, \sigma)$. Using the method of moments, we get:  $$\frac{1}{n} \sum_{i=1}^{n} X_{i} = \hat{\beta}$$  $$\frac{1}{n} \sum_{i=1}^{n} X_{i}^{2} = \hat{\sigma}^2 + \hat{\beta}^2$$  Solving the two equations for the two parameters, we get:  $$\hat{\beta} = \overline{X} $$   and   $$\hat{\sigma}^2 = \frac{1}{n} \sum_{i=1}^{n} (X_{i} - \overline{X}_n)^2 $$  My question is: How do we get to the term of $\hat{\sigma}^2$? If I try to compute it, I get:  $$\hat{\sigma}^2 = \frac{1}{n} \sum_{i=1}^{n} X_{i}^2 - (\frac{1}{n} \sum_{i=1}^{n} X_{i} )^2 $$  Can someone show me how to simplify the last term to what is stated above and what "rule" or law" is used there? Since it is used so often, I really wanna understand this.
56697
What is your opinion about oversampling in classification in general, and the SMOTE algorithm in particular? Why would we not just apply a cost/penalty to adjust for imbalance in class data and any unbalanced cost of errors? For my purposes, accuracy of prediction to a future set of experimental units is the ultimate measure.  For reference, the SMOTE paper:
85242
I have a process which consists of a number of events and what is known is the timings between the events. What I'm trying to determine is a distribution that allows me to determine a likelyhood that a new sample fits the distibution.  The issue is mainly that if you have lots of samples you can approximate the result using a standard gaussian and use the mean and standard deviation. But if you only have a handful of samples, the gaussian does not accurately represent the situation.  From what I've read it is common to model waiting times using the gamma distribution. Looking at how the process evolves it looks like it matches well. The unknown is the scale parameter, since the shape parameter I think should be the number of samples. What I've worked out so far is that given the timings $X_1 ... X_N$ you can say:  $$ \sum_{n=1}^N X_n \sim \Gamma(N,\theta) $$  ($N$ is known and fixed)  However, $\theta$ is unknown, but the maximum likelihood parameter is the average of the $X_i$ (according to wikipedia anyway).   My question is, can I use this to estimate a distribution for $X_i$, that is, since the $X_i$ are independent:  $$ N X_i | \sum_n X_n \sim \Gamma(N, \tfrac{1}{N}\sum_n X_n) $$  Something else I've wondered about. Suppose I do have information about $\theta$, say a distribution. How can I incorporate this into the model?  Edit: Clarified that N is fixed.
4773
I've fitted my model and am trying to understand whether it's any good. I've calculated the recommended metrics to assess it ($R^2$/ AUC / accuracy / prediction error / etc) but do not know how to interpret them. In short, how do I tell if my model is any good based on the metric? Is an $R^2$ of 0.6 (for example) sufficient to let me proceed to draw inferences or base scientific/business decisions?     This question is intentionally broad, to cover a wide variety of situations that members frequently encounter; such questions could be closed as duplicates of this one. Edits to broaden the scope beyond the metrics mentioned here are welcome, as are additional answers - particularly those that offer insight about other classes of metrics.
226
On page 19 of the textbook  (by James, Witten, Hastie and Tibshirani--it is freely downloadable on the web, and very good), the following is stated:  Consider a given estimate $$\hat{Y} = \hat{f}(x)$$ Assume for a moment that both $$\hat{f}, X$$ are fixed. Then, it is easy to show that: $$\mathrm{E}(Y - \hat{Y})^2 = \mathrm{E}[f(X) + \epsilon - \hat{f}(X)]^2$$ $$ = [f(X) - \hat{f}(X)]^2 + \mathrm{Var}(\epsilon)$$  It is further explained that the first term represents the reducible error, and the second term represents the irreducible error. I am not fully understanding how the authors arrive at this answer. I worked through the calculations as follows: $$\mathrm{E}(Y - \hat{Y})^2 = \mathrm{E}[f(X) + \epsilon - \hat{f}(X)]^2$$ This simplifies to $[f(X) - \hat{f}(X) + \mathrm{E}[\epsilon]]^2 = [f(X) - \hat{f}(X)]^2$ assuming that $\mathrm{E}[\epsilon] = 0$. Where is the $\mathrm{Var}(x)$ indicated in the text coming from? Any suggestions would be greatly appreciated.
93781
I just have a question and I did not find clear explanation, hinge loss is upper bound of 0-1 loss, so does this mean upper bound regard to a single point loss such as 0 or 1 ?!. So, for example the 0-1 loss for set of points is 0 or 1, and for hinge loss we have continuous loss values from 0 to inf. So, by comparing the set of loss values for 0-1 loss and hinge loss we find the lower bound is 0 and upper bound is 1. I am right?!. This is the link: . Please see the page 3, the first paragraph under the title &quot;Conditional Φ-risk and classification calibration&quot;. Thanks
69288
I have trouble with the following problem:     We assume that X has a Exp(λ)-distribution with a unknowns value of λ.   We test H0: λ=0.2 versus H1: λ &lt; 0.2. We use X as test statistic.  a.   Calculate the critical area for alpha = 0.05 b. The measured X=10,   calculate the P-value   ans: [14.98, infinity) and p = 0.135    I know that the critical area is the collection of values for which H0 is rejected in favor of H1. In the case of a normal distribution I know how to calculate the critical region. Depending on the kind of alternate hypothesis and the significance level I can caluclate the Z-value. With the Z-value I can calculate the X.   Unfortunately I don't know how to do this with an exponential distribution. I have issues with visualising it, since a different λ gives a different curve.   Can I please get some feedback?  Thanks,  Ter
80564
I am unable to find the bias of the sample variance estimator in this problem. Unfortunately I keep coming up with   Question: Suppose that the true linear model is   $y = X_1\beta_1 + X_2\beta_2 + e$,   where $e \sim MVN(0,\sigma^2 I)$, $X_1: N \times p_1, \beta_1: p_1 \times 1, X_2: N \times p_2, \beta_2: p_2 \times 1$. But we wrongly fit the model  $y = X_1\beta_1+e$ and calculate the estimators of $\beta_1$ and $\sigma^2$ as:  $\hat{\beta}_1 = (X_1'X_1)^{-1}X_1'y$,  $s^2 = \large \frac{y'[I-X_1(X_1'X_1)^{-1}X_1']y}{N-p_1}$.  (a) Show that, in general, $\hat{\beta}_1$ is a biased estimator with  Bias$(\hat{\beta}_1) = (X_1'X_1)^{-1}X_1'X_2\hat{\beta}_2$  Under what condition on $X_1$ and $X_2$ is $\hat{\beta}_1$ unbiased?  (b) Show that, in general, $s^2$ is a biased estimator with   Bias$(s^2) = \frac{1}{N-p_1}\beta_2'X_2'[I-X_1(X_1'X_1)^{-1}X_1']X_2\beta_2$  Is the bias positive or negative? Under what condition on $\beta_2$ is $s^2$ unbiased?  Attempt:  For part (a):  Bias$(\hat{\beta}_1) = E((X_1'X_1)^{-1}X_1'y - \beta_1)$  $\hspace{15mm}= (X_1'X_1)^{-1}X_1'E(y) - \beta_1$  $\hspace{15mm}= (X_1'X_1)^{-1}X_1'(X_1\beta_1 + X_2\beta_2) - \beta_1$  $\hspace{15mm}= (X_1'X_1)^{-1}X_1'X_1\beta_1 + (X_1'X_1)^{-1}X_1'X_2\beta_2) - \beta_1$  $\hspace{15mm}= I\beta_1 + (X_1'X_1)^{-1}X_1'X_2\beta_2 - \beta_1$  $\hspace{15mm}= \beta_1 + (X_1'X_1)^{-1}X_1'X_2\beta_2 - \beta_1$  $\hspace{15mm}= (X_1'X_1)^{-1}X_1'X_2\beta_2$  If $X_1$ and $X_2$ are orthogonal, then $\hat{\beta}_1$ is unbiased.  For part (b), I am unable to figure out how to derive:  Bias$(s^2) = \frac{1}{N-p_1}\beta_2'X_2'[I-X_1(X_1'X_1)^{-1}X_1']X_2\beta_2$  I can get   $\frac{1}{N-p_1}\beta_1'X_1'+\beta_2'X_2'[I-X_1(X_1'X_1)^{-1}X_1']X_1\beta_1+X_2\beta_2$  after taking the expectation of y on both sides.   Any help would be much appreciated.
97543
Suppose I have the following model: $$Y = \mu + \epsilon = X\beta + \epsilon,$$ where $Y$ is $n \times 1$, $X$ is $n \times p$, $\beta$ is $p \times 1$, and $\epsilon$ is $n \times 1$. I assume that $\epsilon$ are independent with mean 0 and variance $\sigma^2I$. In OLS, the fitted values are $\hat{Y} = HY$, where $H = X(X^TX)^{-1}X^T$ is the $N \times N$ hat matrix. I want to find the MSE of $\hat{Y}$. By the bias-variance decomposition, I know that \begin{align*} MSE(\hat{Y}) &amp;= bias^2(\hat{Y}) + var(\hat{Y})\\ &amp;= (E[HY] - \mu)^T(E[HY] - \mu) + var(HY)\\ &amp;= (H\mu - \mu)^T(H\mu - \mu) + \sigma^2H\\ &amp;= 0 + \sigma^2H \end{align*} I'm confused by the dimension in the last step. The $bias^2$ term is a scalar. However, $var(\hat{Y})$ is an $N \times N$ matrix. How can one add a scalar to an $N \times N$ matrix where $N \neq 1$?
71336
For a prediction interval in linear regression you still use $\hat{E}[Y|x] = \hat{\beta_0}+\hat{\beta}_{1}x$ to generate the interval. You also use this to generate a confidence interval of $E[Y|x_0]$. What's the difference between the two?
21440
$X_1, X_2 \dots X_n$ represent independently observed Bernoulli random variables.  $Z_1, Z_2, \dots Z_n$ are unobserved.  $Z_i | \theta_i \sim N(\theta_i,1)$  $\theta_i \sim N(\epsilon, \sigma^2)$  $X_i = \begin{cases} 0 &amp;  Z_i \leq u \\ 1 &amp; Z_i &gt; u \end{cases}$ where $u$ is known.  Show that the EM sequence for estimation of $\epsilon$ is given by:  $\epsilon^{t+1} = \frac{1}{n} \sum_{i=1}^n E[Z_i|X_i, \epsilon^t, \sigma^2]$  $\Phi$ represents CDF of standard normal.  Attempt  $Z_i \sim N(\epsilon, \sigma^2+1)$   \begin{align*} f(X,Z| \epsilon, \sigma^2) &amp;= \prod_{i=1}^n p_i^{x_i} (1-p_i)^{1-x_i}\\ p_i &amp;= P(Z_i \leq u)\\ &amp;=\Phi(\frac{u-\epsilon}{\sqrt{\sigma^2+1}}) \end{align*}  Complete log likeihood = $\sum_{i=1}^nx_i \log p_i + (n-\sum_i x_i) \log(1-p_i)$  I am not able to proceed further.
56710
Given a random walk $x_t$,   $$x_t=x_{t-1}+\varepsilon_t,$$  consider estimating the slope coefficient $\beta$ in   $$x_t=\beta x_{t-1}+\varepsilon_t$$  by OLS.  question and the following answer noted that $\hat \beta^{OLS}$ has a skewed distribution. My question is,  Are any of OLS assumptions violated in the model $x_t=\beta x_{t-1}+\varepsilon_t$ given that $x_t$ is a random walk? If so, what are the violations?