Given the Bayesian paradigm, is there actually such a thing as a fixed effects model since Bayesians treat all parameters as random variables?
121101
In Bayesian data analysis, parameters are treated as random variables. This stems from the Bayesian subjective conceptualization of probability. But do Bayesians theoretically acknowledge that there is one true fixed parameter value out in the 'real world?'  It seems like the obvious answer is 'yes', because then trying to estimate the parameter would almost be nonsensical. An academic citation for this answer would be greatly appreciated.
77126
As a motivating example, consider this hypothesis: A friend and I were discussing how it might be the case that, the more one person is given control in the process of making a movie, the higher the variance of the quality of that film. The more hands that go into making a film, the safer of a bet it is—but the less likely it will be that it is great.  Here are some simulated data that could characterize this hypothesis:  set.seed(1839) x &lt;- runif(1000, 0, 100) y &lt;- rnorm(1000, 0, x) dat &lt;- data.frame(x, y)  library(ggplot2) ggplot(dat, aes(x = x, y = y)) +   geom_jitter() +   theme_light() +   labs(y = "Quality of Film",         x = "Amount of Artistic Control in One Person")     Note that here, I am not interested in predicting the expected value, but the expected variance of $y$ from $x$.  I am familiar with how one can create models for both location and dispersion in beta regression and how one can create models for both location and scale in conditionally heteroskedastic truncated regression.  However, is there a way to model conditional heteroskedasticity in an ordinary least squares regression like this? Are there regression models that exist where one can model the variance of $y$ as a function of $x$? When I try looking for these models, I usually end up with people wanting to use sandwich estimators or weighted least squares or something.  I don't want to treat heteroskedasticity as a problem with the data, but something to be explicitly modeled as a function of predictors $X$. Is there a way to do this with linear regression?
54761
In meta-analysis packages, both fixed effects and random effects models are available. How do one choose between these two models? Since one is assessing different studies, should one not choose random effects model all the time?
584
Is it ever valid to include a two-way interaction in a model without including the main effects?  What if your hypothesis is only about the interaction, do you still need to include the main effects?
20477
Some sources say likelihood function is not conditional probability, some say it is. This is very confusing to me.  According to most sources I have seen, the likelihood of a distribution with parameter $\theta$, should be a product of probability mass functions given $n$ samples of $x_i$:  $$L(\theta) = L(x_1,x_2,...,x_n;\theta) = \prod_{i=1}^n p(x_i;\theta)$$  For example in Logistic Regression, we use an optimization algorithm to maximize the likelihood function (Maximum Likelihood Estimation) to obtain the optimal parameters and therefore the final LR model. Given the $n$ training samples, which we assume to be independent from each other, we want to maximize the product of probabilities (or the joint probability mass functions). This seems quite obvious to me.  According to , "likelihood is not a probability and it is not a conditional probability". It also mentioned, "likelihood is a conditional probability only in Bayesian understanding of likelihood, i.e., if you assume that $\theta$ is a random variable."  I read about the different perspectives of treating a learning problem between frequentist and Bayesian.   According to a source, for Bayesian inference, we have a priori $P(\theta)$, likelihood $P(X|\theta)$, and we want to obtain the posterior $P(\theta|X)$, using Bayesian theorem:  $$P(\theta|X)=\dfrac{P(X|\theta) \times P(\theta)}{P(X)}$$   I'm not familiar with Bayesian Inference. How come $P(X|\theta)$ which is the distribution of the observed data conditional on its parameters, is also termed the likelihood? In , it says sometimes it is written $L(\theta|X)=p(X|\theta)$. What does this mean?  is there a difference between Frequentist and Bayesian's definitions on likelihood??  Thanks.    EDIT:  There are different ways of interpreting Bayes' theorem - Bayesian interpretation and Frequentist interpretation (See: ).
41984
I'm reading  to work on the foundations of my understanding of Bayesian analysis. In the first chapter, about normal models, I'm having some trouble working the posterior distribution by myself.  For the model with known variance and conjugate prior for the mean $N(0, \sigma^2)$, the derivation seems straightforward (pages 29 and 30):  $$\pi(\mu\mid D_n) \propto \mathrm{exp}\{-\mu^2/2\sigma^2\}\mathrm{exp}\{-[n(\bar{x}-\mu)^2 + s^2]/2\sigma^2\}$$ $$\propto \mathrm{exp}\left\{-\frac{\mu^2 + n\mu^2 - 2n\mu\bar{x} +n\bar{x}^2 + s^2}{2\sigma^2}\right\}$$  $s^2$ drops off because it does not depend on the $\mu$ parameter; $n\bar{x}^2$ is almost what we need to complete the square, but not quite so, so I guess we also drop it off and substitute it for the proper term:  $$\propto \mathrm{exp}\left\{-\frac{(n+1)\mu^2 - 2n\mu\bar{x}}{2\sigma^2}\right\}$$ $$\propto \mathrm{exp}\left\{-\frac{(n+1)\left(\mu^2 - \frac{2n\mu\bar{x}}{n+1} + \frac{n^2\bar{x}^2}{(n+1)^2}\right)}{2\sigma^2}\right\}$$ $$\propto \mathrm{exp}\left\{-\frac{(n+1)\left(\mu - \frac{n\bar{x}}{n+1}\right)^2}{2\sigma^2}\right\}$$  So far, so good. But when it comes to the unknown mean, unknown variance model with prior $N(0, \sigma^2)$ and $\frac{1}{\sigma^2} \sim Exp(1)$, we can't drop off the $n\bar{x}^2$ term anymore because now the denominator inside the exponential function depends on the parameter $\sigma^2$.  So I have come to a similar point deriving the joint posterior for $\mu$ and $\sigma^2$:  $$\propto \mathrm{exp}\left\{-\frac{2 + \mu^2 + n\mu^2 - 2n\mu\bar{x} +n\bar{x}^2 + s^2}{2\sigma^2}\right\}\sigma^{-(n+5)}$$ $$\propto \sigma^{-1/2}\mathrm{exp}\left\{-\frac{(n+1)\mu^2 - 2n\mu\bar{x} +n\bar{x}^2}{2\sigma^2}\right\}\mathrm{exp}\left\{-\frac{2+s^2}{2\sigma^2}\right\}(\sigma^2)^{-(\frac{n+2}{2})-1}$$  The part to the right is already the core of the Inverse-Gamma function derived in the book, but I can't see how to complete the square for the normal density function without adding another term to the second Inverse-Gamma parameter. Indeed, in the general case, this parameter value should also have a $\frac{n\bar{x}^2}{n+1}$ term (as, e.g., ).   So I decided to compare both solutions by simulation using Metropolis-Hastings to obtain posterior draws of both parameters. Surprisingly (or not), the book solution uses the correct formula for the scale parameter for the Inverse-Gamma, $\frac{2 + s^2}{2}$, and if we add the term based on the sample mean, the density curve does not quite fit (Edit: not really, see below) . What am I missing here?    Edit: I'm starting to think that the book might indeed be mistaken. Completing the square in the first exponential makes it necessary to add a $\frac{n\bar{x}^2}{n+1}$ term that can only go in the second exponential term. So I tried my hand again with simulations, adding a JAGS model with the same definition to compare to my homemade solution (which had a coding error when I posted the question), and now it seems that the correct formula for the second Inverse-Gamma parameter is indeed $\frac{2 + s^2 + \frac{n\bar{x}^2}{n+1}}{2}$. Here's a plot with the simulated and theoretical densities for the posterior distribution (data = rnorm(100, 50, 10)):
5175
I have read in the abstract of  that:     "The maximum likelihood (ML) procedure of Hartley aud Rao is modified by adapting a transformation from Patterson and Thompson which partitions the likelihood render normality into two parts, one being free of the fixed effects. Maximizing this part yields what are called restricted maximum likelihood (REML) estimators."   I also read in the abstract of  that REML:     "takes into account the loss in degrees of freedom resulting from estimating fixed effects."   Sadly I don't have access to the full text of those papers (and probably would not understand if I did).  Also, what are the advantages of REML vs. ML? Under what circumstances may REML be preferred over ML (or vice versa) when fitting a mixed effects model? Please give an explanation suitable for someone with a high-school (or just beyond) mathematics background!
20815
The  claims that likelihood and probability are distinct concepts.     In non-technical parlance, "likelihood" is usually a synonym for "probability," but in statistical usage there is a clear distinction in perspective: the number that is the probability of some observed outcomes given a set of parameter values is regarded as the likelihood of the set of parameter values given the observed outcomes.    Can someone give a more down-to-earth description of what this means?  In addition, some examples of how "probability" and "likelihood" disagree would be nice.
29152
For a simulation study I have to generate random variables that show a predefined (population) correlation to an existing variable $Y$.  I looked into the R packages copula and CDVine which can produce random multivariate distributions with a given dependency structure. It is, however, not possible to fix one of the resulting variables to an existing variable.  Any ideas and links to existing functions are appreciated!  &nbsp;  Conclusion: Two valid answers came up, with different solutions:   An R  by caracal, which calculates a random variable with an exact (sample) correlation to a predefined variable An R  I found myself, which calculates a random variable with a defined population correlation to a predefined variable     [@ttnphns' addition: I took the liberty to expand the question title from single fixed variable case to arbitrary number of fixed variables; i.e. how to generate a variable having predefined corretation(s) with some fixed, existing variable(s)]
102052
I am looking for uninformative priors for beta distribution to work with a binomial process (Hit/Miss). At first I thought about using $\alpha=1, \beta=1$ that generate an uniform PDF, or Jeffrey prior $\alpha=0.5, \beta=0.5$. But I am actually looking for priors that have the minimum effect on posterior results, and then I thought about using an improper prior of $\alpha=0, \beta=0$. The problem here is that my posterior distribution only works if I have at least one hit and one miss. To overcome this I then thought about using a very small constant, like $\alpha=0.0001, \beta=0.0001$, just to assure that posterior $\alpha$ and $\beta$ will be $&gt;0$.  Does anyone knows if this approach is acceptable? I see numerical effects of changing these prior, but someone could give me a sort of interpretation of putting small constants like this as priors?
23605
Bayes theorem goes $$ P(\textrm{model}|\textrm{data}) = \frac{P(\textrm{model}) \times  P(\textrm{data}|\textrm{model})}{P(\textrm{data})} $$  This is all fine. But, I've read somewhere:     Basically, P(data) is nothing but a normalising constant, i.e., a constant that makes the posterior density integrate to one.   We know that $0 \leq P(\textrm{model}) \leq 1$ and $ 0 \leq P(\textrm{data}|\textrm{model}) \leq 1$.    Therefore, $P(\textrm{model}) \times P(\textrm{data}|\textrm{model})$ must be between 0 and 1 as well.  In such a case, why do we need a normalizing constant to make the posterior integrate to one?
10819
Simpson's effect is key to understanding potential pitfalls in medical research, particularly in the area of meta-analysis, where multiple studies with dissimilar methodologies are brought together to provide a more definite answer to a problem. The cases reported tend to deal with count tables and ratios that change because of a "lurking" variable or confounder that is ignored (usually different group sizes). However, it is interesting to look at the continuous case to understand it conceptually.  Unsurprisingly, there are a number of posts touching on this issue, but hopefully this Q&amp;A-post will provide with some discussion or entry level intuition, unrelated to any more specific question. To look into the paradox different studies composing a meta-analysis are regarded as random effects in a mixed model.
112905
As far as I know, the variance inflation factor is not computed with pseudo-$R^{2}$ or generalized $R^{2}$ in binary outcome models (e.g. logistic regression).  Are there other measures of multi-colinearity than VIF appropriate to such models?  Why ought or ought not we think about multi-colinearity in such models?
21403
In an astronomical context, the authors of a paper desire to use a Gibbs algorithm. Please note: I am inexperience in MCMC algorithms, and specifically in Gibbs sampling.  What we want, in essence, is the full posterior distribution, given some data: $P(X,Y|data)$. To achieve this, we sample from the marginal distributions $P(X|Y,data)$ and $P(Y|X,data)$. In the stationary situation, sampling from these would be statistically identical to sampling from the full joint probability distribution, as I understand it.  Now, the authors note that in this particular case,   " (...) unfortunately the two marginal distributions $P(X|Y,data)$ and $P(Y|X,data)$ are in general both much narrower than the marginalized distribution $P(X|data)$: given a particular a particular $Y$ [in this case, lensing potential], $X$ [in this case, the delensed CMB] is given essentially by a delta function. This means that naive Gibbs iterations will not converge within a reasonable time."   Now these statements are completely lost on me. So my question is twofold:   The statement about the delta function is meant as explanatory supplement I think, but it doesn't clarify anything for me. Why is this 'essentially a delta function'?  Second, and more importantly: OK, suppose the conditionals are 'much narrower'. So what? Why is $P(X|data)$ even relevant, aren't we interested in $P(X|Y,data)$, $P(Y|X,data)$ and ultimately $P(X,Y|data)$ only in Gibbs sampling? Why would narrow conditions mean slow convergence?   The paper in question is a review paper by Challinor and Lewis, 2006. The arxiv print can be found here:    And the text I'm referring to is at the end of section 8, delensing the sky.
118258
I am analyzing panel data.  First, I have to decide whether to use a random or fixed effect estimator. The Hausman test suggests to use the fixed effect estimator (also named within group estimator). Thus, this is what I am using. However, this method eliminates the individual fixed effects, that is, the Ui's, which is what I am more interested about. Hence, I proceed with a second step.  Second, I run a between-group estimator where I regress the predicted individual effects, that is the Ui's predicted from the first step, against a list of time invariant fixed effects I am interested in  My questions are three:   I was taught this two-step method in a graduate summer course, and no book I have read mentions it, do you know what is its name? Is it meaningful to cluster the standard errors on individuals in the first step? (i.e. within-group estimator)  Is it meaningful to use a variable both as a covariate and as cluster for the standard error in the same model?
45111
Question(s): What is the idea and intuition behind quasi maximum likelihood estimation (QMLE; also known as pseudo maximum likelihood estimation, PMLE)? What makes the estimator work when the actual error distribution does not match the assumed error distribution?  The  for QMLE is fine (brief, intuitive, to the point), but I could use some more intuition and detail, perhaps also an illustration. Other references are most welcome. (I remember going over quite a few econometrics textbooks looking for material on QMLE, and to my surprise, QMLE was only covered in one or two of them, e.g. Wooldridge  (2010), Chapter 13 Section 11, pp. 502-517.)
46312
I've come across a very good text on Bayes/MCMC. IT suggests that a standardisation of your independent variables will make an MCMC (Metropolis) algorithm more efficient, but also that it may reduce (multi)collinearity. Can that be true? Is this something I should be doing as standard.(Sorry).  Kruschke 2011, Doing Bayesian Data Analysis. (AP)  edit: for example       &gt; data(longley)      &gt; cor.test(longley$Unemployed, longley$Armed.Forces)  Pearson's product-moment correlation       data:  longley$Unemployed and longley$Armed.Forces       t = -0.6745, df = 14, p-value = 0.5109      alternative hypothesis: true correlation is not equal to 0       95 percent confidence interval:      -0.6187113  0.3489766       sample estimates:       cor       -0.1774206        &gt; standardise &lt;- function(x) {(x-mean(x))/sd(x)}      &gt; cor.test(standardise(longley$Unemployed), standardise(longley$Armed.Forces))  Pearson's product-moment correlation       data:  standardise(longley$Unemployed) and standardise(longley$Armed.Forces)       t = -0.6745, df = 14, p-value = 0.5109       alternative hypothesis: true correlation is not equal to 0       95 percent confidence interval:       -0.6187113  0.3489766        sample estimates:        cor       -0.1774206    This hasn't reduced the correlation or therefore the albeit limited linear dependence of vectors.  What's going on?  R