So, I have been studying causality and came across with Googles Causality Impact Analysis.  I already read the paper and watched Kay's keynote. However, I wonder if it is also possible to use a simple regression or a machine learning model build the synthetic post intervention series.
123985
I am often told that the crucial difficulty in causal inference is that we only observe one value between $Y(1)$ and $Y(0)$ while we want to estimate $E[Y(1) - Y(0)]$. There is always an unobserved value.  Here is my problem: why don't we simply use the samples with treatment $z_i = 1$ to regress $y(1) \sim x$ , and similarly use the samples with treatment $z_i = 0$ to regress $y(0) \sim x$, and combine them to estimate $E[Y(1) - Y(0)]$?  From this perspective, causal inference is just two regression problems and needn't be treated as a special area. I am sure that there must be something wrong, but what is it?
17671
I wish to forecast Y for year 2018 but I only have two data points of Y in years 2006 and 2012.   I already did multiple linear regression (since I have a lot of predictors) but multiple linear regression does not consider the time aspect so my predictions are limited.  I was advised to do causal forecasting instead, which will apparently give me a forecast for the next year. I researched about it but only found regression once again and a few others (econometric, input-output etc)  Is there an actual model in causal forecasting that can solve my problem?
99785
I'm fairly new to Machine Learning/Modelling and I'd like some background to this problem. I have a dataset where the number of observations is $n&lt;200$ however the number of variables is $p\sim 8000$. Firstly does it even make sense to consider building a model on a dataset like this or should one consider a variable selection technique to start with such as ridge regression or Lasso? I've read that this situation can lead to over-fitting. Is that the case for all ML techniques or do some techniques handle this better than others? Without too much maths a simple explanation on why the maths start to breakdown for $p&gt;n$ would be appreciated.
84149
I work in the field of data mining and have had very little formal schooling in statistics. Lately I have been reading a lot of work  that focuses on Bayesian paradigms for learning and mining, which I find very interesting.   My question is (in several parts), given a problem is there a general framework by which it is possible to construct a statistical model? What are the first things you do when given a dataset of which you'd like to model the underlying process? Are there good books/tutorials out there that explain this process or is it a matter of experience? Is inference in the forefront of your mind when constructing your model or do you first aim to describe the data before you worry about how to use it to compute?  Any insight would be greatly appreciated! Thanks.
13478
The first sentence of this  page claims that  "In econometrics, an endogeneity problem occurs when an explanatory variable is correlated with the error term. "  My question is that how can this ever happen? Isn't regression beta chosen such that the error term is orthogonal to the column space of the design matrix?
91250
I am six years into a business role and have a bachelor's in physics and applied math/stats. Sean Carroll's (Caltech physicist)  opened me to the idea that Bayesian statistics is one useful way of thinking about anything - inevitably you hold a prior and you should update your credence as additional information becomes available.   Is there a path to training your intuition to thinking this way? Critically, it would require repeated practice with verifiable answers through either a course, or self study that includes many problems and solutions. I do not believe simply reading will do.  Possible resources, having read every related question on this site I could find:   "Probability Theory" by Jaynes. Pro: analytic; intuitive explanation of bayesian statistics. Con: prerequisites; missing problems/solutions. "Doing Bayesian Data Analysis" by Kruschke. Pro: includes problems &amp; solutions; requires only "algebra and rusty calculus". Con: works in R, which I think provides for less intuitive learning than the analytical (I may be wrong).   If it is a multi-year path I need to take, starting elsewhere, I am happy to do so! Ideally, I would avoid the frequentist methods, as I have no use for them.  My goal is not to be a scientist, but to leverage insight into how reality works to go above and beyond the established thinking in business.  Many thanks for any suggestions!
116759
I have a GEE logistic regression model that I've built that predicts that probability of mortality given a rather extensive set of covariates.  I wanted to write here as a sanity check of sorts to make sure that what I'm doing seems sensible and to get some clarification on how to go about obtaining confidence intervals for this analysis.  Allow me to explain:  The purpose of the study is to examine the relationship between a continuous intervention variable (which is a percentage in the range of 0 to 100%) on mortality while controlling for a number of other covariates.  I build a model in GEE due to the data having some dependencies and repeated patient visits.  After building the model, I made predictions on the datasets, holding all the variables in my model constant with the exception of the intervention, which I initially set fixed at 0% (and later to 10%, 20%, . .  100%) for every patient in the dataset.  This gave me predictions of mortality for each individual.  Now, within each of four risk categories (a categorical variable that appears in the model), I obtained the mean predicted mortality rate, the mean predicted upper 95% confidence interval and the mean predicted lower 95% confidence interval.  So in the end, I have four risk categories, each with a mean predicted mortality and mean 95% upper and lower confidence bounds. This processed was repeated once for each 10% increment of my intervention variable.  My intention was to essentially look at the all the patient characteristics and form a counterfactual where the only thing I change is the intervention, and made predictions to see, hypothetically, what could we expect if, all else being constant, the intervention had actually been 10% (or 20%, 30%, etc.) instead of what was actually observed.  So my questions are: (1)  Does this sound like a sound/reasonable approach and if so (2) Is it appropriate to simply take the mean of the individual 95% upper and lower confidence intervals to find confidence intervals for each level of categorical risk factor and intervention percentage?  If not, how else can I go about this, without having to separately tabulate the confidence intervals for very specific linear combinations of an extensive list of predictors?  Essentially, I'm trying to avoid having to say something like, for those with high oxygen saturation levels, having never experienced cancer before, who were previously admitted for the same condition, and high cancer risk, the confidence interval is (x, y).  In other words, I'd like to be able to just say something about the the intervention in each of the groups, across all patients reflective of their other current clinical characteristics en masse.  UPDATED: I have read a bit about marginal means and this seems close to what I'm trying to do, but not quite.  Instead of using marginal means, I'm trying to obtain the mean predicted mortality across the entire population under study.  It seems simultaneous confidence intervals are quite easy to provide for marginal means, but I'm not sure how to go about obtaining confidence intervals for the methodology I've described, other than bootstrap methods.  Any additional advice would be greatly appreciated.
24468
I am trying to understand the benefit of propensity matching over non-parametric regression for causal inference from non-experimental data. As background: the way I understand it, parametric regressions are generally a poor choice for causal inference when selection bias exists. One can try to create a model that takes into account the different baselines between treatment and control groups, but she/he will be be extremely vulnerable to model misspecification.* Matching solves this issue by making the two datasets look &quot;alike.&quot; The exact meaning of alike depends on the matching algorithm used, but all the algorithms strive to eradicate model dependence, with the most common types being stratified and weighted propensity matching. The mathematics for these algorithms are well-developed, and in particular, we know how to extract error bars and statistical significance. What I'm struggling with is why this is superior to just using a non-parametric regression like a decision tree or random forest, which are also designed to prevent model misspecification. After creating the forest, one could run individuals through it assuming treatment or no treatment, and call the difference the estimated treatment effect for that individual. My first guess is that extracting significance, which is critical to causal inference, from trees is difficult, but it seems that statisticians have made strides in that regard over the last decade or so. To be clear, I am not asking about using a tree to develop the propensity scores, but to use one instead of propensity matching. To help kick off the conversation, I've developed five hypotheses for why matching is preferred to non-parametric regression, but haven't been able to find anything proving or disproving any:  Empirical research demonstrates that stratified or weighted propensity matching (the most common types) yield results closer to causal experiments than non-parametric regressions like trees.  Though it is possible to extract significance from non-parametric regressions like random forests, the math isn't settled, or the notion of &quot;significance&quot; for a decision tree variable doesn't map precisely to the notion of &quot;one minus the odds of a type I error.&quot;  Though it is possible to extract significance from non-parametric regressions, the code is difficult to write.  Model misspecification actually is an issue for decision trees due to the tuning required to run them. Empirical observation has demonstrated this is more of an issue for decision trees than it is for matching algorithms.  We don't actually know a lot about whether we can use non-parametric regressions for causal inference, but we do know that matching works, so it there's no reason to reinvent the wheel.   *As detailed in the first ten minutes of this wonderfully intuitive Youtube:
31290
In nearly any field there is a number of important results which look obvious to experts post factum but somehow are not that easy to come by in the first place (e.g. in mathematics some important definitions look exactly like this). Unfortunately, this apparent post factum simplicity makes conveying the importance of the idea to expert audience (and in particular to the journals' editors and referees) very difficult. To make things worse, sometimes the author is unable to illustrate the application of the idea by sufficiently striking examples.      The question is whether it is possible (and if yes, how) to mitigate this apparent post factum simplicity in the talks and research articles, i.e., what can be done to adequately convey the significance of "post-factum-obvious" results to the audience and, in particular, to get these results to the journals they truly deserve?   I am particularly interested in the advice applicable to mathematics/mathematical physics but the suggestions suitable for other fields are very welcome too, as the situation in question does not seem to be all that field-specific.
16999
Hi I am trying to find the non-parametric equivalent of a two-way ANOVA (3x4 design) which is capable of including interactions. From my reading in Zar 1984 "Biostatistical analysis" this is possible using a method put forth in Scheirer, Ray, and Hare (1976), however, according to other posts online it was inferred that this method is no longer appropriate (if it ever was).  Does anyone know what method would be appropriate for doing so, and if so the corresponding functions in R or Stata?
89556
Can someone explain to me why would anyone choose a parametric over a nonparametric statistical method for hypothesis testing or regression analysis?   In my mind, it's like going for rafting and choosing a non-water resistant watch, because you may not get it wet. Why not use the tool that works on every occasion?
105084
As we make inroads into Machine Learning, there seem to be plenty of respectable courses available online via Coursera, edX, etc. on the topic. As quantum computing is still in its infancy, not to mention, incredibly daunting, it is vital that easy to understand, introductory courses are made available.  I was able to find these courses:      However, I am not certain how relevant the second course might be.  Obviously, this question may benefit from receiving answers in the near future as the topic steadily becomes mainstream.
43225
I assume it is OK to mix different data transformations in the same analysis. I've had to transform some variables to squared and some to cubed in order to meet normal distribution requirements. I'm assuming that it's OK to then use the transformed variables together for regression analysis?
94322
During my literature survey for a project, I came across papers that have very interesting results and have decent number of citations. Unfortunately, the sub-area I am exploring doesn’t have any standard datasets on which I can test my implementation of the algorithm presented in their paper. The authors have generated the dataset by applying certain actions on a subset of publicly available information, which are Flickr images in my case.   So I have contacted the authors, asking whether they can provide me with their code or the dataset that they used for their implementation so that I may use it to perform my experiments and possibly try to extend their approach. But almost a week has passed by, and I haven’t received any response from them. Does that mean that,   My method of asking was not correct? (In that case, what would be appropriate?) They don’t have the code or dataset available? (The papers were published in 2004.) They are not interested?   Would it be polite to remind them again?
584
Is it ever valid to include a two-way interaction in a model without including the main effects?  What if your hypothesis is only about the interaction, do you still need to include the main effects?
1017
For my own understanding, I am interested in manually replicating the calculation of the standard errors of estimated coefficients as, for example, come with the output of the lm() function in R, but haven't been able to pin it down. What is the formula / implementation used?
112001
I have a dataset with 10 variables ...is it feasible to transform each variable differently while doing multiple regression...  for example new_V1 = log(v1)  New_V2= V2^2  New_V3= 1/V3  Likewise differently for different variables and then applying multiple regression?