I am conducting separate regression models for two of my hypotheses. If I spot outliers in one case of conducting the regression model can I then bring them back for my second? Or do these outliers have to be removed from the dataset completely?
42876
I am performing multiple regression analyses and I am not sure whether outliers in my data should be deleted. The data I am concerned about appear as "circles" on the SPSS boxplots, however there are no asterisks (which makes me think they are not 'that bad'). The cases I am concerned about do appear under the "casewise diagnostics" table in the output - therefore should I delete these cases?
99785
I'm fairly new to Machine Learning/Modelling and I'd like some background to this problem. I have a dataset where the number of observations is $n&lt;200$ however the number of variables is $p\sim 8000$. Firstly does it even make sense to consider building a model on a dataset like this or should one consider a variable selection technique to start with such as ridge regression or Lasso? I've read that this situation can lead to over-fitting. Is that the case for all ML techniques or do some techniques handle this better than others? Without too much maths a simple explanation on why the maths start to breakdown for $p&gt;n$ would be appreciated.
11199
I am running linear regression models and wondering what the conditions are for removing the intercept term.   In comparing results from two different regressions where one has the intercept and the other does not, I notice that the $R^2$ of the function without the intercept is much higher. Are there certain conditions or assumptions I should be following to make sure the removal of the intercept term is valid?
75810
I did a principal component analysis, which resulted in 7 components that I am now using for a principal component regression on my independent variable.  However, I want to add 2 control variables, but these are not component scores, but "normal" variables (on a Likert scale, the same sort of variables my independent variables were).  Is that okay to do or do I have to make these into components as well? I have done the regression both with those two as variables and combined in a component and the results are practically the same.
123658
Can anyone please explain how splitting is performed in regression trees when we only have continuous features. I have referred to different papers, but all I could find is formulas or theorems.   Can someone please explain, with an example, how we can build a regression tree from scratch?  That would be a great help.
19394
I am building a classification model (let's say logistic regression) using 2 different data sets. The data sets do not come from the same population. Let's say the data sets comes from different space-time coordinates (Asia in Fall vs North America in Winter). I have been told that simply mixing the data sets into one and applying a classification model is not right (closest thing I understood was concerns around bias). Are there any techniques or models that I may use to analyze these 2 data sets as one? Note that the variables/features all align (including the target variable, which is binary).
66997
Suppose I have a multiple linear regression of the form: $$y_i = \beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2} +\epsilon$$ Then suppose I am interested in testing this set of hypotheses: $$H_0: \beta_1 = 0 \text{ OR } \beta_2 = 0$$ $$H_A: \beta_1 \neq 0 \text{ AND } \beta_2 \neq 0$$ I am interested in deriving a test statistic for this hypothesis.  I understand that I need to derive this under the null hypothesis.  However, it is not as straightforward as just plugging in $\beta_1 = 0 \text{ and } \beta_2 = 0$, since our null is that either/or is zero, not both. Any input would be much appreciated, thank you!
77958
Hi I'm studying regression techniques.  My data has 15 features and 60 million examples (regression task).  When I tried many known regression techniques (gradient boosted tree, Decision tree regression, AdaBoostRegressor etc) linear regression performed great.   Scored almost best among those algorithms.  What can be the reason for this? Because my data has so many examples so DT based method can fit well.   regularized linear regression ridge, lasso performed worse   Can anyone tell me about other performing-well regression algorithms?   Is Factorization Machine and Support vector regression good regression technique to try?
75187
I was wondering if we need to check the multicollinearity between variables when we run the stepwise regression, If yes, why? Also, If we have several IV, how we should know which interaction we should include in the model?
69363
Provided a sample size "N" that I plan on using to forecast data. What are some of the ways to subdivide the data so that I use some of it to establish a model, and the remainder data to validate the model?  I know there is no black and white answer to this, but it would be interesting to know  some "rules of thumb" or usually used ratios. I know back at university, one of our professors used to say model on 60% and validate on 40%.
93970
Collinearity can pose certain problems in various kinds of regression problem. In particular, it can make the parameter estimates have high variance and be unstable.   Various methods have been proposed to deal with this including ridge regression, partial least squares regression, principal components regression, dropping variables and getting more data.   One method that is controversial is standardizing or scaling the independent variables, with various experts saying it is a good (e.g. Garcia) or bad (e.g. Belsley) idea. Belsley's issue seems to be (in non-technical terms) that changing the IVs just pushes the problem under the rug.  But other experts seem not to agree. And the authors tend to get rather heated in defending their own positions.   Back when I did my dissertation (on collinearity diagnostics) I found Belsley's arguments convincing, but that was long ago (I got my degree in 1999).  I'm looking either for expert guidance or for any current review article that is unbiased.
101959
I've read in a few sources, including , that Random Forests are not sensitive to outliers (in the way that Logistic Regression and other ML methods are, for example).  However, two pieces of intuition tell me otherwise:   Whenever a decision tree is constructed, all of the points must be classified.  This means that even outliers will get classified, and hence will affect the decision trees where they were selected during boosting. Bootstrapping is a part of how a RandomForest does sub-sampling.  Bootstrapping is susceptible to outliers.   Is there any way to reconcile my intuition about its sensitivity to outliers, with sources that disagree?
81953
I found many articles that state that boosting methods are sensitive to outliers, but no article explaining why.   In my experience outliers are bad for any machine learning algorithm, but why are boosting methods singled out as particularly sensitive?  How would the following algorithms to rank in terms of sensitivity to outliers: boost-tree, random forest, neural network, SVM, and simple regression methods such as logistic regression?
84149
I work in the field of data mining and have had very little formal schooling in statistics. Lately I have been reading a lot of work  that focuses on Bayesian paradigms for learning and mining, which I find very interesting.   My question is (in several parts), given a problem is there a general framework by which it is possible to construct a statistical model? What are the first things you do when given a dataset of which you'd like to model the underlying process? Are there good books/tutorials out there that explain this process or is it a matter of experience? Is inference in the forefront of your mind when constructing your model or do you first aim to describe the data before you worry about how to use it to compute?  Any insight would be greatly appreciated! Thanks.
24698
What diagnostic plots (and perhaps formal tests) do you find most informative for regressions where the outcome is a count variable?  I'm especially interested in Poisson and negative binomial models, as well as zero-inflated and hurdle counterparts of each. Most of the sources I've found simply plot the residuals vs. fitted values without discussion of what these plots "should" look like.  Wisdom and references greatly appreciated. The back story on why I'm asking this, if it's relevant, is .  Related discussions:
33305
I have one-sided hypotheses that I want to test with multiple regressions. Does anyone know how to perform a one-sided hypothesis testing? I heard of different positions regarding this matter ("not possible to test one-tailed with regression" and "for one-sided hypotheses just cut the p-value in half"). Or maybe I need another command in with my statistics program (I use R)!?  I didn't think this to be a difficult question but couldn't find a consistent and satsfying solution.  Thank you in advance for any helping comments.
54583
Virtually any database we want to make predictions using machine learning algorithms will find missing values ​​for some of the characteristics.  There are several approaches to address this problem, to exclude lines that have missing values ​​until they fill with the mean values ​​of the characteristics.  I would like to use for a somewhat more robust approach, which would basically run a regression (or another method) where the dependent variable (Y) would be each of the columns that have missing values ​​but only with the rows of the table that contain all the data , and predict the missing values ​​with this method, complete the table by the table and move to the next 'column' with missing values ​​and repeat the method until everything is filled.  But that gives me some doubts.  Why any column start? I believe that the one with the smallest missing values ​​until the one with the most  Is there any threshold of missing values ​​that is not worth trying to complete it? (for example, if this characteristic only has 10% of the values ​​filled would not it be more interesting to exclude it)  Is there any kind of implementation in traditional packages or other methods that are robust to missings?