I'm just looking for ideas for this. Say we want to predict the presence of a certain &quot;thing&quot; in a country by each postcode - let's take the UK, so there could be around 1.7 million postcodes - and you want to build a model where for each postcode, you're outputting either a 1 or a 0. A constraint could be the &quot;thing&quot; has uptake in 400k postcodes. You can imagine typical demographic data for predictor variables: population, income data, age range... However I feel independence between the outputs is too strong of an assumption - or at least, I'd like to this before applying models that assume it. I'm not really sure where to start so perhaps someone could give me some guidance - It wouldn't quite be logistic regression since the output variable would have some spatial correlation. There's also possibly enough training data with this large a set of postcodes - the model could be built using post sectors first, perhaps, but we'd still like a level of spatial correlation included... Any ideas or food for thought welcome, I'm kind of stumbling around in the dark so far. Am I right for instance in saying that multivariate regression would not work here, because the binary output variables are not independent?
122822
I am using a binomial regression model for presence/absence, with 20 independent variables to test. The data has x and y coordinates and I would like to understand how can I take into account the spatial autocorrelation.  I already studied the correlation between the variables and run the same model for 1000 different samples (I have a big dataset that allows me to do this) to understand the distribution of each parameter and check for variables that might be introducing problems in my model.   glm_model &lt;- glm(PA ~ Var1 + Var2 + Var3 + Var4 + Var5,family=binomial(link=logit))  However I believe I also need to account for spatial autocorrelation. I saw that there is a package that might help me (spdep), however I am not sure I completely understand if I can use my model or not. My question is what are my options ?
25624
I have several hundred measurements. Now, I am considering utilizing some kind of software to correlate every measure with every measure. This means that there are thousands of correlations. Among these there should (statistically) be a high correlation, even if the data is completely random (each measure has only about 100 datapoints).  When I find a correlation, how do I include the information about how hard I looked for a correlation, into it?   I am not at a high level in statistics, so please bear with me.
22442
I have a dataset that consists of only categorical variables and a target variable. I want to predict the (binary) target variable with the categorical variables.  I am trying to do this in Python and sklearn.  The categorical variables have many different values. I was thinking to create dummy variables for each value in all the categorical variables. However, in the end this results in a sparse data set with thousands of variables.  How would you go about to reduce the number of dummy variables? Would you use Chi2 to select useful features?  Or would you not make dummy variables at all?
118642
I'm looking to generate a set of 5 random variables and enforce a dependence structure between them and onto a dependent variable $Y$.  I understand how to generate correlated random variables for multivariate normal, but not when mixing different types.  Below is a little more than I need, but I'm hoping someone can give me a general way of solving this problem...   $X_1$ and $X_2$ need to be highly correlated Bernoulli variables.  $X_3$ needs to take one of 5 categorical values, call them "A"..."E".  $X_4$ needs to be normal, and negatively correlated with $X_1$, $X_2$.  $X_5$ needs to approximate test scores from $0$ to $100$ with a high skew, so gamma probably.  $X_5$ needs to be positively correlated with $X_1$, $X_2$, $X_4$.   Each of these variables must impact a "success/occurrence" Bernoulli distributed variable $Y$.  How would I begin?  I would like to enforce correlation both between the values of $X$, and also between each $X$ and $Y$.  (The categorical correlations seem particularly confusing to me.)
46161
Suppose we have data set $(X_i,Y_i)$ with $n$ points. We want to perform a linear regression, but first we sort the $X_i$ values and the $Y_i$ values independently of each other, forming data set $(X_i,Y_j)$. Is there any meaningful interpretation of the regression on the new data set? Does this have a name?  I imagine this is a silly question so I apologize, I'm not formally trained in statistics. In my mind this completely destroys our data and the regression is meaningless. But my manager says he gets "better regressions most of the time" when he does this (here "better" means more predictive). I have a feeling he is deceiving himself.  EDIT: Thank you for all of your nice and patient examples. I showed him the examples by @RUser4512 and @gung and he remains staunch. He's becoming irritated and I'm becoming exhausted. I feel crestfallen. I will probably begin looking for other jobs soon.
23464
First, don't panic. Yes, there are many similar question on this site. But I believe none gives a conclusive answer to the question below. Please bear with me.  Consider a data generation process $\text{D}_X(x_1, ... , x_n|\theta)$, where $\text{D}_X(\cdot)$ is a joint density function, with $n$ variables and parameter set $\theta$. It is well known that a regression of the form $x_n = f(x_1, ... , x_{n-1}|\theta)$ is estimating a conditional mean of the joint distribution, namely, $\text{E}(x_n|x_1,...,x_{n-1})$. In the specific case of a linear regression, we have something like $$ x_n = \theta_0 + \theta_1 x_1 + ... + \theta_{n-1}x_{n-1} + \epsilon $$ The question is: under which assumptions of the DGP $\text{D}_X(\cdot)$ can we infer the regression (linear or not) represents a causal relationship? It is well known that experimental data does allow for such interpretation. For what I can read elsewhere, it seems the condition required on the DGP is exogeneity: $$ \text{E}(x_1, ... x_{n-1}|\epsilon) = 0$$ The nature of randomisation involved in experimental data ensures the above is the case. The story goes then to argue why observational data normally fails in achieving such condition, reasons including omitted variable bias, reverse causality, self-selection, measurement errors, and so on. I am however uncertain about this condition. It seems too weak to encompass all potential arguments against regression implying causality. Hence my question above. UPDATE: I am not assuming any causal structure within my DGP. I am assuming that the DGP is complete, in the sense that there must be some causality (an ontological position it could well be debated), and all relevant variables are included. The key is to identify the set of assumptions which ensure me causality goes from certain variables to the other, without assuming from the outset such direction of causality.  Many similar posts on the site spend time mentioning why correlation does not imply causation, without providing hard arguments about when it does. That is the case, for instance, of  very popular post. Additionally, in the  on the site about the topic, the accepted answer gives the very vague answer:  Expose all hidden variables and you have causation.  I do not know how to translate that to my question above. Neither is the second most upvoted answer. And so on. That is why I believe this post does not have an answer elsewhere.
17051
I am trying to run a Multiple Regression model but I am not sure what should I do to prepare the data input. In my model the dependent variable is the artificial surfaces in hectares, the independent are the GDP (monetary value) and Population (inhabitants). The histogram shows that the data is not normal distributed.    I would like to know if I need to normalize the distribution of the data in order to get the "bell" curve and/or  if I need to normalize the data (with the max-min or standardize).
9017
I'm struggling with the interpretation of a regression model where a categorial variable (5 levels) is dummy coded. Here is the result of my calculation in R:  Call: lm(formula = DV ~ Age + Gender + factor(Categorial) +  Continuous 1 + Continuous 2 + Continuous 3,  data = dat)  Residuals:  Min       1Q   Median       3Q      Max  -1.30058 -0.25326  0.00349  0.28123  1.49877   Coefficients:                       Estimate Std. Error t value Pr(&gt;|t|)    (Intercept)           -0.42367    0.30694  -1.380  0.16842    Age                   -0.05949    0.02026  -2.936  0.00356 ** Gender                -0.01800    0.04828  -0.373  0.70952    factor(Categorial)2   -0.30625    0.12645  -2.422  0.01596 *  factor(Categorial)3   -0.03441    0.07752  -0.444  0.65736    factor(Categorial)4   -0.12603    0.09914  -1.271  0.20453    factor(Categorial)5   -0.08417    0.13269  -0.634  0.52630     Continuous 1           0.12080    0.04346   2.779  0.00575 ** Continuous 2          -0.06592    0.04383  -1.504  0.13354    Continuous 3          -0.06230    0.03475  -1.793  0.07392 .  --- Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1  Residual standard error: 0.4259 on 336 degrees of freedom   (6 observations deleted due to missingness) Multiple R-squared:  0.1315,    Adjusted R-squared:  0.1057  F-statistic: 5.089 on 10 and 336 DF,  p-value: 6.353e-07   Ok. Age, Factor 2 of the categorial variable and the first continuous variable are significant predictors of the dependent variable. so far so good.   What I'm not understanding is:   The reference category of the dummy coded categorial variable is the intercept and the first category of the categorial variable. right? How do I interpret this?  When doing an anova with the categorial variable as a independent variable, this factor is a significant predictor. With the results of the linear model, one could conclude that this is only due to category 2, right? Can I test contrasts with this linear regression model (e.g. Category1 vs. Category2)? Should I include interactions?   I'd be glad for any help :-)
69881
I'm trying to play around with classification models and started off with logistic regression in R. When I have all the numeric variables in the data set the model works correctly and I was able to interprer the results.  The question is, with another data set I have more than 5 variables which are categorical but they are important though.   1) How do I deal with the variables with categorical values? the total unique categorical values are more than 100 for almost 5 predictors.  2) My dependent variable is having 3 classes, is it okay to using logistic regression for this purpose? (I would still need to explore other classification techniques but right now I'm exploring logistic regression.
95815
a very naive question for some of you. I have a population of 80000 data points. Some points in the population are many SD away from the rest of the population and they could be easily identified and visualized assigning them a p-value. I know that excluding those values, the population follow a nice normal distribution. What I ask you today is whether you can suggest me an approach to assigning a pvalue to each data point in the population where the null hypothesis is that the data contribute to the normality.
113630
We have a series of observations of different properties (such as heart rate or blood sugar level and others as well) across different days from different people from different geographical regions. The number of days on which the observations are made is the same for all regions; but there could be different number of participants in each region. The result of the data collection will be as follows: DATA We have a matrix, $M_{i,j}$, for each region $i$ and property $j$, with $T$ observations across time (rows) and $N_{i,j}$ (columns) for number of participants from region $i$ and property $j$. Here, we consider all properties together as representing a region. OBJECTIVE The goal is to identify people from which regions have a more similar constitution. QUESTION How can we compute the Bhattacharya Distance across the various matrices $M_{i,j}$ for all the properties combined for a region (corresponding to Case 2 of the data collection)? Dimension of Matrix: $Dim(M_{i,j})=(T,N_{i,j})$ There are two cases here. 1) $N_{i,j}=N_{i,k}$ $,\forall j,k$ That is the number of participants in a region are the same for all properties being measured. 2) $N_{i,j} \neq N_{i,k}$ $,\forall j,k$ That is the number of participants in a region could be different for some of the properties being measured. But if helpful we can assume the simple case 1) $N_{i,j}=N_{i,k}$. Please let me know if anything is not clear and if you need any further information. Related Case When we have a matrix, $M_i$, for each region $i$, with $T$ observations across time (rows) and $N_i$ (columns) for number of participants from that region. Here, we consider each property separately. How can we compute the Bhattacharya Distance across the various matrices $M_i$ (in pairs or combined, whichever is simpler) separately for each property (corresponding to Case 1 of the data collection)? Answer:
4148
I have a continuous outcome variable. I understand that if I have a binary predictor, and a continuous predictor, and an interaction, then the model looks like this:  $y_{i} = \beta_{0} + \beta_{1}x_{1} + \beta_{2}x_{2} + \beta_{3}x_{1}x_{2} + \varepsilon_{i}$  However, I'm thinking of making the binary predictor a categorical one instead, with three categories.    What will the model equation look like once I make that change? I understand that I'll need to have two dummy variables, but I can't conceptualize what the interaction will look like under that circumstance.  Does it make any difference whether I use 0,1 to code the binary predictor, or some other values like 1,2? This question also applies to the model with the categorical predictor, since in that case I'll have to decide how to code the dummy variables.  Prior to the interaction term being in my model I was encouraged to center the continuous predictor. Is centering (still) a good idea now that I have this interaction term?
6137
I am thinking of building a model predicting a ratio $a/b$, where $a \le b$ and $a &gt; 0$ and $b &gt; 0$. So, the ratio would be between $0$ and $1$.  I could use linear regression, although it doesn't naturally limit to 0..1.  I have no reason to believe the relationship is linear, but of course it is often used anyway, as a simple first model.  I could use a logistic regression, although it is normally used to predict the probability of a two-state outcome, not to predict a continuous value from the range 0..1.  Knowing nothing more, would you use linear regression, logistic regression, or hidden option c?
32341
Apologies in advance if any of the terminology I use is incorrect. I'd welcome any correction. If what I describe as a "cut-off" goes by a different name, let me know and I can update the question.  The situation I'm interested in is this: you have independent variables $\bf{x}$ and a single dependent variable $y$. I'll leave it vague, but assume that it would be relatively straightforward getting a good regression model for these variables.  However, the model you're aiming to create is for independent variables $\bf{x}$ and dependent variable $w = \min(y,a)$, where $a$ is some fixed value within the range of $y$. Equally, the data you have access to does not include $y$, only $w$.  A (somewhat unrealistic) example of this would be if you were trying to model how many years people will collect their pension for. In this case, $\bf{x}$ could be relevant information such as gender, weight, hours of exercise per week, etc. The 'underlying' variable $y$ would be life expectancy. However the variable you'd have access to and be trying to predict in your model would be $w = \min(0, y-r)$ where r is the retirement age (assuming for simplicity it's fixed).  Is there a good approach for dealing with this in regression modelling?
35328
I understand that when label encoding is used ,the numeric number can be interpreted to have an order and a model could assume a linear relationship. However shouldn't this be a problem when there are in-fact many levels in a categorical variable e.g. country.  How about the case of binary variables which for e.g. instead of gender.male ( 1,0) what if I just used Gender ( 1,0) where gender = 0 is female and 1 is male for instance. This shouldn't impact the model as much as a label encoding to a feature with multiple levels? How would this be for a case where I have a feature three levels ( -1,0,1) where -1 means not applicable, 0 means 'No' and 1 means 'Yes', so instead of having 2 columns feature.not_applicable(1,0) and feature.No(1,0) - mathematically how would models be impacted - models here would be GLMS,  boosting models, Random forests etc. Is label encoding recommended when feature has say &lt;=3 to 4 levels and above that is recommended to do one hot encoding or $n-1$ dummy variables?
13077
I'm analysing some data where I would like to perform ordinary linear regression, however this is not possible as I am dealing with an on-line setting with a continuous stream of input data (which will quickly get too large for memory) and need to update parameter estimates while this is being consumed. i.e. I cannot just load it all into memory and perform linear regression on the entire data set.  I'm assuming a simple linear multivariate regression model, i.e.  $$\mathbf y = \mathbf A\mathbf x + \mathbf b + \mathbf e$$  What's the best algorithm for creating a continuously updating estimate of the linear regression parameters $\mathbf A$ and $\mathbf b$?   Ideally:   I'd like an algorithm that is most $\mathcal O(N\cdot M)$ space and time complexity per update, where $N$ is the dimensionality of the independent variable ($\mathbf x$) and $M$ is the dimensionality of the dependent variable ($\mathbf y$). I'd like to be able to specify some parameter to determine how much the parameters are updated by each new sample, e.g. 0.000001 would mean that the next sample would provide one millionth of the parameter estimate. This would give some kind of exponential decay for the effect of samples in the distant past.
19575
X and Y are not correlated (-.01); however, when I place X in a multiple regression predicting Y, alongside three (A, B, C) other (related) variables, X and two other variables (A, B) are significant predictors of Y. Note that the two other (A, B) variables are significantly correlated with Y outside of the regression.   How should I interpret these findings? X predicts unique variance in Y, but since these are not correlated (Pearson), it is somehow difficult to interpret.  I know of opposite cases (i.e., two variables are correlated but regression is not significant) and those are relatively simpler to understand from a theoretical and statistical perspective. Note that some of the predictors are quite correlated (e.g., .70) but not to the extent that I would expect substantial multicollinearity. Maybe I am mistaken, though.    NOTE: I asked this question previously and it was closed. The rational was that this question is redundant with the question "". Perhaps I do not understand the other question, but I believe these are entirely separate questions, both mathematically and theoretically. My question is entirely independent from if "a regression is significant". Furthermore, several predictors are significant, while the other question entails variables not being significant, so I don't see the overlap. If these questions are redundant for reasons I do not understand, please insert a comment prior to closing this question. Also, I was hoping to message the moderator who closed the other question to avoid identical questions, but I couldn't find an option to do so.