I am modeling rates of people given drug $X$ or drug $Y$, working out probability of getting drug.  The drug $X$ and drug $Y$ points have been matched on exposure time (person years) to specific drug.   In this case, is there a need to include an offset term or should it now be treated as count data?
9472
Does anybody know why offset in a Poisson regression is used? What do you achieve by this?
4148
I have a continuous outcome variable. I understand that if I have a binary predictor, and a continuous predictor, and an interaction, then the model looks like this:  $y_{i} = \beta_{0} + \beta_{1}x_{1} + \beta_{2}x_{2} + \beta_{3}x_{1}x_{2} + \varepsilon_{i}$  However, I'm thinking of making the binary predictor a categorical one instead, with three categories.    What will the model equation look like once I make that change? I understand that I'll need to have two dummy variables, but I can't conceptualize what the interaction will look like under that circumstance.  Does it make any difference whether I use 0,1 to code the binary predictor, or some other values like 1,2? This question also applies to the model with the categorical predictor, since in that case I'll have to decide how to code the dummy variables.  Prior to the interaction term being in my model I was encouraged to center the continuous predictor. Is centering (still) a good idea now that I have this interaction term?
99785
I'm fairly new to Machine Learning/Modelling and I'd like some background to this problem. I have a dataset where the number of observations is $n&lt;200$ however the number of variables is $p\sim 8000$. Firstly does it even make sense to consider building a model on a dataset like this or should one consider a variable selection technique to start with such as ridge regression or Lasso? I've read that this situation can lead to over-fitting. Is that the case for all ML techniques or do some techniques handle this better than others? Without too much maths a simple explanation on why the maths start to breakdown for $p&gt;n$ would be appreciated.
79024
I have a large dataset of tennis-points (who served and if they won etc).  Now I would like to check if the points have any correlation with each other (does winning/losing the previous point affect the current or the one after this?).  I also have an "estimate" of each players serve-winning probability as well as his "actual" serve-winning probability in the game (calculated from the dataset).  Ive tried looking into autocorrelation but I honestly dont know where to start. Does anyone have any idea how to test if Point X have any effect on the outcome of Point X+1?  Any help is greatly appreciated  PS. I tried asking a similar question describing how I thought and the only answer I get was "You are lacking fundamental understanding about probability-theory", hence this time I havent included any of my own thought-process, but feel free to ask for more info if I have excluded something important DS.
2191
I have read through existing answers on CrossValidated (plus elsewhere online) and can't find what I'm looking for, but do please point me to existing sources if I've missed them.  Let's say I have a data set of N=1000 records, each of which can be manually sampled and labelled as either 'Valid' or 'Invalid' (or True/False, Right/Wrong, etc).    I want to achieve a given level of confidence that all records in the data set are Valid.  As I sample records, if I find a single Invalid one I would go back and amend how the data set is created to rectify that and similar problems.    So, after some iterations of spotting Invalids, fixing and recreating the data set, I do some sampling that only includes Valid records.  If I want to be (say) 99% or 95% sure that all records are Valid, how big does my sample have to be? (Ideally as a function of N.)  I've tried playing around with Hypergeometric tests () - in that context I want to know what k should be, but I don't have a fixed value of K. Rather I want to choose k such that K is likely to be equal to N - but setting K=N obviously works out to a Probability of 1! I'm also wondering if I need to use a Bayesian approach but I don't understand Bayesian stats enough.
62050
I have data from a survey experiment in which respondents were randomly assigned to one of four groups:  &gt; summary(df$Group)        Control     Treatment1     Treatment2     Treatment3              59             63             62             66    While the three treatment groups do vary slightly in the stimulus applied, the main distinction that I care about is between the control and treatment groups. So I defined a dummy variable Control:  &gt; summary(df$Control)      TRUE FALSE         59   191    In the survey, respondents were asked (among other things) to choose which of two things they preferred:   &gt; summary(df$Prefer)       A   B  NA's      152  93   5    Then, after receiving some stimulus as determined by their treatment group (and none if they were in the control group), respondents were asked to choose between the same two things:  &gt; summary(df$Choice)   A    B  149  101    I want to know if the being in one of the three treatment groups had an effect on the choice that respondents made in this last question. My hypothesis is that respondents who received a treatment are more likely to choose A than B.       Given that I am working with categorical data, I have decided to use a logit regression (feel free to chime in if you think that's incorrect). Since respondents were randomly assigned, I am under the impression that I shouldn't necessarily need to control for other variables (e.g. demographics), so I have left those out for this question. My first model was simply the following:  &gt; x0 &lt;- glm(Product ~ Control + Prefer, data=df, family=binomial(link="logit")) &gt; summary(x0)  Call: glm(formula = Choice ~ Control + Prefer, family = binomial(link = "logit"),      data = df)  Deviance Residuals:      Min       1Q   Median       3Q      Max   -1.8366  -0.5850  -0.5850   0.7663   1.9235    Coefficients:                     Estimate Std. Error z value Pr(&gt;|z|)     (Intercept)           1.4819     0.3829   3.871 0.000109 *** ControlFALSE         -0.4068     0.3760  -1.082 0.279224     PreferA              -2.7538     0.3269  -8.424  &lt; 2e-16 *** --- Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1  (Dispersion parameter for binomial family taken to be 1)      Null deviance: 328.95  on 244  degrees of freedom Residual deviance: 239.69  on 242  degrees of freedom   (5 observations deleted due to missingness) AIC: 245.69  Number of Fisher Scoring iterations: 4   I am under the impression that the intercept being statistically significant is not something that holds interpretable meaning. I thought perhaps that I should include an interaction term as follows:  &gt; x1 &lt;- glm(Choice ~ Control + Prefer + Control:Prefer, data=df, family=binomial(link="logit")) &gt; summary(x1)  Call: glm(formula = Product ~ Control + Prefer + Control:Prefer, family = binomial(link = "logit"),      data = df)  Deviance Residuals:      Min       1Q   Median       3Q      Max   -2.5211  -0.6424  -0.5003   0.8519   2.0688    Coefficients:                                  Estimate Std. Error z value Pr(&gt;|z|)     (Intercept)                         3.135      1.021   3.070  0.00214 **  ControlFALSE                       -2.309      1.054  -2.190  0.02853 *   PreferA                            -5.150      1.152  -4.472 7.75e-06 *** ControlFALSE:PreferA                2.850      1.204   2.367  0.01795 *   --- Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1  (Dispersion parameter for binomial family taken to be 1)      Null deviance: 328.95  on 244  degrees of freedom Residual deviance: 231.27  on 241  degrees of freedom   (5 observations deleted due to missingness) AIC: 239.27  Number of Fisher Scoring iterations: 5   Now the respondents status as in a treatment group has the expected effect. Was this a valid set of steps? How can I interpret the interaction term ControlFALSE:PreferA? Are the other coefficients still the log odds?
24143
An investigator wishes to produce a combined analysis of several datasets.  In some datasets there are paired observations for treatment A and B.  In others there are unpaired A and/or B data.  I am looking for a reference for an adaptation of the t-test, or for a likelihood ratio test, for such partially paired data.  I am willing to (for now) to assume normality with equal variance and that the population means for A are the same for each study (and likewise for B).
37588
I know this sounds contradictory, if we think of main effects, as the random factor would remove some of the variation that we want to explain by the fixed effect.  But let’s look at a study design where fields were paired (according to land use in the surroundings, distance etc.). In each field pair one was randomly assigned to be sown with an insecticide while the other one served as control. In each field several bee colonies were placed and a response variable (e.g. the weight) was measured.   # Example data:   set.seed(123)  colony = as.factor(1:96) colony_effect = rnorm(96, mean = 2)  field = as.factor(sort(rep(c(1:16),6))) field_e = rnorm(16, mean = 2) field_effect = rep(field_e, each = 6)  field_pair = as.factor(sort(rep(c(1:8),12))) field_pair_e = rnorm(8, mean = 2) field_pair_effect = rep(field_pair_e, each = 12)  treatment = as.factor(rep(c(rep("control", 6), rep("treat", 6)), 8)) treatment_effect = rep(c(rep(0, 6), rep(1, 6)), 8)       response = treatment_effect + field_effect + field_pair_effect + colony_effect df1 = data.frame(treatment, field_pair, field, colony, response)   Intuitively, I would specify the model like this, if I want to test for a treatment effect on the response:   library(lme4) lmm2 = lmer(response ~ treatment + (1|field_pair/field), data = df1) summary(lmm2) anova(lmm2, update(lmm2,.~1 + (1|field_pair/field)), test = "LRT") # p = 0.008118   I think my colleagues used the same model only that they specified it slightly differently:  lmm2_2 = lmer(response ~ treatment + (1|field_pair) + (1|field_pair:treatment), data = df1) summary(lmm2_2) anova(lmm2_2, update(lmm2_2,.~1 + (1|field_pair) + (1|field_pair:treatment)), test = "LRT") #  p = 0.008118   However, an answer in this  on a question on the choice of random factors, made me wonder whether or not field pair should be included in the model. The asker wanted to explain the fish growth of two fish populations that were placed in several tanks and exposed to two temperature regimes (crossed fixed effects: population type and temperature).  @Aron answered:     ”It doesn't make sense to both include tank as a random effect and   nest tank within the pop/temp fixed effect. You only need one of   these, depending on how tank is coded.”   And:     “Including the tank random effect is only desired if the tanks were   first divided into two groups and then randomized to treatment; if the   eight tanks were completely randomized to treatment, this is not   necessary.”   This made me wonder whether the random structure that my colleagues used (and that I intend to use) is correct, since   Treatment is both fixed and random effect Field pair was included rather than just field (which I believe would be analogous to tank in the fish example, if the tanks were grouped beforehand)  For comaparison, model without field pair:   lmm1 = lmer(response ~ treatment + (1|field), data = df1) summary(lmm1) anova(lmm1, update(lmm1,.~1 + (1|field)), test = "LRT") # p = 0.0126
78617
I Googled this, but some answers still leave me confused. One of the answers says:       "I just think of it in terms of the fitted line having 2 parameters, the intercept and the slope. But the intercept doesn't count after we adjust for the mean (which we do in obtaining SStotal), so that leaves just one df for regression."   How can you drop the intercept after adjust for the mean?
81436
When I estimate a difference in differences model with two time periods, the equivalent regression model would be  a.  $Y_{ist} = \alpha +\gamma_s*Treatment + \lambda d_t + \delta*(Treatment*d_t)+ \epsilon_{ist}$   where $Treatment$ is a dummy which is equal to 1 if the observation is from the treatment group and  $d$ is a dummy which is equal to 1 in the time period after the treatment occured   Thus the equation takes the following values.   Control group, before treatment: $\alpha$ Control group, after treatment: $\alpha +\lambda$ Treatment group, before treatment: $\alpha +\gamma$ Treatment group, after treatment:  $\alpha+ \gamma+ \lambda+ \delta$   Hence, in a two period model the difference in differences estimate is $\delta$.  But what happens concerning $d_t$ if I have more than one pre and post treatment period? Do I still use a dummy that indicates whether a year is before or after the treatment?  Or do I add year dummies instead without specifying whether each year belong to the pre or post treatment period? Like this:  b. $Y_{ist} = \alpha +\gamma_s*Treatment + yeardummy +     \delta*(Treatment*d_t)+ \epsilon_{ist}$  Or can I include both (i.e $yeardummy +\lambda d_t$)?  c. $Y_{ist} = \alpha +\gamma_s*Treatment  + yeardummy + \lambda d_t +     \delta*(Treatment*d_t)+ \epsilon_{ist}$  In conclusion, how do I specify a difference in differences model with multiple time periods (a,b or c)?
107887
Accidentally posted this in stackoverflow, so reposting here. I have a repeated-measures design experiment. Each subject took part in some experimental conditions, each one associated with several events. Normally I would analyse this with an ANOVA by grouping across all events, but I want to include those individual events in the model, and also account for subject effects. Example data import numpy as np import pandas as pd  condition = ['A']*3 + ['B']*3 + ['C']*3 events = list(range(1, len(condition)+1)) subject = [1]*len(events)   data = {     'subject':subject+[i+1 for i in subject]+[i+2 for i in subject],     'events':events*3,     'condition':condition*3     }  df = pd.DataFrame(data) np.random.seed(1) df['y'] = [np.random.randint(0,2) for i in range(df.shape[0])]  print(df)   print(df.groupby(['subject', 'condition'], as_index=False)['y'].mean())  So the full data:    subject  events  condition   y 1   1   A   1 1   2   A   1 1   3   A   0 1   4   B   0 1   5   B   1 1   6   B   1 1   7   C   1 1   8   C   1 1   9   C   1 2   1   A   0 2   2   A   0 2   3   A   1 2   4   B   0 2   5   B   1 2   6   B   1 2   7   C   0 2   8   C   0 2   9   C   1 3   1   A   0 3   2   A   0 3   3   A   0 3   4   B   1 3   5   B   0 3   6   B   0 3   7   C   1 3   8   C   0 3   9   C   0   And the grouped data: subject condition   y 1   A   0.666666667 1   B   0.666666667 1   C   1 2   A   0.333333333 2   B   0.666666667 2   C   0.333333333 3   A   0 3   B   0.333333333 3   C   0.333333333  I would like to know how to do a multi-level (logistic regression?) model to account for events and subjects as well as the condition as factors. It's a bit strange to me because when we group together and analyse the means we can use a standard ANOVA, but when we retain the full dataset and analyse it the outcome measure is of course binary, meaning we should use some sort of logistic regression. Is this correct? And if so, is this possible given the within-subjects design? If so, how do we achieve this in python (or R)?
19394
I am building a classification model (let's say logistic regression) using 2 different data sets. The data sets do not come from the same population. Let's say the data sets comes from different space-time coordinates (Asia in Fall vs North America in Winter). I have been told that simply mixing the data sets into one and applying a classification model is not right (closest thing I understood was concerns around bias). Are there any techniques or models that I may use to analyze these 2 data sets as one? Note that the variables/features all align (including the target variable, which is binary).
6777
I've read that it's bad to do this, but am looking for details as to why.  Suppose we're trying to fit the linear model $Y_i = \beta_0 + \beta_1 X_{1i} + \beta_2 X_{2i} + \epsilon_i$ where $Y$ is continuous and $X$s are categorical with many levels. We don't have any opinion on how the levels should be grouped. We then calculate the average response for each level of the predictors, then collapse levels with similar averages.  How badly does this screw up parameter estimates and inferences on them, and why?
4982
My data consists of two raters interpreting one specific phenomenon to occur at different points in time (the observations are not paired, the raters actually identified different amounts of observations). I have two questions:  1) What do I call these data? "Time-series data" seems too general and usually refers to metric data changing continuously over time (while I have just points of data along the time line). Under "time-point data" I don't find problems of the kinds described in question (2).  2) What indices for interrater reliability can I use (in R)? (If an index requires defining how much offset is tolerated, that could be 0.120 seconds.)  example data (in seconds)  rater1:  181.23   181.566  181.986  182.784  183.204  191.352  193.956  195.426  197.568  197.82   198.576  202.02   205.8    206.136  208.53   209.034  216.216  220.08   220.584  230.706  238.266  238.518  239.442  241.5    241.836  244.398    rater2:  181.902  182.784  183.204  193.956  195.384  197.694  197.82   198.576  199.5    202.146  205.8    206.136  208.53   216.258  219.576  220.542  222.096  222.558  226.002  228.312  229.11   230.244  230.496  230.832  231.504  232.554  238.266  238.518  238.602  238.938  241.5    241.836  244.272   I asked about a different part of this problem already . But I guess that was not the ideal community and I supplied insufficient data.
12475
Imagine the following common design:   100 participants are randomly allocated to either a treatment or a control group the dependent variable is numeric and measured pre- and post- treatment   Three obvious options for analysing such data are:   Test the group by time interaction effect in mixed ANOVA Do an ANCOVA with condition as the IV and the pre- measure as the covariate and post measure as the DV Do a t-test with condition as the IV and pre-post change scores as the DV   Question:    What is the best way to analyse such data?  Are there reasons to prefer one approach over another?
68218
Not so uncommon occurrence when dealing with complex maximal mixed models (estimating all possible random effects for given data and model) is perfect (+1 or -1) or nearly perfect correlation among some random effects. For the purpose of the discussion, let's observe the following model and model summary   Model: Y ~ X*Cond + (X*Cond|subj)  # Y = logit variable   # X = continuous variable   # Condition = values A and B, dummy coded; the design is repeated  #             so all participants go through both Conditions   # subject = random effects for different subjects    Random effects:  Groups  Name             Variance Std.Dev. Corr               subject (Intercept)      0.85052  0.9222                              X                0.08427  0.2903   -1.00                      CondB            0.54367  0.7373   -0.37  0.37                X:CondB          0.14812  0.3849    0.26 -0.26 -0.56 Number of obs: 39401, groups:  subject, 219  Fixed effects:                  Estimate Std. Error z value Pr(&gt;|z|)     (Intercept)       2.49686    0.06909   36.14  &lt; 2e-16 *** X                -1.03854    0.03812  -27.24  &lt; 2e-16 *** CondB            -0.19707    0.06382   -3.09  0.00202 **  X:CondB           0.22809    0.05356    4.26 2.06e-05 ***   The supposed reason behind these perfect correlations is that we have created a model which is too complex for the data that we have. The common advice that is given in these situations is (e.g., Matuschek et al., 2017; ) to fix the overparameterized coefficients to 0, because such degenerate models tend to lower the power. If we observe a marked change in fixed effects in a reduced model, we should accept that one; if there is no change, then there is no problem in accepting the original one.  However, let's presume that we are not interested only in fixed effects controlled for RE (random effects), but also in the RE structure. In the given case, it would be theoretically sound to assume that Intercept and the slope X have non-zero negative correlation. Several questions follow:   What to do in such situations? Should we report the perfect correlation and say that our data is not "good enough" to estimate the "real" correlation? Or should we report the 0 correlation model? Or should we maybe try to set some other correlation to 0 in hope that the "important" one will not be perfect anymore? I don't think there are any 100% correct answers here, I would mostly like to hear your opinions. How to write the code which fixes the correlation of 2 specific random effects to 0, without influencing the correlations between other parameters?
55238
This is a problem of my homework. An asset price is modeled by using a sequence of independent and identically distributed continuous random variables $X_1,X_2,...$, with common density function $f(x)$. We say that a record price occurs at time $n$ if $Xn &gt; \max(X_1,X_2,...,X_{n-1})$. Compute $P[a\ record\ price\ occurs\ at\ time\ n]$. Next, consider the variable $Y_i$ defined as: $Y_i = 1$ if a record occurs at time $i$. $Y_i = 0$ otherwise. Give the expected number of records by time n.  I already know the probability of a record price occurs at time $n$ is $\frac{1}{n}$. And let $Z_n$ be the number of records by time n. Then $Z_n = Y_1 + Y_2 + Y_3 +...+ Y_n$. Then the expected value of $Z_n$ should be $1 + 1/2 + 1/3 +...+ 1/n$. But in my homework, my professor suggested that "do not attempt to calculate the sum". Is calculating the sum here wrong? Or there is a better way to do the problem?  Thank you!