After running the lm regression model using R, sometime one is bound to get very small P values or values in the covariance matrix. Something of the sort: -1.861246e-04 for example in a covariance matrix. My question is how do you read this? how could I perform calculations on it manually? I needed to multiply this covariance matrix out with a vector but since I do not know how to manipulate those numbers, I am lost.
33635
I have a problem with the interpretation of a test result in which the p-value is 8.6e-28. How should it be interpreted? What is the e for?  Specifically, I used White's Test for the homoscedasticity assumption of a linear regression:  White's general test statistic :  150.0535  Chi-sq( 9)  P-value =  8.6e-28
41718
I did the multiple linear regression on a dataset of 412 observations, with one response variable (Y) and 25 explanatory variables(X1-X25). Y and most of Xs are not normally distributed. Besides, there are some correlation between several Xs. The plot show that the residuals strongly correlated with Y positively and weakly correlated with fitted Y negatively.(Sorry.As I'm newer in this website, I am n't allowed to post images.)  To address these problems, I have tried the principle component regression, the weighted least squared regression and the ridge regression. They all didn't work. I want to know what's wrong with the regression. Why did the residuals correlate with observed Y so obvious?
44585
Is it possible to obtain a positive correlation between a regressor and a response (+0,43) and, after that, obtain a negative coefficient in the fitted regression model for this regressor?  I'm not talking about changes in the sign of the regressor among some models. The coefficient sign always remains.  Could  the remaining variables of the fitted model influence the changing of the sign?
102210
When computing the covariance matrix of a sample, is one then guaranteed to get a symmetric and positive-definite matrix?  Currently my problem has a sample of 4600 observation vectors and 24 dimensions.
37344
I watched several videos on linear regression, mainly from . As I have no background in statistics, I thought this was a good way to get an idea of the topic. However I'm currently writing a bachelor thesis on Structural Equation Modeling and I want to get a deeper understanding of Regression Analysis.  The Book should roughly include these topics:   linear least squares regression variance, covariance regression coefficient coefficient of determination residual analysis (esp. leverage effect) multiple regression   I especially enjoyed the videos on khan academy, of the proof of minimizing the squared error to the regression line or illustrations like this one from Wikipedia:    I don't like it when books only present a formula and some common rules etc. without any further explanation of these.  F.e. Residuals should be normally distributed.  However I'm also not comfortable with strictly mathematical books, so I'm searching for something in between. A plus would be example code in R, but definitely not a must have.
92298
I have just read a paper in which the authors carried out a multiple regression with two predictors. The overall r-squared value was 0.65. They provided a table which split the r-squared between the two predictors. The table looked like this:              rsquared beta    df pvalue whole model     0.65   NA  2, 9  0.008 predictor 1     0.38 1.01 1, 10  0.002 predictor 2     0.27 0.65 1, 10  0.030   In this model, ran in R using the mtcars dataset, the overall r-squared value is 0.76.  summary(lm(mpg ~ drat + wt, mtcars))  Call: lm(formula = mpg ~ drat + wt, data = mtcars)  Residuals:     Min      1Q  Median      3Q     Max  -5.4159 -2.0452  0.0136  1.7704  6.7466   Coefficients:             Estimate Std. Error t value Pr(&gt;|t|)     (Intercept)   30.290      7.318   4.139 0.000274 *** drat           1.442      1.459   0.989 0.330854     wt            -4.783      0.797  -6.001 1.59e-06 *** --- Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1  Residual standard error: 3.047 on 29 degrees of freedom Multiple R-squared:  0.7609,    Adjusted R-squared:  0.7444  F-statistic: 46.14 on 2 and 29 DF,  p-value: 9.761e-10   How can I split the r-squared value between the two predictor variables?
37989
I have following simple X and Y vectors:   &gt; X [1] 1.000 0.063 0.031 0.012 0.005 0.000 &gt; Y [1] 1.000 1.000 1.000 0.961 0.884 0.000 &gt;  &gt; plot(X,Y)     I want to do regression using log of X. To avoid getting log(0), I try to put +1 or +0.1 or +0.00001 or +0.000000000000001 :   &gt; summary(lm(Y~log(X))) Error in lm.fit(x, y, offset = offset, singular.ok = singular.ok, ...) :    NA/NaN/Inf in 'x' &gt; summary(lm(Y~log(1+X)))  Call: lm(formula = Y ~ log(1 + X))  Residuals:        1        2        3        4        5        6  -0.03429  0.22189  0.23428  0.20282  0.12864 -0.75334   Coefficients:             Estimate Std. Error t value Pr(&gt;|t|)   (Intercept)   0.7533     0.1976   3.812   0.0189 * log(1 + X)    0.4053     0.6949   0.583   0.5910   --- Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1  Residual standard error: 0.4273 on 4 degrees of freedom Multiple R-squared:  0.07838,   Adjusted R-squared:  -0.152  F-statistic: 0.3402 on 1 and 4 DF,  p-value: 0.591  &gt; summary(lm(Y~log(0.1+X)))  Call: lm(formula = Y ~ log(0.1 + X))  Residuals:        1        2        3        4        5        6  -0.08099  0.20207  0.23447  0.21870  0.15126 -0.72550   Coefficients:              Estimate Std. Error t value Pr(&gt;|t|)   (Intercept)    1.0669     0.3941   2.707   0.0537 . log(0.1 + X)   0.1482     0.2030   0.730   0.5058   --- Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1  Residual standard error: 0.4182 on 4 degrees of freedom Multiple R-squared:  0.1176,    Adjusted R-squared:  -0.103  F-statistic: 0.5331 on 1 and 4 DF,  p-value: 0.5058  &gt; summary(lm(Y~log(0.00001+X)))  Call: lm(formula = Y ~ log(1e-05 + X))  Residuals:        1        2        3        4        5        6  -0.24072  0.02087  0.08796  0.13872  0.14445 -0.15128   Coefficients:                Estimate Std. Error t value Pr(&gt;|t|)     (Intercept)     1.24072    0.12046  10.300 0.000501 *** log(1e-05 + X)  0.09463    0.02087   4.534 0.010547 *   --- Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1  Residual standard error: 0.1797 on 4 degrees of freedom Multiple R-squared:  0.8371,    Adjusted R-squared:  0.7964  F-statistic: 20.56 on 1 and 4 DF,  p-value: 0.01055  &gt;  &gt; summary(lm(Y~log(0.000000000000001+X)))  Call: lm(formula = Y ~ log(1e-15 + X))  Residuals:         1         2         3         4         5         6  -0.065506  0.019244  0.040983  0.031077 -0.019085 -0.006714   Coefficients:                Estimate Std. Error t value Pr(&gt;|t|)     (Intercept)     1.06551    0.02202   48.38 1.09e-06 *** log(1e-15 + X)  0.03066    0.00152   20.17 3.57e-05 *** --- Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1  Residual standard error: 0.04392 on 4 degrees of freedom Multiple R-squared:  0.9903,    Adjusted R-squared:  0.9878  F-statistic: 406.9 on 1 and 4 DF,  p-value: 3.565e-05   The output is different in all cases. What is the correct value to put to avoid log(0) in regression? What is the correct method for such situations.  Edit: my main aim is to improve prediction of the regression model by adding log term, i.e.: lm(Y ~ X + log(X))
5647
Let's have some linear model, for example just simple ANOVA:  # data generation set.seed(1.234)                       Ng &lt;- c(41, 37, 42)                     data &lt;- rnorm(sum(Ng), mean = rep(c(-1, 0, 1), Ng), sd = 1)       fact &lt;- as.factor(rep(LETTERS[1:3], Ng))   m1 = lm(data ~ 0 + fact) summary(m1)   Result is as follows:  Call: lm(formula = data ~ 0 + fact)  Residuals:      Min       1Q   Median       3Q      Max  -2.30047 -0.60414 -0.04078  0.54316  2.25323   Coefficients:       Estimate Std. Error t value Pr(&gt;|t|)     factA  -0.9142     0.1388  -6.588 1.34e-09 *** factB   0.1484     0.1461   1.016    0.312     factC   1.0990     0.1371   8.015 9.25e-13 *** --- Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1   Residual standard error: 0.8886 on 117 degrees of freedom Multiple R-squared: 0.4816,     Adjusted R-squared: 0.4683  F-statistic: 36.23 on 3 and 117 DF,  p-value: &lt; 2.2e-16    Now I try two different methods to estimate confidence interval of these parameters  c = coef(summary(m1))  # 1st method: CI limits from SE, assuming normal distribution cbind(low = c[,1] - qnorm(p = 0.975) * c[,2],      high = c[,1] + qnorm(p = 0.975) * c[,2])  # 2nd method confint(m1)   Questions:   What is the distribution of estimated linear regression coefficients? Normal or $t$? Why do both methods yield different results? Assuming normal distribution and correct SE, I'd expect both methods to have the same result.   Thank you very much!  data ~ 0 + fact  EDIT after an answer:  The answer is exact, this will give exactly the same result as confint(m1)!  # 3rd method cbind(low = c[,1] - qt(p = 0.975, df = sum(Ng) - 3) * c[,2],      high = c[,1] + qt(p = 0.975, df = sum(Ng) - 3) * c[,2])
36884
Hey guys I found one or two papers which use ridge regression (for basketball data). I was always told to standardize my variables if I ran a ridge regression, but I was simply told to do this because ridge was scale variant (ridge regression wasn't really part of our course, so our lecturer skimmed through it).  These papers I read didn't standardize their variables, which I found a bit surprising. They also ended up with large values of lambda (around the 2000-4000 level) via cross-validation, and I was told that this is due to not standardizing the variables.  How exactly does leaving the variable(s) unstandardised lead to high lambda values and also, what are the consequences of not standardizing the variables in general? Is it really such a big deal?  Any help is much appreciated.
20438
What is Hat matrix and leverages in classical multiple regression? What are their roles? And Why do use them?   Please explain them or give satisfactory book/ article references to understand them.
92654
Suppose I have a linear regression model Y ~ b1*x1 + b2*x2 with the sample data:          x1      x2        y 0  -0.7251 -0.6033  10.2508 1  -1.3148  3.7067  10.3054 2  -1.2818 -5.8136   7.2010 3   0.6154  3.6336  11.2807 4   1.4887 -2.8337  10.6299 5  -2.0224 -0.3931   7.4502 6   0.5419 -1.7807  10.7339 7  -1.0745 -5.0296   5.8734 8   3.2116 -0.4681  11.7184 9   1.6767 -4.6040   9.0145 10 -1.3355 -2.6874   6.5014 11  0.8296  2.6793  11.8295 12 -2.8647  4.2916   7.8414 13  1.8445 -3.3924   9.6520 14 -0.1036  2.4273  11.9766 15  1.0886 -3.5636  10.6023 16 -3.3026 -6.9837   4.9962 17  1.4050 -3.8453  10.4630 18 -0.3111 -1.4278   9.2283 19 -3.5454  2.4960   7.3720   And using the following R code to get the Analysis of Variance Table:  &gt; df &lt;- read.csv('f:/data.csv', header=T, sep=",") &gt; model_All &lt;- lm(y ~ x1 + x2, data=df) &gt; anova(model_All) Analysis of Variance Table  Response: y           Df Sum Sq Mean Sq F value    Pr(&gt;F)     x1         1 44.440  44.440  58.277 6.865e-07 *** x2         1 26.640  26.640  34.934 1.711e-05 *** Residuals 17 12.964   0.763                       --- Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1   And I can calculate the F value of hypothesis test H0： b2=0 via:  &gt; model_Drop2 &lt;- lm(y ~ x1, data=df) &gt; SSR_All = sum(resid(model_All)^2) &gt; SSR_Dp2 = sum(resid(model_Drop2)^2) &gt; F_x2 = ((SSR_Dp2 - SSR_All) / 1) / (SSR_All / (20 - 3)) &gt; F_x2 [1] 34.93434   We can see that F_x2 is equal to F value of x2 in Analysis of Variance Table. However, the value F_x1 calculated as follow is not equal to F value of x1, I can't understand why:  &gt; model_Drop1 &lt;- lm(y ~ x2, data=df) &gt; SSR_Dp1 = sum(resid(model_Drop1)^2) &gt; F_x1 = ((SSR_Dp1 - SSR_All) / 1) / (SSR_All / (20 - 3)) &gt; F_x1 [1] 70.30392   So, what is the mathematics definition of the F value in table?  EDIT The result of the features reversed as commented by @Michael M:  &gt; model_All2 &lt;- lm(y ~ x2 + x1, data=df) &gt; anova(model_All2) Analysis of Variance Table  Response: y           Df Sum Sq Mean Sq F value    Pr(&gt;F)     x2         1 17.468  17.468  22.907 0.0001718 *** x1         1 53.612  53.612  70.304 1.914e-07 *** Residuals 17 12.964   0.763                       --- Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
76747
I'm trying to fit a linear model on some data with just one predictor (say (x,y)). The data is such that for small values of x, the y values give a tight fit to a straight line, however as x values increase, the y values become more volatile. Here is an example of such data (R code)  y = c(3.2,3.4,3.5,3.8,4.2,5.5,4.5,6.8,7.4,5.9) x = seq(1,10,1)   I'm curious to know if there exists any power transform (Box cox perhaps?) that lets me get a better fit for the data than simply doing a linear fit as shown below.  fit = lm(y ~ x)
35890
I am trying to generate two continuous predictors and one continuous outcome from mvrnorm. Now, I am trying to set up both standardized slopes at 0.3. This is my R code. n.tr=10 mu = c(3, 3, 3) Sigma = cbind(c(0.91,0,0.3),               c(0 0.91,0.3),               c(0.3,0.3,0.91) dat =  as.data.frame(mvrnorm(n=n.tr, mu=mu, Sigma=Sigma,                       empirical=TRUE)) colnames(dat) = c('z', 'w','y')  I want both of the standardized slopes to be 0.3, and 0 correlation between z and w. I also want the variance of y to be 1, so the residual variance of y is 0.91. If i want the diagonal of the matrix to be residual variance, how can I set up the matrix?
42482
My situation:   small sample size: 116 binary outcome variable long list of explanatory variables: 44 explanatory variables did not come from the top of my head; their choice was based on the literature.   Statistical test chosen: logistic regression  I need to find the variables that best explain variations in the outcome variable (I am not interested in making predictions).  The problem: This question is a follow-up on the 2 questions listed below. From them, I got that performing automated stepwise regression has its downsides. Anyway, it seems that my sample size would be too small for that. It seems that my sample is also too small to enter all variables at once (using the SPSS 'Enter' method). This leaves me with my issue unresolved: how can I select a subset of variables from my original long list in order to perform multivariate logistic regression analysis?  UPDATE1: I am not an statistician, so I would appreciate if jargons can be reduced to the minimum. I am working with SPSS and am not familiar with other packages, so options that could be run with that software would be highly preferable.  UPDATE2: It seems that SPSS does not support LASSO for logistic regression. So following one of your suggestions, I am now struggling with R. I have passed through the basics, and managed to run a univariate logistic regression routine successfully using the glm code. But as I tried glmnet with the same dataset, I am receiving an error message. How could I fix it? Below is the code I used, followed by the error message:  data1 &lt;- read.table("C:\\\data1.csv",header=TRUE,sep=";",na.string=99:9999)  y &lt;- data1[,1]  x &lt;- data1[,2:45]  glmnet(x,y,family="binomial",alpha=1)    **in lognet(x, is.sparse, ix, jx, y, weights, offset, alpha, nobs,  :  (list) object cannot be coerced to type 'double'**   UPDATE3: I got another error message, now related to missing values. My question concerning that matter is .
119620
Suppose, I have performed multiple regression analysis on the following data set where X1 and X2 are independent variables and Y is the dependent variable.    And achieved the following multiple regression analysis table from where it is clear that X2 is not correlated with Y as the p-value is greater than 0.05 which is 0.108 for X2 variable     So, if p value is greater than 0.05 for a variable, then why it is considered in regression equation  Y = 2.709 + 0.763(X1) + 0.463(X2) which has been shown in any online multiple regression equation calculator.  My question is Why it is not like the following Y = 2.709 + 0.763(X1) omitting the [0.463(X2)] part from the equation, as X2 is not correlated with Y  Please help to clarify the conceptual idea....
106714
After searching here:    I am still curious about this.  Is it possible to derive the standard error of a regression coefficient from summary data alone?   E.g., assume we are given the following variance-covariance matrix.  $\begin{bmatrix}Var(X) &amp; Cov(X,Y)\\Cov(X,Y) &amp; Var(Y)\end{bmatrix}$  We can derive the regression coefficient $\beta_{XY} = Cov(X,Y) / Var(X)$.  Given a specific n, is it possible to derive the standard error of $\beta$ as well? If so, which formula is being used? It appears that all formulas for regression standard errors that I could find assume that you know the variance of residuals of the regression, which we don't know from summary data alone.  ---UPDATE--- Based on feedback by Greg and whuber, I am now at a point, where I know that  $\sigma^2(b) = \sigma^2(X'X)^{-1}=\sigma^2(Var(X)(n-1))^{-1}$,  but typically matrix formulas for the standard error of $b$ then note that $\sigma^2$, the unknown variance of the errors, is estimated with MSE.   Based on comments by whuber, sum of squares residuals is:  $Y'Y - bX'X$, which according to Greg can be spelled out as  $Var(Y)(n-1) - b(Var(x)(n-1))$  And to go from here to MSE, I think (?) all that is left is divide by n-2,  so $(Var(Y)(n-1) - b(Var(x)(n-1))) / (n-2)$  And plugging this all in yields:  $\sigma^2(b) =((Var(Y)(n-1) - b(Var(X)(n-1))) / (n-2))\times(Var(X)(n-1))^{-1}$  I will try to see if this works out in R, and will report back. If anyone sees something blatantly wrong, I appreciate the heads up.
47903
I would like to get a p-value and an effect size of an independent categorical variable (with several levels) -- that is "overall" and not for each level separately, as is the normal output from lme4 in R. It is just like the thing people report when running an ANOVA.  How can I get this?