Source |       SS       df       MS              Number of obs =     300 -------------+------------------------------           F(  5,   294) =16580.50        Model |  1.1499e+10     5  2.2998e+09           Prob &gt; F      =  0.0000     Residual |  40778600.2   294  138702.722           R-squared     =  0.9965 -------------+------------------------------           Adj R-squared =  0.9964        Total |  1.1540e+10   299  38593905.2           Root MSE      =  372.43   Should I be worried about this- does your F statistic size matter?  How do i find out what the p value of my model is - is it the p value attached to _cons in the bottom of my coefficients?  what do i test this F stat against and what warrants significance?
120636
I seem to have confused myself trying to understand if a $r$-squared value also has a $p$-value.  As I understand it, in linear correlation with a set of data points $r$ can have a value ranging from $-1$ to $1$ and this value, whatever it is, can have a $p$-value which shows if $r$ is significantly different from $0$ (i.e., if there is a linear correlation between the two variables).  Moving on to linear regression, a function can be fitted to the data, described by the equation $Y = a + bX$. $a$ and $b$  (intercept and slope) also have $p$-values to show if they are significantly different from $0$.  Assuming I so far have understood everything correct, are the $p$-value for $r$ and the $p$-value for $b$ just the same thing? Is it then correct to say that it is not $r$-squared that has a $p$-value but rather $r$ or $b$ that does?
79869
I have the mean and standard deviation of a large sample (30k obs) and the mean and std of a subsample (2k obs). The means are percentage figures (i.e. between 0 and 100% and hence not a normal variate). Also the means are not independent, since one is based on a subsample from the other.    Despite seemingly a harmless question, I could not find an appropriate testing procedure. Any ideas on how to test whether the mean of the subsample is different to the mean of the entire sample are much appreciated.
3127
When I sum up the sizes of my files, I get one figure. If I run du, I get another figure. If I run du on all the files on my partition, it doesn't match what df claims is used. Why are there so many different figures for the total size of my files? Can't computers add?  Speaking of adding: when I add the “Used” and “Available” columns of df, I don't get the total figure. And that total figure is smaller than the size of my partition. And if I add up my partition sizes I don't get my disk size! What gives?
95747
Suppose we have $\mu = 19.1$ and $\sigma = 1.7$.  I need to find $a \in \mathbb{R}$ such that: $P(|X-19.1|\leq a) = 0.98$ Now in the correction it is given that:  $$P(|X-19.1|\leq a) = 0.98 \iff P(\frac{-a}{1.7} \leq \frac{X - 19.1}{1.7} \leq \frac{a}{1.7})=0.98 \iff P(\frac{X-19.1}{1.7} \leq \frac{a}{1.7})=0.99$$  Can someone explain to me why is the last equality correct?
118525
Performing a regression and need to find out if my residuals are normally distributed.
17052
I have a data that where the outcome is the proportion of a species observed in an area by a machine on 2 separate days. Since the outcome is a proportion and does not include 0 or 1 I used a beta regression to fit the model. Temperature is used as  an independent variable. Here is some toy R code:  set.seed(1234) library(betareg) d &lt;- data.frame(   DAY = c(1,1,1,1,2,2,2,2),   Proportion = c(.4,.1,.25, .25, .5,.3,.1,.1),   MACHINE = c("A","B","C","D","H","G","K","L"),   TEMPERATURE = c(rnorm(8)*100) ) b &lt;- betareg(Proportion ~ TEMPERATURE,   data= d, link = "logit", link.phi = NULL, type = "ML") summary(b) ## Call: ## betareg(formula = Proportion ~ TEMPERATURE, data = d, link = "logit", link.phi = NULL, type = "ML") ##  ## Standardized weighted residuals 2: ##     Min      1Q  Median      3Q     Max  ## -1.2803 -1.2012  0.3034  0.6819  1.6494  ##  ## Coefficients (mean model with logit link): ##               Estimate Std. Error z value Pr(&gt;|z|)     ## (Intercept) -1.0881982  0.2620518  -4.153 3.29e-05 *** ## TEMPERATURE  0.0003469  0.0023677   0.147    0.884     ##  ## Phi coefficients (precision model with identity link): ##       Estimate Std. Error z value Pr(&gt;|z|)   ## (phi)    9.305      4.505   2.066   0.0389 * ## --- ## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1    Above you can see the TEMPERATURE coefficient is .0003469. Exponentiating, exp(.0003469) = 1.000347  Update incorporating replies and comments:  You can see here how increasing temperature by 1 unit from -10 to 10 increase the proportion  nd &lt;- data.frame(TEMPERATURE = seq(-10, 10, by = 1)) nd$Proportion &lt;- predict(b, newdata = nd) nd$proportion_ratio &lt;- nd$Proportion/(1 - nd$Proportion) plot(Proportion ~ TEMPERATURE, data = nd, type = "b")   The interpretation is: A 1-unit change in TEMPERATURE leads to a relative change of 1.000347 ≈0.04%  in the Proportion: $$\frac{\mathrm{E}(\mathtt{Proportion})}{1-\mathrm{E}(\mathtt{Proportion})}$$   The key word there is relative change so when you compare exp(coef(b))[2] to nd$proportion_ratio[2] / nd$proportion_ratio[1]  you'll see they are the same  ## ratio of proportion nd$proportion_ratio[2] / nd$proportion_ratio[1]  exp(coef(b))[2] nd$proportion_ratio[-1] / nd$proportion_ratio[-20]
42935
My PDF:     M was estimated and found to be 5.   I need to work out the quartiles for the PDF above. In addition, I need to use different methods of estimation to estimate the parameters. So far I've successfully used the method of moments. I'm finding it difficult to form the likelihood function.  This is the approach I took to work out the quartiles, which I believe is not correct; quartile 1 below:    And this is what I have so far for the likelihood function:    Any help would be truly appreciated! Thank you.
78617
I Googled this, but some answers still leave me confused. One of the answers says:       "I just think of it in terms of the fitted line having 2 parameters, the intercept and the slope. But the intercept doesn't count after we adjust for the mean (which we do in obtaining SStotal), so that leaves just one df for regression."   How can you drop the intercept after adjust for the mean?
118926
I'm doing a project in predicting default risk for SBA loans using R. My data has 187,986 positives and only 17796 negatives (9.46%). Clearly this is an imbalanced dataset. That's why when I run accuracy rate checking for my model using probability cutoff value=0.5, the accuracy result is way too low as below,  &gt; mytestset$LoanStatus_PIF&lt;-ifelse(mytestset$LoanStatus_PIF=="PIF",1,0) &gt; pred&lt;-predict(final,newdata=mytestset,type="response") &gt; y_pred_num&lt;-ifelse(pred&gt;0.5,1,0) &gt; mean(y_pred_num==mytestset$LoanStatus_PIF)   Result: [1] 0.009682553  When I change the cutoff value to 0.998 and up, the result is way much impressive:  &gt; y_pred_num&lt;-ifelse(pred&gt;0.5,1,0) &gt; mean(y_pred_num==mytestset$LoanStatus_PIF) Result:  [1] 0.9997752   Question: 1. Will my model be rejected because I manually choose the cut-off value?   Does 0.999 make sense for a cutoff value? As common cut-off value is usually 0.5. Thanks !
17985
I'm trying to use those parameters and I receive a difference of ~0.5m    ALON and BSHM APN Points:  coords = [   (dms2rad([31,42,28.5694]),  dms2rad([34,36,23.92196]),  55.671),   (dms2rad([32,46,44.34472]), dms2rad([35,1,22.74061]), 225.046), ]   Transform:  def lla_to_ig05(lon, lat, alt):     import pyproj     lla = pyproj.Proj(proj='latlong', ellps='WGS84', datum='WGS84')     ig05 = pyproj.Proj("+proj=tmerc +lat_0=31.7343936111111 +lon_0=35.2045169444445 +k=1.0000067 +x_0=219529.584 +y_0=626907.39 +ellps=GRS80 +towgs84=-24.002400,-17.103200,-17.844400,-0.33077,-1.852690,1.669690,5.424800 +units=m +no_defs")     x, y, h = pyproj.transform(lla, ig05, lon, lat, alt, radians=True)     return x, y, h for lat, lon, alt in coords:   print(pd.DataFrame(lla_to_ig05(lon, lat, alt))) # pd for pandas   print("")   Output:                 0 0  162790.443392 1  624087.953905 2      55.671000                 0 0  202456.903152 1  742717.095893 2     225.046000   Real numbers:                 0 0  162789.564 1  624088.355 2      55.671000                 0 0  202456.453 1  742718.779 2     225.046000   Any hint will be very helpful!
17973
I recently came across the paper . The author raised a few common misconceptions regarding hypothesis testing and p-values, about which I have two specific questions:   The p-value is technically $P({\rm observation}|H_{0})$, which, as pointed out by the paper, generally does not tell us anything about $P(H_{0}|{\rm observation})$, unless we happen to know the marginal distributions, which is rarely the case in "everyday" hypothesis testing. When we obtain a small p-value and "reject the null hypothesis," what exactly is the probabilistic statement that we are making, since we cannot say anything about $P(H_{0}|{\rm observation})$? The second question relates to a particular statement from page 6(652) of the paper:      Since the p-value, or range of p-values indicated by stars, is not set a priori, it is not the long-run probability of making a Type I error but is typically treated as such.   Can anyone help to explain what is meant by this statement?
58420
The coefficient of determination is $$r^2 = 1 - \frac{SS_\text{res}}{SS_\text{tot}}$$ where $SS_\text{res}=\sum_{i=1}^n (y_i-\hat{y_i})^2$ and $SS_\text{tot}=\sum_{i=1}^n (y_i - \overline{y})^2$.  Why is this used for assessing the fit of a least squares line? Why is the comparison based on looking at $SS_\text{res}/SS_\text{tot}$ as opposed to say $SS_\text{res}/n$ or something else?
7003
I am observing strange patterns in residuals for my data:   [EDIT] Here are the partial regression plots for the two variables:      [EDIT2] Added the PP Plot   The distribution  seems to be doing fine (see below) but I have no clue where this straight line might be coming from. Any ideas?   [UPDATE 31.07]  It turns out you were absolutely right, I had cases where the retweet count was indeed 0 and these ~ 15 cases resulted in those strange residual patterns.   The residuals look much better now:   I've also included the partial regressions with a loess line.
17738
I'm using R to run some logistic regression. My variables were continuous, but I used cut to bucket the data. Some particular buckets for these variables always result in dependent variable being equal to 1. As expcted, the coefficient estimate for this bucket is very high, but the p-value is also high. There are about ~90 observations in either these buckets, and around 800 total observations, so I don't think it's a problem of sample size. Also, this variable should not be related to other variables, which would naturally reduce their p-values.  Are there any other plausible explanations for the high p-value?  Example:  myData &lt;- read.csv("application.csv", header = TRUE) myData$FICO &lt;- cut(myData$FICO, c(0, 660, 680, 700, 720, 740, 780, Inf), right = FALSE) myData$CLTV &lt;- cut(myData$CLTV, c(0, 70, 80, 90, 95, 100, 125, Inf), right = FALSE) fit &lt;- glm(Denied ~ CLTV + FICO, data = myData, family=binomial())   Results are something like this:  Deviance Residuals:       Min        1Q    Median        3Q       Max   -1.53831  -0.77944  -0.62487   0.00027   2.09771    Coefficients:                 Estimate Std. Error z value Pr(&gt;|z|)     (Intercept)     -1.33630    0.23250  -5.747 9.06e-09 *** CLTV(70,80]     -0.54961    0.34864  -1.576 0.114930     CLTV(80,90]     -0.51413    0.31230  -1.646 0.099715 .   CLTV(90,95]     -0.74648    0.37221  -2.006 0.044904 *   CLTV(95,100]     0.38370    0.37709   1.018 0.308906     CLTV(100,125]   -0.01554    0.25187  -0.062 0.950792     CLTV(125,Inf]   18.49557  443.55550   0.042 0.966739     FICO[0,660)     19.64884 3956.18034   0.005 0.996037     FICO[660,680)    1.77008    0.47653   3.715 0.000204 *** FICO[680,700)    0.98575    0.30859   3.194 0.001402 **  FICO[700,720)    1.31767    0.27166   4.850 1.23e-06 *** FICO[720,740)    0.62720    0.29819   2.103 0.035434 *   FICO[740,780)    0.31605    0.23369   1.352 0.176236     --- Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1  (Dispersion parameter for binomial family taken to be 1)      Null deviance: 1037.43  on 810  degrees of freedom Residual deviance:  803.88  on 798  degrees of freedom AIC: 829.88  Number of Fisher Scoring iterations: 16   FICO in the range [0, 660) and CLTV in the range (125, Inf] indeed always results in Denial = 1, so their coefficients are very large, but why are they also "insignificant"?
12253
I am trying to get my regression right. I want to see, if subs increase how much increase in revenue is seen. The dependent variable is Revenue while the independent variable is subscribers.  Least squares method regression shows the following equation 24.4x - 189,883,443. I can't make sense of it. If subs increase means increase in revenue, then why this?  There is no 0 value and the lowest value for subscribers starts with 10 m while revenue starts with 180 m  How can i make sense of y = 24.4x - 189,883,443. Why is the intercept negative. Does it mean that one sub increase would mean a revenue increase of 24.4  regression excel
58895
Been trying to this line to parse for a good 20mins, but all attempts have failed:  \beta = $-$0.84, t = $-$2.09, P = 0.04   Could anyone tell me why I'm getting a Missing $ inserted error here?
74353
I understand from this question -    that the R squared value of a linear regression model can be negative if the intercept is constrained. And this makes sense if you define R squared as -   $$R^2 = 1-\frac{SSE}{SST}$$  One says $SSE&gt;SST$. But then,  $$SST = SSA + SSE $$ Total Sum of squares  = Sum of squared errors + Sum of squared residual. And with this we get -  $$R^2 = \frac{SSA}{SST}$$ And now it is hard to imagine how $R^2$ can be negative. Aren't SSA and SST >0 always?