For a set of 48 randomly generated points using runif(), I use the command quantile(x,0.25) to calculate the first quartile in R. I don't understand what formula is used by R to calculate quartiles of ungrouped data points. The value I am getting as a result is not the median of the data from 1 to 23rd position.
34635
Given the following data:  972, 975, 985, 993, 993, 995, 998, 1001, 1004, 1007, 1008, 1009, 1011, 1015, 1016, 1020, 1022, 1032  I calculate the lower quartile as follows:  Number of values: 18, Number of 'gaps': 17.  Lower quartile is at 17/4 + 1 = 5.25th value. 5.25th value is 993 + (995 - 993) x 0.25 = 993.5  Upper quartile is at 3 x 17/4 + 1 = 13.75th value. 13.75th value is 1011 + (1015 - 1011) x 0.75 = 1014  Both the Excel QUARTILE function and the R quantile function agree with me. However the book I am using (Understanding Statistics by Graham Upton and Ian Cook) give different answers (993 and 1015 respectively). I'm confused. Which is correct?
104452
I'm looking for optimal binning method (discretization) of a continuous variable with respect to a given response (target) binary variable and with maximum number of intervals as a parameter.  example: I have a set of observations of people with "height" (numeral continuous) and "has_back_pains" (binary) variables. I want do discretize height into 3 intervals (groups) at most with different proportion of people with back aches, so that the algorithm maximizes the difference between the groups (with given restrictions for instance, that each interval has at least x observations).  The obvious solution to this problem would be to use decision trees(a simple one-variable model) , but I can't find any function in R that would have "maximal number of branches" as a parameter - all of them divide the variable into 2 gropus  (&lt;=x and >x). SAS miner has a "maximum branch" parameter but I'm looking for a non commercial solution.  some of my variables have just a few unique values (and could be treated as discrete variables) but I want to discretize them as well into a smaller number of intervals.  The closest solution to my problem is implemented in the smbinning package in R (which relies on ctree function from party package) but it has two drawbacks: it's impossible to set the number of intervals (however, you can find a way around it by changing the p parameter) and it doesn't work when data vector has less than 10 unique values. Anyway, you can see the example output here(Cutpoint and Odds columns are crucial):  Cutpoint CntRec CntGood CntBad CntCumRec CntCumGood CntCumBad PctRec BadRate   Odds  LnOdds     WoE     IV 1   &lt;= 272   9081     169   8912      9081        169      8912 0.1874  0.9814 0.0190 -3.9653 -0.6527 0.0596 2   &lt;= 311   8541     246   8295     17622        415     17207 0.1762  0.9712 0.0297 -3.5181 -0.2055 0.0068 3   &lt;= 335   2986     163   2823     20608        578     20030 0.0616  0.9454 0.0577 -2.8518  0.4608 0.0163 4  Missing  27852    1125  26727     48460       1703     46757 0.5747  0.9596 0.0421 -3.1679  0.1447 0.0129 5    Total  48460    1703  46757        NA         NA        NA 1.0000  0.9649 0.0364 -3.3126  0.0000 0.0956   Oh, I'm fully aware that binning results in information loss and that there are better methods, but I'm going to use it for data visualization and treat those variables as a factor.
97624
I'm new to this and I have a question about the results that I get from checking whether my data contains seasonality or not. I have a csv file which contains date, period and year. However, R reads the date as a factor and I didn't know how to deal with is. So I extracted the frequencies per month, so I have very little points now. The point contain the frequencies of the years 2012 and 2013. My code: k &lt;- c(1110,989,1141,674,790,739,842,1004,751,765,1246,1045,        1106,1078,1012,965,978,991,856,703,784,639,821,518)  z &lt;- ts(k, start =c(2012,1), frequency =12)  plot(decompose(z))   I really don't know how to read this an whether I can draw a conclusion about seasonality by this methods or not.
104969
I am hoping to get an intuitive, accessible explanation of quantile regression.  Let's say I have a simple dataset of outcome $Y$, and predictors $X_1, X_2$.  If, for example, I run a quantile regression at .25,.5,.75, and get back $\beta_{0,.25},\beta_{1,.25}...\beta_{2,.75}$.  Are the $\beta$ values found by simply ordering the $y$ values, and performing a linear regression based on the examples which are at/near the given quantile?  Or do all of the samples contribute to the $\beta$ estimates, with descending weights as the distance from the quantile increases?  Or is it something totally different? I've yet to find an accessible explanation.
42890
We have ground-truth data $\mathbf{x}^* = (0,0)^T \in \mathbb{R}^{2}$. Furthermore, we have $N$ measurements $\mathbf{x}_i \in \mathbb{R}^{2}, i\in \{1,\ldots,N\}$. We calculate the $N$ error vectors $\mathbf{e}_i = \mathbf{x}_i - \mathbf{x}^*$. We assume that the two components of $\mathbf{e}_i = (e_{x,i}, e_{y,i})^T $ are independent of each other and normally distributed with equal mean $\mathbf{\mu} = (0,0)^T$ and variance $\sigma^2$. The absolute errors $e_i = \sqrt {e_{x,i}^2 + {e}_{y,i}^2} $ follow a Rayleigh distribution, is that correct?  The probability density function (PDF) of the Rayleigh function is zero at a value of zero. Why is this the case (intuitively)? I would assume that our errors should most likely be 0 (i.e. PDF should be highest at 0)...
8749
Inspired by Peter Donnelly's talk at , in which he discusses how long it would take for a certain pattern to appear in a series of coin tosses, I created the following script in R. Given two patterns 'hth' and 'htt', it calculates how long it takes (i.e. how many coin tosses) on average before you hit one of these patterns.  coin &lt;- c('h','t')  hit &lt;- function(seq) {     miss &lt;- TRUE     fail &lt;- 3     trp  &lt;- sample(coin,3,replace=T)     while (miss) {         if (all(seq == trp)) {             miss &lt;- FALSE         }         else {             trp &lt;- c(trp[2],trp[3],sample(coin,1,T))             fail &lt;- fail + 1         }     }     return(fail) }  n &lt;- 5000 trials &lt;- data.frame("hth"=rep(NA,n),"htt"=rep(NA,n))  hth &lt;- c('h','t','h') htt &lt;- c('h','t','t')  set.seed(4321) for (i in 1:n) {     trials[i,] &lt;- c(hit(hth),hit(htt))     } summary(trials)   The summary statistics are as follows,        hth             htt          Min.   : 3.00   Min.   : 3.000    1st Qu.: 4.00   1st Qu.: 5.000    Median : 8.00   Median : 7.000    Mean   :10.08   Mean   : 8.014    3rd Qu.:13.00   3rd Qu.:10.000    Max.   :70.00   Max.   :42.000    In the talk it is explained that the average number of coin tosses would be different for the two patterns; as can be seen from my simulation. Despite watching the talk a few times I'm still not quite getting why this would be the case. I understand that 'hth' overlaps itself and intuitively I would think that you would hit 'hth' sooner than 'htt', but this is not the case. I would really appreciate it if someone could explain this to me.
66210
I have gone through various formulas but all were used for a small data set like 100, 200 rows, but I want to know how to calculate critical values of r for a large data set containing >5000 rows.   I have calculated using T.INV(1-alpha/2,df)/((sqrt(T.INV(1-alpha/2)^2+df)), but this formula is for small data sets, when I used it for a large data set, mentioned above, I got results which were not expected.
115415
I am trying to calculate the &quot;overall&quot; median of a variable that is spread across two datasets. I have access to the raw data in each dataset but can't bring their raw data together. What would be the best way to calculate the median (or any quantile for that matter)? What I was thinking to do was to get the quantiles from the &quot;remote&quot; dataset, say with N_1 samples and bring them to the other dataset; then sample N_1 values from these quantiles (essentially discrete empirical CDF) and use these values as &quot;raw&quot; scores to calculate the overall median or any other quantile. I wonder if there is a more sophisticated way of doing this? I can't combine the two datasets because of data governance issues, where I have to analyse them separately on different computers.
105675
I have 320 data points - each has a redshift and a turnover-frequency, and I want to fit a correlation between them (a linear fit). However, 120 of the turnover-frequency values are upper limits. As shown below the relationship is very weak:   the red points have upper limits for their turnover-frequency value. I would like to fit a straight line to this data that accounts for the upper limits. I think the tobit model in R can do this, but I'm not too sure (I'm new to R by the way); before I start with that, are there other methods in R I should consider?
80896
How does one go about obtaining a summary of the counts of cell values within a raster in QGIS? I know QGIS is doing this, in order to provide the summary statistics and histogram in the properties of the layer. I do not need the summary statistics nor a histogram image, but rather counts of individual values.   It looks like there was at one point an attempt to add a special feature for this (Feature #4321, ).   Am I missing something really obvious?
108582
If I give two quantiles $(q_1,q_2)$ and their corresponding locations $(l_1,l_2)$ (each) in the open interval $(0,1)$, can I always find parameters of a beta distribution that has those quantiles at the specified locations?
114304
I have repeated samples from the following process: Most samples will only contain points that are randomly distributed on a 2-dimensional plane. Sometimes, however, the sample will contain not only randomly distributed points but also some points that will be arranged roughly on a line. I say roughly because the points I observe may not all be on the line exactly but be randomly displaced somewhat. Here's the R code for a little toy example to demonstrate what I mean:  line_active &lt;- TRUE  line_n &lt;- 50 noise_n &lt;- 100 perturbation.sd &lt;- 0.01  # background noise noise_x &lt;- runif(noise_n) noise_y &lt;- runif(noise_n)  # line coordinates start &lt;- list(x=0.2,y=0.1) end &lt;- list(x=0.8,y=0.9)  if(line_active) {   line_pos &lt;- runif(line_n)   line_x &lt;- start$x+line_pos*(end$x - start$x)       line_y &lt;- start$y+line_pos*(end$y - start$y)    line_x_perturbed &lt;- line_x+rnorm(length(line_x),0,perturbation.sd)   line_y_perturbed &lt;- line_y+rnorm(length(line_y),0,perturbation.sd) } else {   line_x_perturbed &lt;- c()   line_y_perturbed &lt;- c() }  points &lt;- data.frame(x=c(noise_x,line_x_perturbed),y=c(noise_y,line_y_perturbed))  # plot! plot(points$x,points$y) points(line_x_perturbed,line_y_perturbed,col='red')   My questions are:   How would I go about detecting whether or not there is a line in any sample I'm looking at? How would I go about detecting the slope and intercept of the line if it is there? How much more tricky does this problem get if the noise does not come from some known distribution but has to be estimated?
5733
I have data for three unequal groups: $N = 44$, $N = 354$ and $N = 347$. Is it possible to compare all three groups running a one-way ANOVA or is the first group too small?
97850
I have plotted some data using box plots. I am comparing Condition 1 (left) and Condition 2 (Right) values. My aim is to find a point at which we make a decision where the value changes from point Condition 1 to Condition 2.   Will this conclusion make sense, if I say if I do the experiment again and get any value than the median of Condition 1, then it is likely that the value will be of Condition 2?   Or is there any other way which I can represent this data  to to get conclusion that if I get random value I can say if it is from condition 1 or condition 2?   Data presented as code for R input:  Cond.1 &lt;- c(2.9, 3.0, 3.1, 3.1, 3.1, 3.3, 3.3, 3.4, 3.4, 3.4, 3.5, 3.5, 3.6, 3.7, 3.7,             3.8, 3.8, 3.8, 3.8, 3.9, 4.0, 4.0, 4.1, 4.1, 4.2, 4.4, 4.5, 4.5, 4.5, 4.6,             4.6, 4.6, 4.7, 4.8, 4.9, 4.9, 5.5, 5.5, 5.7) Cond.2 &lt;- c(2.3, 2.4, 2.6, 3.1, 3.7, 3.7, 3.8, 4.0, 4.2, 4.8, 4.9, 5.5, 5.5, 5.5, 5.7,             5.8, 5.9, 5.9, 6.0, 6.0, 6.1, 6.1, 6.3, 6.5, 6.7, 6.8, 6.9, 7.1, 7.1, 7.1,             7.2, 7.2, 7.4, 7.5, 7.6, 7.6, 10, 10.1, 12.5)   Each condition has 39 values.
77445
I am searching for the formula that will generate the critical values in a t-test instead of using a static critical value table.   I am programming in Java and not R so the qt() function is not what I am looking for.  Any help is appreciated.
44667
I was wondering where there is a general formula to relate the expected value of a continuous random variable as a function of the quantiles of the same r.v. The expected value of r.v. $X$ is defined as: $E(X) = \int x dF_X(x) $ and quantiles are defined as :  $Q^p_X = \{x : F_X(x) = p \} =F_X^{-1}(p) $ for $p\in(0,1)$.  Is there for instance a function function $G$ such that:  $E(X) = \int_{p\in(0,1)} G(Q^p_X) dp $
33007
I have count data (demand/offer analysis with counting number of customers, depending on - possibly - many factors). I tried a linear regression with normal errors, but my QQ-plot is not really good. I tried a log transformation of the answer: once again, bad QQ-plot.  So now, I'm trying a regression with Poisson Errors. With a model with all significant variables, I get:  Null deviance: 12593.2  on 53  degrees of freedom Residual deviance:  1161.3  on 37  degrees of freedom AIC: 1573.7  Number of Fisher Scoring iterations: 5   Residual deviance is larger than residual degrees of freedom: I have overdispersion.  How can I know if I need to use quasipoisson? What's the goal of quasipoisson in this case? I read this advise in "The R Book" by Crawley, but I don't see the point nor a large improvement in my case.