I have a mixed data set where I want to compute distances between observations with Gower in R (daisy function). When I compute distances between different number of observations, the distances seem to change, and I don't fully understand why that is. I assume it's some weights for the observations being used, but I can't find anything explaining this from the documentation  library(cluster) daisy(iris[1:3,], metric = "gower") #Distance between object 1&amp;2 is 0.30 daisy(iris[1:4,], metric = "gower") #Now the distance between object 1&amp;2 is 0.28  Does anyone know why this happens?  Thanks
121524
In my dataset we have both continuous and naturally discrete variables. I want to know whether we can do hierarchical clustering using both type of variables. And if yes, what distance measure is appropriate?
80227
I'm reading a textbook and I see this question:    So there are 200 women, and the DF is 196, implying that the equation for DF is $n - k - 1$. There are 3 variables: bp, age, and type so $k == 3$. What's the intuition behind this?  Also, why is the degrees of freedom for linear regression n - 2?
40992
I'm looking at how the expected minimum Euclidean distance between randomly uniform points and the origin changes as we increase the density of random points (points per unit square) around the origin. I have managed to come up with a relationship between the two described as such:  $$\text{Expected Min Distance} =\frac{1}{2\sqrt{\text{Density}}}$$  I came up with this by running some Monte Carlo simulations in R and fitting a curve manually (code below).   My question is: could I have derived this result theoretically rather than through experimentation?  #Stack Overflow example library(magrittr) library(ggplot2)   #--------- #FUNCTIONS #--------- #gen random points within a given radius and given density gen_circle_points &lt;- function(radius, density) {   #round radius up then generate points in square with side length = 2*radius   c_radius &lt;- ceiling(radius)   coords &lt;- data.frame(     x = runif((2 * c_radius) ^ 2 * density, -c_radius, c_radius),     y = runif((2 * c_radius) ^ 2 * density, -c_radius, c_radius)   )   return(coords[sqrt(coords$x ^ 2 + coords$y ^ 2) &lt;= radius, ])#filter in circle }  #Example plot plot(gen_circle_points(radius = 1,density = 200)) #200 points around origin points(0,0, col="red",pch=19) #colour origin     #return euclidean distances of points generated by gen_circle_points() calculate_distances &lt;- function(circle_points) {   return(sqrt(circle_points$x ^ 2 + circle_points$y ^ 2)) }  #find the smallest distance from output of calculate_distances() calculate_min_value &lt;- function(distances) {   return(min(distances)) }   #Try a range of values density_values &lt;- c(1:100)  expected_min_from_density &lt;- sapply(density_values, function(density) {   #simulate each density value 1000 times and take an average as estimate for   #expected minimum distance   sapply(1:1000, function(i) {     gen_circle_points(radius=1, density=density) %&gt;%       calculate_distances() %&gt;%       calculate_min_value()   }) %&gt;% mean() })  results &lt;- data.frame(density_values, expected_min_from_density)  #fit based off exploration theoretical_fit &lt;- data.frame(density = density_values,                                fit = 1 / (sqrt(density_values) * 2))  #plot monte carlo (black) and fit (red dashed) ggplot(results, aes(x = density_values, y = expected_min_from_density)) +   geom_line() +    geom_line(     data = theoretical_fit,     aes(x = density, y = fit),     color = "red",     linetype = 2   )
35890
I am trying to generate two continuous predictors and one continuous outcome from mvrnorm. Now, I am trying to set up both standardized slopes at 0.3. This is my R code. n.tr=10 mu = c(3, 3, 3) Sigma = cbind(c(0.91,0,0.3),               c(0 0.91,0.3),               c(0.3,0.3,0.91) dat =  as.data.frame(mvrnorm(n=n.tr, mu=mu, Sigma=Sigma,                       empirical=TRUE)) colnames(dat) = c('z', 'w','y')  I want both of the standardized slopes to be 0.3, and 0 correlation between z and w. I also want the variance of y to be 1, so the residual variance of y is 0.91. If i want the diagonal of the matrix to be residual variance, how can I set up the matrix?
64709
I have conducted an experiment with multiple (categorical) conditions per subject, and multiple subject measurements.  My data-frame in short: A subject has one property, is_frisian which is either 0 or 1 depending on the subject. And it is tested for two conditions, person and condition. The measurement variable is error, which is either 0 or 1.  My mixed linear model in R is:  &gt; model &lt;- lmer(error~is_frisian*condition*person+(1|subject_id), data=output)   However, the residuals plot of this model gives an unexpected (?) result.    I was taught that this plot should show randomly scattered points, and they should be normal distributed. When plotting the density of the fitted and the residuals, it shows a reasonable normal distribution. The lines you can see in the graph, however, how is this to be explained? And is this okay?  The only thing I could come up with is that the graph has two lines due to the categorical variables. The output variable error is either 0 or 1. But I do not have that much knowledge of the underlying system to confirm this. And then again, the lines also seem to have a low negative slope, is this then perhaps a problem?  UPDATE:  &gt; model &lt;- glmer(error~is_frisian*condition*person + (1|subject_id), data=output, family='binomial') &gt; binnedplot(fitted(model),resid(model))   Gives the following result:    FINAL EDIT:  The density-plots have been omitted, they have nothing to do with satisfaction of assumptions in this case. For a list of assumptions on logistic regression (when using family=binomial),
19801
I'm trying to predict a binary outcome using 50 continuous explanatory variables (the range of most of the variables is $-\infty$ to $\infty$). My data set has almost 24,000 rows. When I run glm in R, I get:  Warning messages:   1: glm.fit: algorithm did not converge   2: glm.fit: fitted probabilities numerically 0 or 1 occurred    I've read the other responses that suggest perfect separation might be occurring, but I'm confident that isn't the case in my data (though quasi-complete separation could exist; how can I test to see if that's the case?). If I remove some variables, the "did not converge" error might go away. But that's not always what happens.  I tried using the same variables in a bayesglm function and got the same errors.  What steps would you take to figure out exactly what's going on here? How do you figure out which variables are causing the problems?
20469
Basic Question  Is there an intuitive explanation of standard deviation in terms of Euclidean distance in $n$ dimensional space?  Longer Version of Question  To begin a more detailed sketch of my question, for simplicity let's just focus on the simple case of a discrete random variable that is uniformly distributed. In this case, the variance is given by the following formula, which I've abducted straight from Wikipedia:  $$ \frac1{n}\sum_{i =1}^n (x_i - \mu)^2$$  where $\mu$ is the mean. The standard deviation is then the square root of this. Now, I can't help noticing that the square root of the sum returns the euclidean distance from the vector $X = (x_1, x_2, \dots, x_n)$ to the vector $\vec \mu = (\mu, \mu, \dots, \mu)$. That is, the standard deviation can be expressed as:  $$ \frac1{\sqrt{n}}|X - \vec \mu |$$  So I wonder, is there any significant conceptual relationship between this distance $|X - \vec \mu |$ and standard devation or is this just a coincidence?  Even More Details...  I have looked up many explanations of standard deviation and its cousin variance. Here are some that I've seen already, each sort of following from the previous one:   We square the values before summing to get rid of the sign, which is obviously not important. This explanation is often criticised by hardcore statisticians and I can sort of see why: it doesn't explain why squaring beats taking the absolute value. We square the values so that we pay a greater price for greater deviations. This explains why squaring beats taking absolute values. But why not raise to the power of $4$, or $6$, or any other even power before summing? What is so special about $2$? The thing that is so special about $2$ is that it's the second moment of intertia, whereas the mean is the first moment, so mechanically it makes sense. I don't follow this. My intuition is totally OK with the mean: the point where, if I put my finger, the weights on either side will balance. But the second moment is harder for me to imagine physically like this.   Note, this is a question about intuition. I "understand" the mathematical formula at a shallow level: what all its terms mean, how to calculate it given a dataset. But I am not comfortable with my grasp on why this formula is "the best" one to use in so many applications e.g. the least squares method to fit data. I'm particularly confused as to why squaring is chosen as opposed to raising to some other even power e.g. $9234324$.   And this is where my intuition steps in and tries to provide an explanation that goes right back to the fundamental theorem of Pythagoras: euclidean distance. Here is my thought process: "The number $2$ is special. It's the unique power that makes Euclidean distance work. So maybe it's also the unique number that makes variance work." But then why the multiplying factor of $\frac1{\sqrt{n}}$? Is it just simply a case of: swallow it up and accept the definition, or can this intuition be resolved somehow?
25743
I performed with R an ANOVA and I got significant differences. However when checking which pairs were significantly different using the Tukey's procedure I did not get any of them. How can this be possible?  Here is the code:  fit5_snow&lt;- lm(Response ~ Stimulus, data=audio_snow) anova(fit5_snow)  &gt; anova(fit5_snow) Analysis of Variance Table  Response: Response           Df Sum Sq Mean Sq F value  Pr(&gt;F)   Stimulus   5  73.79 14.7578  2.6308 0.02929 * Residuals 84 471.20  5.6095                   --- Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1   df&lt;-df.residual(fit5_snow) MSerror&lt;-deviance(fit5_snow)/df  comparison &lt;-  HSD.test(audio_snow$Response, audio_snow$Stimulus, df, MSerror, group=FALSE)  &gt; comparison &lt;-  HSD.test(audio_snow$Response, audio_snow$Stimulus, df, MSerror, group=FALSE)  Study:  HSD Test for audio_snow$Response   Mean Square Error:  5.609524   audio_snow$Stimulus,  means                  audio_snow.Response   std.err replication snow_dry_leaves            4.933333 0.6208034          15 snow_gravel                6.866667 0.5679258          15 snow_metal                 6.333333 0.5662463          15 snow_sand                  6.733333 0.5114561          15 snow_snow                  7.333333 0.5989409          15 snow_wood                  5.066667 0.7713110          15  alpha: 0.05 ; Df Error: 84  Critical Value of Studentized Range: 4.124617   Comparison between treatments means                                Difference   pvalue sig        LCL      UCL snow_gravel - snow_dry_leaves  1.9333333 0.232848     -0.5889913 4.455658 snow_metal - snow_dry_leaves   1.4000000 0.588616     -1.1223246 3.922325 snow_sand - snow_dry_leaves    1.8000000 0.307012     -0.7223246 4.322325 snow_snow - snow_dry_leaves    2.4000000 0.071587   . -0.1223246 4.922325 snow_wood - snow_dry_leaves    0.1333333 0.999987     -2.3889913 2.655658 snow_gravel - snow_metal       0.5333333 0.989528     -1.9889913 3.055658 snow_gravel - snow_sand        0.1333333 0.999987     -2.3889913 2.655658 snow_snow - snow_gravel        0.4666667 0.994348     -2.0556579 2.988991 snow_gravel - snow_wood        1.8000000 0.307012     -0.7223246 4.322325 snow_sand - snow_metal         0.4000000 0.997266     -2.1223246 2.922325 snow_snow - snow_metal         1.0000000 0.855987     -1.5223246 3.522325 snow_metal - snow_wood         1.2666667 0.687424     -1.2556579 3.788991 snow_snow - snow_sand          0.6000000 0.982179     -1.9223246 3.122325 snow_sand - snow_wood          1.6666667 0.393171     -0.8556579 4.188991 snow_snow - snow_wood          2.2666667 0.103505     -0.2556579 4.788991
88493
I am using hierarchical clustering to analyze time series data. My code is implemented using the Mathematica function DirectAgglomerate[...], which generates hierarchical clusters given the following inputs:   a distance matrix D the name of the method used to determine inter-cluster linkage.   I have calculated the distance matrix D using Manhattan distance:  $$d(x,y) = \sum_i|x_i - y_i|$$  where $i = 1,\cdots, n$ and $n \approx 150$ is the number of data points in my time series.  My question is, is it ok to use Ward's inter-cluster linkage with a Manhattan distance matrix?  Some sources suggest that Ward's linkage should only be used with Euclidean distance.  Note that DirectAgglomerate[...] calculates Ward's linkage using the distance matrix only, not the original observations.  Unfortunately, I am unsure how Mathematica modifies Ward's original algorithm, which (from my understanding) worked by minimizing the error sum of squares of the observations, calculated with respect to the cluster mean.  For example, for a cluster $c$ consisting of a vector of univariate observations, Ward formulated the error sum of squares as:  $$(\sum_j||c_j - mean(c)||_2)^2$$  (Other software tools such as Matlab and R also implement Ward's clustering using just a distance matrix so the question isn't specific to Mathematica.)
113630
We have a series of observations of different properties (such as heart rate or blood sugar level and others as well) across different days from different people from different geographical regions. The number of days on which the observations are made is the same for all regions; but there could be different number of participants in each region. The result of the data collection will be as follows: DATA We have a matrix, $M_{i,j}$, for each region $i$ and property $j$, with $T$ observations across time (rows) and $N_{i,j}$ (columns) for number of participants from region $i$ and property $j$. Here, we consider all properties together as representing a region. OBJECTIVE The goal is to identify people from which regions have a more similar constitution. QUESTION How can we compute the Bhattacharya Distance across the various matrices $M_{i,j}$ for all the properties combined for a region (corresponding to Case 2 of the data collection)? Dimension of Matrix: $Dim(M_{i,j})=(T,N_{i,j})$ There are two cases here. 1) $N_{i,j}=N_{i,k}$ $,\forall j,k$ That is the number of participants in a region are the same for all properties being measured. 2) $N_{i,j} \neq N_{i,k}$ $,\forall j,k$ That is the number of participants in a region could be different for some of the properties being measured. But if helpful we can assume the simple case 1) $N_{i,j}=N_{i,k}$. Please let me know if anything is not clear and if you need any further information. Related Case When we have a matrix, $M_i$, for each region $i$, with $T$ observations across time (rows) and $N_i$ (columns) for number of participants from that region. Here, we consider each property separately. How can we compute the Bhattacharya Distance across the various matrices $M_i$ (in pairs or combined, whichever is simpler) separately for each property (corresponding to Case 1 of the data collection)? Answer:
94380
I've compared the logistic regression models on R (glm) and on Spark (LogisticRegressionWithLBFGS) on a dataset of 390 obs. of 14 variables.  The results are completely different in the intercept and the weights. How to explain this?  Here is the results of Spark (LogisticRegressionWithLBFGS) :  model.intercept  :   1.119830027739959 model.weights :  GEST    0.30798496002530473  DILATE  0.28121771009716895  EFFACE  0.01780105068588628  CONSIS -0.22782058111362183  CONTR  -0.8094592237248102  MEMBRAN-1.788173534959893  AGE    -0.05285751197750732  STRAT  -1.6650305527536942  GRAVID  0.38324952943210994  PARIT  -0.9463956993328745  DIAB   0.18151162744507293  TRANSF -0.7413500749909346  GEMEL  1.5953124037323745   Here is the result of R :               Estimate Std. Error z value Pr(&gt;|z|)    (Intercept)  3.0682091  3.3944407   0.904 0.366052     GEST         0.0086545  0.1494487   0.058 0.953821     DILATE       0.4898586  0.2049361   2.390 0.016835 *   EFFACE       0.0131834  0.0059331   2.222 0.026283 *   CONSIS       0.1598426  0.2332670   0.685 0.493196     CONTR        0.0008504  0.5788959   0.001 0.998828     MEMBRAN     -1.5497870  0.4215416  -3.676 0.000236 ***    AGE         -0.0420145  0.0326184  -1.288 0.197725     STRAT       -0.3781365  0.5860476  -0.645 0.518777     GRAVID       0.1866430  0.1522925   1.226 0.220366     PARIT       -0.6493312  0.2357530  -2.754 0.005882 **   DIAB         0.0335458  0.2163165   0.155 0.876760     TRANSF      -0.6239330  0.3396592  -1.837 0.066219 .   GEMEL        2.2767331  1.0995245   2.071 0.038391 *   Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
110976
I have a question regarding the interpretation of the whiskers of a boxplot. I have read the following: "On the top and bottom of the rectangle, the “whiskers” show the range of 1.5 times the distance between the 0.25- and 0.75-quantiles", but do not entirely understand what is meant by "distance".  It can't be that the probability mass is meant, since between the 0.25 and 0.75 quantile we obviously always have the same percentage of data. What then is the idea?
16711
The help pages in R assume I know what those numbers mean, but I don't. I'm trying to really intuitively understand every number here. I will just post the output and comment on what I found out. There might (will) be mistakes, as I'll just write what I assume. Mainly I'd like to know what the t-value in the coefficients mean, and why they print the residual standard error.   Call: lm(formula = iris$Sepal.Width ~ iris$Petal.Width)  Residuals:      Min       1Q   Median       3Q      Max  -1.09907 -0.23626 -0.01064  0.23345  1.17532    This is a 5-point-summary of the residuals (their mean is always 0, right?). The numbers can be used (I'm guessing here) to quickly see if there are any big outliers. Also you can already see it here if the residuals are far from normally distributed (they should be normally distributed).  Coefficients:                  Estimate Std. Error t value Pr(&gt;|t|)     (Intercept)       3.30843    0.06210  53.278  &lt; 2e-16 *** iris$Petal.Width -0.20936    0.04374  -4.786 4.07e-06 *** --- Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1    Estimates $\hat{\beta_i}$, computed by least squares regression. Also, the standard error is $\sigma_{\beta_i}$. I'd like to know how this is calculated.  I have no idea where the t-value and the corresponding p-value come from. I know $\hat{\beta}$ should be normal distributed, but how is the t-value calculated?  Residual standard error: 0.407 on 148 degrees of freedom   $\sqrt{ \frac{1}{n-p} \epsilon^T\epsilon }$, I guess. But why do we calculate that, and what does it tell us?  Multiple R-squared: 0.134,  Adjusted R-squared: 0.1282    $ R^2 = \frac{s_\hat{y}^2}{s_y^2} $, which is $ \frac{\sum_{i=1}^n (\hat{y_i}-\bar{y})^2}{\sum_{i=1}^n (y_i-\bar{y})^2} $. The ratio is close to 1 if the points lie on a straight line, and 0 if they are random.  What is the adjusted R-squared?  F-statistic: 22.91 on 1 and 148 DF,  p-value: 4.073e-06    F and p for the whole model, not only for single $\beta_i$s as previous. The F value is $ \frac{s^2_{\hat{y}}}{\sum\epsilon_i} $. The bigger it grows, the more unlikely it is that the $\beta$'s do not have any effect at all.
63817
I have a series of points representing animal observations. Many of them overlap, having the exact same coordinates, and this is causing me trouble when I want to run some analytical tools which requite no identical points. I was wondering if there's any tool that can add some "noise" to the coordinates, that's to say, adding some random decimal values at the end in order to obtain slightly different coordinates which are still close enough in space so as to not really modify the observation. I've heard about such a tool, but I can't find it, at least in ArcGis 10. Maybe there's another software that can do it? Thanks!
105675
I have 320 data points - each has a redshift and a turnover-frequency, and I want to fit a correlation between them (a linear fit). However, 120 of the turnover-frequency values are upper limits. As shown below the relationship is very weak:   the red points have upper limits for their turnover-frequency value. I would like to fit a straight line to this data that accounts for the upper limits. I think the tobit model in R can do this, but I'm not too sure (I'm new to R by the way); before I start with that, are there other methods in R I should consider?
90415
Say I generate a dataset $X$: the first $i$ samples follow $x_i\sim N(\mu_1,\Sigma_1^2)$, the next j samples follow $N(\mu_2,\Sigma_2^2)$ and the last $l$ samples from $N(\mu_3,\Sigma_3^2)$. Naturally, I denote the clusters $(1,2,3)$ by the order of creation.  When using kmeans(X,3)$clusters I get the predicted assignment of the samples, but here lies the problem: kmeans might enumerate the clusters differently, for example set $N(\mu_2,\Sigma_2^2)$ as cluster#3 and vice versa. In this situation, I cannot compare the 'original' cluster assignment (say $(1,1,2,2,2,3,3,3,3,3)$) with the predicted one (say $(1,1,3,3,3,2,2,2,2,2)$). this will result with only 20% accuracy, even though the actual accuracy is 100%.  Any ideas of how to map the centroids or measure the accuracy of the kmeans performance?
114836
Context  I have two sets of data that I want to compare. Each data element in both sets is a vector containing 22 angles (all between $-\pi$ and $\pi$). The angles relate to a given human pose configuration, so a pose is defined by 22 joint angles.  What I am ultimately trying to do is determine the "closeness" of the two sets of data. So for each pose (22D vector) in one set, I want to find its nearest neighbour in the other set, and create a distance plot for each of the closest pairs.  Questions   Can I simply use Euclidean distance?   To be meaningful, I assume that the distance metric would need to be defined as: $\theta = |\theta_1 - \theta_2| \quad mod \quad \pi$, where $|...|$ is absolute value and mod is modulo. Then using the resulting 22 thetas, I can perform the standard Euclidean distance calculation, $\sqrt{t_1^2 + t_2^2 + \ldots + t_{22}^2}$. Is this correct?  Would another distance metric be more useful, such as chi-square, or Bhattacharyya, or some other metric? If so, could you please provide some insight as to why.