Pardon me if this question does not fit this site (or SE engine in general) but I do not know where to post it. Suppose that you have unknown data set which you want to explore. What functions/methods/techniques should I use to get birds eye of the data? I guess it depends on the data. I'm particularly interests in the ordered integers (but feel free to post other types also). Few examples that comes to my mind are: histograms, barplots, (moving) mean (aka average) median mode variance standard-deviation. What else? Thank you.
122222
I've been reading Tukey's book "Exploratory Data Analysis".  Being written in 1977, the book emphasizes paper/pencil methods.  Is there a more 'modern' successor which takes into account that we can now instantaneosly plot large data sets?
101933
I've looked through a lot of literature available online, including this forum without any luck and hoping someone can help a statistical issue I currently face:  I have 5 lists of of ranked data, each containing 10 items ranked from position 1 (best) to position 10 (worst). For sake of context, the 10 items in each lists are the same, but in different ranked orders as the technique used to decide their rank is different.  Example data:              List 1      List 2      List 3     ... etc Item 1     Ranked 1    Ranked 2    Ranked 1      Item 2     Ranked 3    Ranked 1    Ranked 2 Item 3     Ranked 2    Ranked 3    Ranked 3 ... etc   I am looking for a way to interpret and analyse the above data so that I get a final result showing the overall rank of each item based on each test and its position, e.g.  Result Rank 1 = Item 1 Rank 2 = Item 3 Rank 3 = Item 4 ... etc   So far I have attempted to interpret this information from performing Pearson's Correlation, Spearman's Correlation, Kendall Tau's B, and Friedman tests. I have found however, that these results have generally paired my lists (i.e. compared list 1 to list 2, then list 1 to list 3 .. etc), or have produced results such as Chi-Square, P-Values etc about the overall data.   Does anyone know how I can interpret this data in a statistically sound method (at a post graduate / PhD applicable level) so that I can understand the overall ranks signalling the importance of each item in the list across the 5 tests please? Or, if there is another type of technique or statistical test I can look into I would appreciate any hints or guidance.   (It maybe also worth noting, I have also performed the simpler mathematical techniques such as sums, averaging, minimum - maximum tests etc, but do not feel these are statistically important enough at this level).  Any help or advice would be greatly appreciated, thank you for your time.
15959
Equivalently, about variance?  I realize it measures the spread of a distribution, but many other metrics could do the same (e.g., the average absolute deviation). What is its deeper significance? Does it have    a particular geometric interpretation (in the sense, e.g., that the mean is the balancing point of a distribution)?  any other intuitive interpretation that differentiates it from other possible measures of spread?   What's so special about it that makes it act as a normalizing factor in all sorts of situations (for example, convert covariance to correlation)?
95335
We're showing a pie graph that would show something like 53% male, 41% female and 7% unknown.  The "unknown" category includes many reasons we can't tell: we don't know, we can't know for legal reasons, we can't know for self-imposed privacy policy reasons, our data sources don't know, etc. The reasons are quite innumerable.  This seems like a common problem so I wanted to ask if there is a common solution?
13641
I am a grad student developing an interest for statistics. I like the material over-all, but I sometimes have a hard time thinking about applications to real life. Specifically, my question is about commonly used statistical distributions (normal - beta- gamma etc.). I guess for some cases I get the particular properties that make the distribution quite nice - memoryless property of exponential for example. But for many other cases, I don't have an intuition about both the importance and application areas of the common distributions that we see in textbooks.   There are probably a lot of good sources addressing my concerns, I would be glad if you could share those. I would be a lot more motivated into the material if I could associate it with real-life examples.
25909
I'm embarrassed to say that I have a PhD and hold an asst professorship, but get tripped up when reading statistics research. I am in a field of Business that is similar to IO Psychology or Social Psych. I spend too much time reading applied stats books, but I find even with all the reading I don't have a firm grasp of what I'm actually doing. Everything is very 'seat of the pants.' (As sad as it seems, I think this is not a unique situation among the faculty in the social sciences...) The biggest problem comes when I need to apply a rarely used stat technique. I can find an article from a mathematical stats journal with the equations that would solve my problem, but I don't have the math to convert those into code. I am forever relying on other prof's R packages, and crossing my fingers hoping it will work (I can't even check to verify if it did or not). It's been over 15 years since I took Calculus and Algebra in undergrad, and I think I want to start at the beginning and truly understand probability and statistics.   I am starting with  and  for a quick refresher of the basics -- I know it's hard to believe, but in an applied research field we rarely have use for sin or cos. I'm even trying to finally learn how to correctly do a proof, using the books from Velleman ("") and Houston ("") -- I'm serious about doing this right and understanding the subject. From there I want to move on to (correctly) learn the Calculus and Linear Algebra I need to tackle probability and statistics. I was thinking of using  and  books. But  comes highly recommended as well. After that I am completely at a loss. Further, I don't know how far to go into Calculus or Linear Algebra before I reach diminishing returns. (Apostle introduces Probability in the  -- is it vital that I work through everything preceding it before tackling Probability?)  So my question is: if you had to do it over again with the goal of truly, deeply understanding statistics, where would you start? What books are the modern path to deep understanding? I would like to follow a modern path so that I can understand current research in statistics, including Bayesian approaches. But not in a machine learning context (which seems to be the all the rage at the moment), rather a social science / design and analysis of experiments / multilevel modeling context. Perhaps my goal would be the work of Andrew Gelman;  showed me how I should be looking at modeling and statistics (simulation, uncertainty estimates everywhere, bayesian inference, and so on).  How should I go about relearning this material with that end goal in mind?    Update 1: Possible texts, starting from scratch with a focus on proofs and deep understanding. Not necessarily one after another.  Relearn the basics:    by Velleman  by Houston  and  by Gelfand (for understanding why and how instead of what)  by Simmons (for reference)  by Lockhart (for inspiration)   Calculus (which one(s), and how deep?):    by Strang  and  by Apostol   by Spivak ()   by Curant (and II/1 II/2?)   Linear Algebra (which one(s) and how deep?):    by Strang  by Searle  by Gentle   Probability (which one(s)?):    and  by Feller  (for intuitive understanding)  by Hoel, Port, Stone  by Resnick (for measure theoretic / modern approach?)  by Mosteller   Core Statistics (which one(s)?):    by DeGroot and Schervish  Casella and Berger   Other suggestions? Again with the goal of understanding and developing (or at least implementing) new methods in hierarchical modelling (generalized and linear).
84850
I am working on creating a website, which displays the census data for a user selected Polygons &amp; would like to graphically show the distribution of various parameters (one graph per parameter).  The data usually has the following properties:   The sample size tend to be large (say around 10,000 data points) The range in values tends to be quire large (for example, the minimum population can be less than 100 &amp; the maximum can be something like 500,000) q1 usually is close to the minimum (say 200) while q2 &amp; q3 will be within 10,000 It doesn't look anything like a normal distribution   I am not a statistician and hence my description might not be exactly clear.  I would like to show this distribution on a graph, which will be seen by citizens (the layman, if you like).  I would have best liked to use a histogram, but it is not possible due to the large range of values, due to which making bins is not really easy &amp; straight forward.  From what little I know about statistics,  a box plot is what is often used to show this kind of data, but I feel that for a layperson, deciphering the Box plot is not easy.  What are my options to show this data in an easy to understand manner?
9473
I have a large set of data points of the form (mean, stdev). I wish to reduce this to a single (better) mean, and a (hopefully) smaller standard deviation.  Clearly I could simply compute $\frac{\sum data_{mean}}{N}$, however this does not take in to account the fact that some of the data points are significantly more accurate than others.  To put it simply, I wish to preform a weighted average of these data points, but do not know what the weighting function should be in terms of the standard deviation.
101532
I have found plenty of information on what STATISTICS are: how they are maintained, how they can be created manually or automatically from queries or indexes, and so on. But, I have been unable to find any guidance or "best practices" information regarding when to create them: what situations benefit more from a manually created STATISTICS object than from an Index. I have seen manually created filtered statistics helping queries on partitioned tables (because the statistics created for the indexes cover the whole table and are not per partition--brillaint!), but surely there must be other scenarios that would benefit from a statistics object while not needing the detail of an index, nor worth the cost of maintaining the index or increasing the chances of blocking / dead-locks.  @JonathanFite, in a comment, mentioned a distinction between indexes and statistics:     Indexes will help SQL find the data faster by creating lookups that are sorted differently than the table itself. Statistics help SQL determine how much memory/effort is going to be required to satisfy the query.   That is great info, mostly because it helps me clarify my question:  How does knowing this (or any other technical info on the whats and hows related to the behaviors and nature of STATISTICS) help determine when to choose CREATE STATISTICS over CREATE INDEX, especially when creating an Index will create the related STATISTICS object? What scenario would be better served by having only the STATISTICS info and not having the Index?  It would be super-duper helpful, if possible, to have a working example of a scenario where the STATISTICS object is a better fit than an INDEX.    Since I am a visual learner / thinker, I thought it might help to see the differences between STATISTICS and INDEXes, side-by-side, as a possible means of helping determine when STATISTICS are the better choice.  Thingy           PROs                             CONs -------          ----------                       ------------------- INDEX            * Can help sorts.                * Takes up space.                  * Contains data (can             * Needs to be maintained (extra I/O).                    "cover" a query).              * More chances for blocking / dead-locks.  STATISTICS       * Takes up very little space.    * Cannot help sorts.                  * Lighter maintenance / won't    * Cannot "cover" queries.                    slow down DML operations.                  * Does not increase chances                    of blocking / dead-locks.     The following are some resources that I found while looking for this, one that even asks this same question, but it was not answered:        ** To be clear, I do not have an answer for this and am actually looking to get feedback from hopefully a few people to provide what appears to be oddly missing information out here in the interwebs.
15149
I am preparing a paper in the field of Computer Science.  In order to report test results, we usually run a number of tests and report the average of those tests.  For each test, we generate random data.  Because of the randomness, at some points, the results may come out not as expected.  For instance, a graph may be like:   Usually, one should explain why on points 8, 11 and 12 there is a decrease on the plot. Probably, it is because of that randomness.  Not hand-crafting all the graph, but just manipulating a few points makes the graph acceptable:   Since three weeks or so, I work my ass off and try to figure out why my resulting graph looks like the first one. Sometimes I feel like yielding to temptation and just modify the raw data before I go crazy.  I believe, at this point the title became misleading, so let me make it clear:  I am not seeking an advice on data manipulation. I will not manipulate my data. However, I ask to myself "how the hell this can be detected?"  And now, I don't only ask to myself, but to whole community. How is this detected? For editors, referees out there, have you ever detected something like this?
24143
An investigator wishes to produce a combined analysis of several datasets.  In some datasets there are paired observations for treatment A and B.  In others there are unpaired A and/or B data.  I am looking for a reference for an adaptation of the t-test, or for a likelihood ratio test, for such partially paired data.  I am willing to (for now) to assume normality with equal variance and that the population means for A are the same for each study (and likewise for B).
6453
I have a dataset of sample observations, stored as counts within range bins. e.g.:  min/max  count 40/44    1 45/49    2 50/54    3 55/59    4 70/74    1   Now, finding an estimate of the average from this is pretty straight forward.  Simply use the mean (or median) of each range bin as the observation and the count as a weight and find the weighted average:  $$\bar{x}^* = \frac{1}{\sum_{i=1}^N w_i} \sum_{i=1}^N w_ix_i$$  For my test case, this gives me 53.82.  My question now is, what's the correct method of finding the standard deviation (or variance)?  Through my searching, I've found several answers, but I'm unsure which, if any, is actually appropriate for my dataset.  I was able to find the following formula both  and .  $$s^{2*} = \frac{ \sum_{i=1}^N w_i (x_i - \bar{x}^*)^2 }{ \frac{(M-1)}{M} \sum_{i=1}^N w_i }$$  Which gives a standard deviation of 8.35 for my test case.  However, the Wikipedia article on  gives both the formula:  $$s^{2*} = \frac{ \sum_{i=1}^N w_i}{(\sum_{i=1}^N w_i)^2 - \sum_{i=1}^N w_i^2} \sum_{i=1}^N w_i(x_i-\bar{x}^*)^2$$  and  $$s^{2*} = \frac{1}{(\sum_{i=1}^N w_i) - 1} \sum_{i=1}^N w_i(x_i-\bar{x}^*)^2$$  Which give standard deviations of 8.66 and 7.83, respectively, for my test case.  Update  Thanks to @whuber who suggested looking into Sheppard's Corrections, and your helpful comments related to them.  Unfortunately, I'm having a difficult time understanding the resources I can find about it (and I can't find any good examples).  To recap though, I understand that the following is a biased estimate of variance:  $$s^{2*} = \frac{1}{\sum_{i=1}^N w_i} \sum_{i=1}^N w_i(x_i-\bar{x}^*)^2$$  I also understand that most standard corrections for the bias are for direct random samples of a normal distribution.  Therefore, I see two potential issues for me:   These are binned random samples (which, I'm pretty sure, is where Sheppard's Corrections come in.)  It's unknown whether or not the data is for a normal distribution (thus I'm assuming not, which, I'm pretty sure, invalidates Sheppard's Corrections.)   So, my updated question is; What's the appropriate method for handling the bias imposed by the "simple" weighted standard deviation/variance formula on a non-normal distribution? Most specifically with regards to binned data.  Note: I'm using the following terms:     $s^{2*}$ is the weighted variance   $N$ is the number of observations.  (i.e. the number of bins) $M$ is the number of nonzero weights.  (i.e. the number of bins with counts) $w_i$ are the weights  (i.e. the counts) $x_i$ are the observations.  (i.e. the bin means) $\bar{x}^*$ is the weighted mean.
46273
I am trying to work on the data of my master thesis. I have to create a model, but I don't know which analysis perform. My data look like this     number ID sp ab alt SWdiv     1 A1 4 6 630 0.25802965   2 A2 9 16 750 0.29235324   3 A3 8 14 850 0.411620203   4 A4 9 22 950 0.444079553   5 A5 2 12 1050 0.294855765   6 A6 13 40 1130 0.298451014   7 B1 2 2 750 0.275329152   8 B2 5 6 850 0.043352118   9 B3 14 16 950 0   10 B4 17 58 1050 0.230358688   11 B5 7 9 1150 0.298759725   12 B6 8 68 1250 0.012192474   13 C1 12 20 830 0.467542842   14 C2 13 19 930 0.454480722   15 C3 6 30 1030 0.464584038   16 C4 7 14 1130 0.448191833   17 C5 5 15 1250 0.052045275   18 C6 2 7 1315 0.051767368   19 D1 3 3 650 0.294855541   20 D2 11 17 750 0.291717599   21 D3 9 46 850 0.084284864   22 D4 8 57 950 0.088574421   23 D5 4 32 1050 0.160873283   24 D6 13 22 1150 0.297638962   25 D7 6 22 1250 0.290733158   26 E1 11 23 800 0.250878932   27 E2 8 8 900 0.418021272   28 E3 10 13 1000 0.132159303   29 E4 5 10 1100 0.058487554   30 E5 5 15 1200 0.250310822   31 E6 10 19 1350 0.01600807   32 F1 6 20 800 0.462497319   33 F2 15 54 900 0.376552904   34 F3 6 13 1000 0.298264791   35 F4 6 13 1100 0.394736812   36 G1 14 22 830 0.391455956   37 G2 11 16 930 0.282922551   38 G3 5 10 1030 0.282398234   39 G4 10 15 1130 0.153502726   40 G5 13 18 1230 0.051586706   41 H1 10 11 700 0.3009609   42 H2 7 14 800 0.579228102   43 H3 5 11 900 0.528108703   44 H4 20 48 1000 0.28689587   45 H5 13 18 1100 0.264542772   46 H6 10 20 1250 0.29928462   47 I1 12 13 535 0.35326876   48 I2 3 5 635 0.273947657   49 I3 18 30 720 0.467208221   50 I4 16 188 835 0.276669711   51 I5 7 69 935 0.000867067   52 L1 6 10 435 0.36910895   53 L2 7 20 606 0.236700524   54 L3 14 33 697 0.111132519   55 L4 17 42 797 0.296757045   56 L5 5 7 910 0.27539552   57 L6 10 15 1000 0.29748451   58 L7 6 11 1101 0.276755556     58 sample points each has two value: altitude and diversity index related to habitat heterogeneity. What I sample are different insects and I have two values: sp = species (total number of different species for each sample point) and ab = abundance (total number of insects for each sample point). My question are are the species richness or the abundance affected by the altitude or the habitat heterogeneity? which is the best analysis to perform? and why? thank you for your help.
55443
I've heard/seen in several places that you can transform the data set into something that is normal-distributed by taking the logarithm of each sample, calculate the confidence interval for the transformed data, and transform the confidence interval back using the inverse operation (e.g. raise 10 to the power of the lower and upper bounds, respectively, for $\log_{10}$).  However, I'm a bit suspicious of this method, simply because it doesn't work for the mean itself: $10^{\operatorname{mean}(\log_{10}(X))} \ne \operatorname{mean}(X)$  What is the correct way to do this? If it doesn't work for the mean itself, how can it possibly work for the confidence interval for the mean?
85678
I've been wondering for a while now if there's any deep mathematical or statistical significance to finding the line that minimizes the square of the errors between the line and the data points.   If we use a less common method like LAD, where we just consider the absolute deviation, then outliers make less difference to the final model, while if we take the cube of the error (or any other power higher than 2), then outliers are far more significant than with the least squares model.  I suppose what I'm really asking is mathematically, is raising the error to the power of 2 really that special. Is it say more "accurate" in some sense than raising the error to the power of 1.95 or 2.05???   Thanks!
76044
One thing that has always tripped me up when trying to learn new methods in statistics is understanding what type of features/variables can this method be applied to.  The variable types that especially trip me up are   Discrete Continuous Categorical Nominal Ordinal   What are some examples and simple explanations of these and other commonly used variables types.
39943
I was wondering if there were any machine learning techniques (unsupervised) for modelling longitudinal data? I've always used mixed effects models (mostly non-linear) but I was wondering if there are any other ways of doing this (using machine learning).  By machine learning, I mean random forest, classification/clustering, decision trees and even deep learning, etc.