I am speeking about  graph, that shows a big probability mass.  This is the way that it finally shows with some data I found online and used to understand Latent Dirichlet Allocation topic modeling. As far as I understand, this graphic is ideal to give out final names to each topic according to the frequency or relevance of each term given the topic but I've got two questions: 1. What would you simply say about topic 8 position on the graph? Meaning that I can't tell what each square represent in this graph, I can't tell in plain words what's the difference between topic 1 and 10, or topic 8 and 3, given their position on this graph. 2. What can you tell about the big gray line in comparisson to the red filling when lambda is 0? I mean, I think I get that when lambda is 1, it's meaning about plain frequency of the term in the whole corpus vs the frequency of the terms in that topic, but I'm not sure I get it with putting more wight in the ratio from red to gray, what is the gray line saying about that term and how is red contransting?
121798
I write about  that shows a big probability mass. It's a web-based interactive visualization of topics estimated using Latent Dirichlet Allocation that is built using a combination of R and D3; the default for scaling the set of inter-topic distances defaults to Principal Components, but other algorithms are also enabled.    This is the way that it finally shows with some data I found online and used to understand Latent Dirichlet Allocation topic modeling. As far as I understand, this graph is ideal to give out final names to each topic according to the frequency or relevance of each term given the topic but I've got two questions:   1. What would you simply say about topic 8 position on the graph? Meaning that I can't tell what each square represents in this graph, I can't tell in plain words what's the difference between topic 1 and 10, or topic 8 and 3, given their position on this graph?   2. What can you say about the big gray line in comparison with the red filling when lambda is 0? I think I get that when lambda is 1, it's meaning about plain frequency of the term in the whole corpus vs the frequency of the terms in that topic, but I'm not sure I get it with putting more weight in the ratio from red to gray: what is the gray line saying about that term and how is red contrasting?  You can get this for a model named lda_model$plot() found .   I am looking for the main idea at least. If there is info I need to add please tell me.
73780
I'm currently starting a course about Graph Theory, and I've been explained about the Havel &amp; Hakimi degree sequence. However, I'm not too sure about it yet. First of all, I want to know if I understand it correctly, so I'm going to try to explain what I think the theory proves.  First, I've shamelessly stolen the following statement from the presentation:    Okay, so what I want to ask is the following.   What does "is graphic" mean exactly. Does that mean it's a graph? I know stupid question, but I cannot think of a degree sequence that's not a graph... What's the practical use of this. What problems could you solve with this.
49908
I've been playing around with random forests for regression and am having difficulty working out exactly what the two measures of importance mean, and how they should be interpreted.  The importance() function gives two values for each variable: %IncMSE and IncNodePurity. Is there simple interpretations for these 2 values?  For IncNodePurity in particular, is this simply the amount the RSS increase following the removal of that variable?
45190
I'm fairly new to Bayesian statistics and I came across a corrected correlation measure, , that uses the Dirichlet process in the backend of it's algorithm.  I have been trying to go through the algorithm step-by-step to really understand what is happening but I am not sure exactly what the alpha vector parameter does in a Dirichlet distribution and how it normalizes the alpha vector parameter?   The implementation is in Python using NumPy:   The docs say:     alpha : array        Parameter of the distribution (k dimension for sample of        dimension k).   My questions:   How do the alphas affect the distribution?; How are the alphas being normalized?; and  What happens when the alphas are not integers?   import numpy as np import pandas as pd import matplotlib.pyplot as plt  # Reproducibility np.random.seed(0)  # Integer values for alphas alphas = np.arange(10) # array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])  # Dirichlet Distribution dd = np.random.dirichlet(alphas)  # array([ 0.        ,  0.0175113 ,  0.00224837,  0.1041491 ,  0.1264133 , #         0.06936311,  0.13086698,  0.15698674,  0.13608845,  0.25637266])  # Plot ax = pd.Series(dd).plot() ax.set_xlabel("alpha") ax.set_ylabel("Dirichlet Draw")
82062
The ultimate goal is to show users, at a glance, if their data is normally distributed.    The first attempt is a kludge that plots the data in a frequency graph.  Then, the observed mean and standard deviation are used to build a "normal curve" graph.  The frequency chart is laid over the the normal-curve chart and put next to some key statistics.  The frequency chart also colors positive bins green and negative bins red.  It looks like this:   I understand the fallacy of this approach, but for now it's practical.   What is a better way to approach this issue?
23824
I have fish density data that I am trying to compare between several different collection techniques, the data has lots of zeros, and the histogram looks vaugley appropriate for a poisson distribution except that, as densities, it is not integer data. I am relatively new to GLMs and have spent the last several days looking online for how to tell which distribution to use but have failed utterly in finding any resources that help make this decision. A sample histogram of the data looks like the following:   I have no idea how to go about deciding on the appropriate family to use for the GLM. If anyone has any advice or could give me a resource I should check out, that would be fantastic.
88502
Question  Suppose we have a multinomial distribution with $N$ possible outcomes, with probabilities $p_1,\ldots,p_N$. We sample this $n$ times, and denote the observed frequency of the $i$th outcome as $\xi_i$. In [1] the author claims that the distribution of the $\xi_i$ in the limit of large $n$ is:  $$p(\xi_1,\ldots,\xi_N)\propto\exp\left(-\frac{n}{2}\sum_{i=1}^N\frac{(\xi_i-p_i)^2}{p_i}\right).\;\;\;\;\;(1)$$  We can see immediately that this must be an approximation, as it assigns nonzero probabilities for $\xi_1+\cdots+\xi_N&gt;1$. However we can see that these have vanishing probability in the limit $n\rightarrow\infty$. My question is how do we derive (1) from the multinomial distribution, and show that they match in the $n\rightarrow\infty$ limit?  My thoughts  My first thought would be to appeal to the central limit theorem. The multinomial distribution has mean $\mu_i=p_i$ and covariance matrix $\Sigma_{ij}=\delta_{ij}p_i-p_ip_j$, so we would expect this in the large $n$ limit to be described by a multivariate Gaussian with mean $\mu$ and covariance $\frac{1}{n}\Sigma$. However, things are complicated by the fact that the multinomial covariance is singular (since $\xi_N$ is determined by the other $\xi_i$s), and so the multivariate Gaussian is not defined.  To address this, we may try and consider only the first $\xi_1,\ldots,\xi_{N-1}$, which have a non-singular covariance matrix and hence well-defined multivariate Gaussian distribution. Let's take the Binomial distribution $N=2$. The frequency $\xi_1$, this has mean $p_1$ and variance $p_1(1-p_1)$, so this would be described the the Gaussian: $$\propto\exp\left(-\frac{n}{2}\frac{(\xi_1-p_1)^2}{p_1(1-p_1)}\right).\;\;\;\;\;(2)$$ The expression (1) gives: $$\propto\exp\left(-\frac{n}{2}\left(\frac{(\xi_1-p_1)^2}{p_1}+\frac{(\xi_2-p_2)^2}{p_2}\right)\right).\;\;\;\;\;(3)$$ If we substitute $\xi_2\rightarrow 1-\xi_1$, $p_2\rightarrow 1-p_1$ into (3), we can verify that this gives the same answer as (2). I have verified that this also works for $N=4$.   I'm sure that if I just bashed out the algebra for general $N$ we would get agreement between the central limit theorem and (1) when we restrict the latter to $\xi_1+\cdots+\xi_N=1,p_1+\cdots+p_N=1$. However, how can we start with the multinomial distribution and derive (1) as a limit which is valid everywhere? One idea would be to say that (1) goes to zero as $n\rightarrow\infty$ when you are not on that plane, however I am a bit uncomfortable with this as it goes to zero everywhere except the mean as $n\rightarrow\infty$, so I don't know if that argument is good enough.  [1] Wootters, William K. "Statistical distance and Hilbert space." Physical Review D 23.2 (1981): 357.
26246
I'm having difficulty wrapping my head around scale parameters.  How exactly do they work?  Why are they sometimes ignored?  (in other words, when is it important to preserve them in a calculation?)   Thank you for any simple, lay explanations.     UPDATE: What motivated the question was the Gamma distribution; in reading of it with relation to Poisson priors, I saw it described as simply $\propto  x^{\alpha-1}  e ^{-x /\beta }$    However, the question is more of a general nature.  While I get the concept of proportionality, I do not yet fully grasp when it is important for items to be equal versus simply proportional; or rather why so often in the literature I see items referred to as proportional with the scaling factor dropped.
90861
This may be too much of a general question but I hope I can find help here. I am starting a RA job in my university and my topic will be related to Internet Traffic Analysis. I am fairly new to the world of analysis but I guess in the world of research this is what I have to do a lot of.  I've been through a few papers and in a lot of them I find they use Probability Density (PDF), CDF, CCDF etc to explain the results they've obtained. For example, PDF of user session duration, CDF of bytes transferred each day etc. I have taken probability and statistics class so I understand what they are but I am still confused to the cases in which such representation would be chosen.  So, if there is anyone out there who does such graphs and analysis (in any other topic in general or this) could you tell me simply in what situation would I use one or the other of these representations
72308
I've been reading over this , trying to make sense of how the mean and variance of a gaussian conditional was derived. I've come to accept that unless I allocate a dozen or so hours to refreshing my linear algebra knowledge, it's out of my reach for the time being. that being said, I'm looking for a conceptual explanation for that these equations represent: $$\mu_{1|2} = \mu_1 + \Sigma_{1,2} * \Sigma^{-1}_{2,2}(x_2 - \mu_2)$$ I read the first as &quot;Take $\mu1$ and augment it by some factor, which is the covariance scaled by the precision (measure of how closely $X_2$ is clustered about $\mu_2$, maybe?) and projected onto the distance of the specific $x_2$ from $mu_2$.&quot; $$\Sigma_{1|2} = \Sigma_{1,1} - \Sigma_{1,2} * \Sigma^{-1}_{2,2} * \Sigma_{1,2}$$ I read the second as, &quot;take the variance about $\mu_1$ and subtract some factor, which is covariance squared scaled by the precision about $x_2$.&quot; In either case, the precision $\Sigma^{-1}_{2,2}$ seems to be playing a really important role. A few questions:  Am I right to treat precision as a measure of how closely observations are clustered about the expectation? Why is the covariance squared in the latter equation? (Is there a geometric interpretation?) So far, I've been treating $\Sigma_{1,2} * \Sigma^{-1}_{2,2}$ as a ratio, (a/b), and so this ratio acts to scale the (second) $\Sigma_{1,2}$, essentially accounting for/damping the effect of the covariance; I don't know if this is valid. Anything else you'd like to add/clarify?
95335
We're showing a pie graph that would show something like 53% male, 41% female and 7% unknown.  The "unknown" category includes many reasons we can't tell: we don't know, we can't know for legal reasons, we can't know for self-imposed privacy policy reasons, our data sources don't know, etc. The reasons are quite innumerable.  This seems like a common problem so I wanted to ask if there is a common solution?
90236
I'm reading from a book that introduces the Dirchilet distribution and then presented figures about it. But I was not really able to understand those figures. I attached the figure here at the bottom. What I don't understand are the meanings of the triangles.  Normally when one wants to plot a function of 2 variables, you take the value of var1 and va2 and then plot the value of the function value of those two variables ... which gives a visualization in a 3D dimension. But here there are 3 dimensions and one other value for the function value so it makes a visualization in 4D space. I can't understand those figures!   I hope someone can clarify them please!  EDIT: here is what I don't understand from figure 2.14a. So we have drawn from K=3 dirichlet a sample theta (which is basically a vector) that is: theta = [theta1, theta2, theta3]. The triangle plots [theta1, theta2, theta3]. The distance from the origin to each theta_i is the value of theta_i. Then for each theta_i it put a vertex and connected all three verteces and made a triangle. I know that if I plug [theta1, theta2, theta3] into dir(theta|a) I will get one number which is the joint probability of the vector theta. I also understand that the probability for continuous random variables is a measure of an area. But here we have 3 dimensions so the joint probability will be the measure of the volume of the space from the pink plane and under ... i.e the pyramid. Now I don't understand what is the role of the triangle here. What is it trying to communicate or visualize?
14197
I am performing hierarchical clustering on data I've gathered and processed from the reddit data dump on Google BigQuery.   My process is the following:   Get the latest 1000 posts in /r/politics Gather all the comments Process the data and compute an n x m data matrix (n:users/samples, m:posts/features) Calculate the distance matrix for hierarchical clustering Choose a linkage method and perform the hierarchical clustering Plot the data as a dendrogram   My question is, how do I determine what the best linkage method is? I'm currently using Ward but how do I know if I should be using single, complete, average, etc?  I'm very new to this stuff but I can't find a clear answer online as I'm not sure there is one. So what might be a good idea for my application? Note the data is relatively sparse in the sense that the n x m matrix has a lot of zeroes (most people don't comment on more than a few posts).
40408
I am reading about KNN      So I made another example to make things clearer  In this example (Image attached)    You can see there are in total 5 Greed Circles and 20 Blue Squares  by standard KNN (k=3) , X should be a Blue Square   This is obvious 2 Blue Squares vs 1 Green Circle.  But in weighted KNN things are difference  In this case we have to calculate the weight (Likelihood) for each instance  Each Green Circle likelihood is $\frac{1}{5} $ , we have 5 Green Circles  While for Blue Squares it is  $\frac{1}{20} $ , we have 20 Blue Squares  Therefore the weights around X will be $\frac{1}{5} $  Green Circle, and $\frac{2}{20} $ Blue Squares.  which means $\frac{1}{5} &gt; \frac{2}{20} $  Then X is Green Circle  But if try to think of it logically then there are more Blue Squares than Green Circles which means X more likely to be Blue Square than Green Circle.  My question is :  Am I doing anything wrong here? Can someone explain why the equation is showing Green Circle while logic says Blue Square?
104907
I have time-series data containing 1440 observations and the plot of the data is   I want to fit the Gaussian Mixture Models (GMM) to the above plot, and for the same I am using Mclust function of  package. Finally, I want a fit somewhat like this:   On using Mclust function, I do get following statistics     mclus_data &lt;- Mclust(givendataseries)    &gt; summary(mclus_data) ---------------------------------------------------- Gaussian finite mixture model fitted by EM algorithm  ----------------------------------------------------  Mclust E (univariate, equal variance) model with 8 components:   log.likelihood    n df      BIC      ICL        9525.438 1440 16 18934.52 18183.67  Clustering table:    1    2    3    4    5    6    7    8  1262    0    0    0    0   13  114   51    In the above statistic, I can not understand following:   Significance of log.likelihood, BIC and ICL. I can understand what each of them is, but what their magnitude/value refers to? It shows there are 8 clusters, but why cluster no. 2,3,4,5 has 0 values? What does this mean? From the plot it is clear that there must be two Guassians, but why Mclust function shows there are 8 Guassians?   Update: Actually, I want to do model based clustering of time series data. But currently  I want to fit the distribution to my raw data, as shown in Figure 1 on page no. 3 of  paper. For your quick reference, mentioned figure in said paper is
35915
Why do we care about eigenvalues of graphs?  Of course, any novel question in mathematics is interesting, but there is an entire discipline of mathematics devoted to studying these eigenvalues, so they must be important.  I always assumed that spectral graph theory extends graph theory by providing tools to prove things we couldn't otherwise, somewhat like how representation theory extends finite group theory.  But most results I see in spectral graph theory seem to concern eigenvalues  not as means to an end, but as objects of interest in their own right.  I also considered practical value as motivation, e.g. using a given set of eigenvalues to put bounds on essential properties of graphs, such as maximum vertex degree.  But I can't imagine a situation in which I would have access to a graph's eigenvalues before I would know much more elementary information like maximum vertex degree.  (EDIT: for example, dtldarek points out that $\lambda_2$ is related to diameter, but then why would we need $\lambda_2$ when we already have diameter?  Is this somehow conceptually beneficial?)     So, what is the meaning of graph spectra intuitively?  And for what practical purposes are they used?  Why is finding the eigenvalues of a graph's adjacency/Laplacian matrices more than just a novel problem?
106596
I ran a xgboost model. I don't exactly know how to interpret the output of xgb.importance.  What is the meaning of Gain, Cover, and Frequency and how do we interpret them?  Also, what does Split, RealCover, and RealCover% mean? I have some extra parameters   Are there any other parameters that can tell me more about feature importances?  From the R documentation, I have some understanding that Gain is something similar to Information gain and Frequency is number of times a feature is used across all the trees. I have no idea what Cover is.  I ran the example code given in the link (and also tried doing the same on the problem that I am working on), but the split definition given there did not match with the numbers that I calculated.  importance_matrix   Output:             Feature         Gain        Cover    Frequence   1:            xxx 2.276101e-01 0.0618490331 1.913283e-02   2:           xxxx 2.047495e-01 0.1337406946 1.373710e-01   3:           xxxx 1.239551e-01 0.1032614896 1.319798e-01   4:           xxxx 6.269780e-02 0.0431682707 1.098646e-01   5:          xxxxx 6.004842e-02 0.0305611830 1.709108e-02  214:     xxxxxxxxxx 4.599139e-06 0.0001551098 1.147052e-05 215:     xxxxxxxxxx 4.500927e-06 0.0001665320 1.147052e-05 216:   xxxxxxxxxxxx 3.899363e-06 0.0001536857 1.147052e-05 217: xxxxxxxxxxxxxx 3.619348e-06 0.0001808504 1.147052e-05 218:  xxxxxxxxxxxxx 3.429679e-06 0.0001792233 1.147052e-05