I'm now reading Learning PySpark, and in the book, the author first creates a dataframe:  df_miss = spark.createDataFrame([(1, 143.5, 5.6, 28, 'M', 100000),                               (2, 167.2, 5.4, 45, 'M', None),                               (3, None , 5.2, None, None, None),                               (4, 144.5, 5.9, 33, 'M', None),                               (5, 133.2, 5.7, 54, 'F', None),                               (6, 124.1, 5.2, None, 'F', None),                              (7, 129.2, 5.3, 42, 'M', 76000), ],                              [' id', 'weight', 'height', 'age','gender', 'income'])   then he uses this method to calculate the percentage of missing value:  df_miss.agg(*[(1 - (fn.count( c) / fn.count('*'))).alias( c + '_missing')                for c in df_miss.columns ]).show()   What are the two * for, especially the second one? Are there any resources about this kind of expression?? Thanks a lot!
866
In the following method definitions, what does the * and ** do for param2?  def foo(param1, *param2): def bar(param1, **param2):
38410
In almost all of the analysis work that I've ever done I use:   set.seed(42)    It's an homage to Hitchhiker's Guide to the Galaxy. But I'm wondering if I'm creating bias by using the same seed over and over.
89690
I'm doing a textmining project about a videogame review dataset. I divided de reviews into a positive (reviewgrade > 8) and negative (reviewgrade &lt; 3) and want to draw conclusions based on the frequency of different grading factors mentioned in the text. I now have a dataframe with counts of certain categories mentioned per review.  0    13249 1     1565 2      292 3       65 4       19 5        6 6        4 7        3 8        2 9        1 Name: positive: gameplay, dtype: int64   0    5043 1     348 2      66 3      18 4       8 5       3 Name: negative: gameplay, dtype: int64   For example this are the distributions of the amount of reviews which have a certain amount of mentions of gameplay related words. There are for example 1565 reviews with precisely 1 mention of a word related to the category gameplay.  Now I want to use this data to determine if on average gameplay gets mentioned more in negative context than in positive or vice versa. Because the data is not normal distributed, I assumed a t-test is not appropriate, and found online that a Mann–Whitney U test might be suitable.  &gt; wilcox.test(positive$gameplay, negative$gameplay, alternative = "two.sided", conf.int = TRUE)      Wilcoxon rank sum test with continuity correction  data:  positive$gameplay and negative$gameplay W = 43702925, p-value &lt; 2.2e-16 alternative hypothesis: true location shift is not equal to 0 95 percent confidence interval:  2.963844e-05 4.797412e-05 sample estimates: difference in location            4.995671e-05    I do however get a very high W value, which doesn't seem to be correct. Is this not the appropriate test, or am I interpreting this wrong?
46032
I have searched the stackexchange for a resolution of this problem, but I am still getting an error message when I put in the following (for now neglecting other things in my document, such as documentclass, begin{document}, etc.):  \begin{frame} \frametitle{explanation} \begin{columns} \begin{column}{width=0.5\textwidth}    some text here \end{column} \begin{column}     \begin{center}      \includegraphics[width=0.5\textwidth]{image1.jpg}            \end{center} \end{column} \end{columns} \end{frame}   When compiled, I get the error: Missing number, treated as zero. I should appreciate any help.
49746
The  (chapter 6.5, set function specification) the following behaviour for aggregate functions on empty result sets:  COUNT(...) = 0 AVG(...) = NULL MIN(...) = NULL MAX(...) = NULL SUM(...) = NULL   Returning NULL for AVG, MIN and MAX makes perfect sense, since the average, minimum and maximum of an empty set is undefined.  The last one, however, bothers me: Mathematically, the SUM of an empty set is well-defined: 0. Using 0, the  of addition, as the base case makes everything consistent:  SUM({})        = 0    = 0 SUM({5})       = 5    = 0 + 5 SUM({5, 3})    = 8    = 0 + 5 + 3 SUM({5, NULL}) = NULL = 0 + 5 + NULL   Defining SUM({}) as null basically makes "no rows" a special case that does not fit in with the others:  SUM({})     = NULL  = NULL SUM({5})    = 5    != NULL + 5 (= NULL) SUM({5, 3}) = 8    != NULL + 5 + 3 (= NULL)   Is there some obvious advantage of the choice that was made (SUM being NULL) that I have missed?
45552
Example code:  (pc.cr &lt;- princomp(USArrests))   summary(pc.cr) loadings(pc.cr)  ## note that blank entries are small but not zero   I am getting different outputs from each, and I am not sure I understand what the difference is.  Here is the output:  &gt; summary(pc.cr) Importance of components:                            Comp.1      Comp.2      Comp.3       Comp.4 Standard deviation     82.8908472 14.06956001 6.424204055 2.4578367034 Proportion of Variance  0.9655342  0.02781734 0.005799535 0.0008489079 Cumulative Proportion   0.9655342  0.99335156 0.999151092 1.0000000000   &gt; loadings(pc.cr)  ## note that blank entries are small but not zero  ...                 Comp.1 Comp.2 Comp.3 Comp.4 SS loadings      1.00   1.00   1.00   1.00 Proportion Var   0.25   0.25   0.25   0.25 Cumulative Var   0.25   0.50   0.75   1.00   P.S: how can I get access to the table created by summary(pc.cr)?? (I can't seem to find it in str.)
105976
I'm trying to create a calculated column that uses the list item's ID field as part of the calculated string value. When I create a new list item, the calculated field seems to always use 0 as the ID value. I know that during creation, the new list items ID is 0 until the content database is updated, which apparently generates the actual ID. Is there a way to get the calculated column to accurately reflect the ID value of the list item?
69363
Provided a sample size "N" that I plan on using to forecast data. What are some of the ways to subdivide the data so that I use some of it to establish a model, and the remainder data to validate the model?  I know there is no black and white answer to this, but it would be interesting to know  some "rules of thumb" or usually used ratios. I know back at university, one of our professors used to say model on 60% and validate on 40%.
4620
I am trying to use a SQL expression in order to select values that are >= 25% of its max value in that column. For instance, VALUE &gt;= MAX(VALUE)*0.25, and those will ultimately be the selected points it pulls out.   Is there syntax to make that happen, or am I going to need to create a new column and use field calculator on it?   Biggest issue with that is I will have to do this for about 250 files. I am also trying to create this in ModelBuilder.  When I try it as above it just returns that there is an error with the expression.
95335
We're showing a pie graph that would show something like 53% male, 41% female and 7% unknown.  The "unknown" category includes many reasons we can't tell: we don't know, we can't know for legal reasons, we can't know for self-imposed privacy policy reasons, our data sources don't know, etc. The reasons are quite innumerable.  This seems like a common problem so I wanted to ask if there is a common solution?
106596
I ran a xgboost model. I don't exactly know how to interpret the output of xgb.importance.  What is the meaning of Gain, Cover, and Frequency and how do we interpret them?  Also, what does Split, RealCover, and RealCover% mean? I have some extra parameters   Are there any other parameters that can tell me more about feature importances?  From the R documentation, I have some understanding that Gain is something similar to Information gain and Frequency is number of times a feature is used across all the trees. I have no idea what Cover is.  I ran the example code given in the link (and also tried doing the same on the problem that I am working on), but the split definition given there did not match with the numbers that I calculated.  importance_matrix   Output:             Feature         Gain        Cover    Frequence   1:            xxx 2.276101e-01 0.0618490331 1.913283e-02   2:           xxxx 2.047495e-01 0.1337406946 1.373710e-01   3:           xxxx 1.239551e-01 0.1032614896 1.319798e-01   4:           xxxx 6.269780e-02 0.0431682707 1.098646e-01   5:          xxxxx 6.004842e-02 0.0305611830 1.709108e-02  214:     xxxxxxxxxx 4.599139e-06 0.0001551098 1.147052e-05 215:     xxxxxxxxxx 4.500927e-06 0.0001665320 1.147052e-05 216:   xxxxxxxxxxxx 3.899363e-06 0.0001536857 1.147052e-05 217: xxxxxxxxxxxxxx 3.619348e-06 0.0001808504 1.147052e-05 218:  xxxxxxxxxxxxx 3.429679e-06 0.0001792233 1.147052e-05
8537
I was asked today in class why you divide the sum of square error by $n-1$ instead of with $n$, when calculating the standard deviation.  I said I am not going to answer it in class (since I didn't wanna go into unbiased estimators), but later I wondered - is there an intuitive explanation for this?!
39781
So I have a question with my teacher's solution below and I'm confused about one of the steps in their solution. I would appreciate some clarification.  Two brands of coffee were compared. Two independent random samples of 50 people each were asked to taste either Brand A or Brand B coffee, and indicate whether they liked it or not. Eighty four percent of the people who tasted Brand A liked it; the analogous sample proportion for Brand B was ninety percent.   (A) At α = 0.01, is there a significant difference in the proportions of individuals who like the two coffees? Use the p-value approach.   So I circled in blue a step where I'm confused about in the teacher's solution. I don't understand why P(Z&lt;-0.89) = 0.2061.   On the z-score table -0.89 is equal to 0.1867 so shouldn't it be that value instead? where did 0.2061 come from? Is it possible that the teacher made an error or am I not understanding something?
43346
How do you calculate fields with null values?  I have multiple fields that may have null values, that aren't being calculated in the third field.    import arcpy with arcpy.da.UpdateCursor(r"...gdb\test.gdb\test",["A", "B", "C"]) as cursor:     for row in cursor:         row[2] = row[0] + row[1]         cursor.updateRow(row)   The result of that calculation is this error:  Traceback (most recent call last):   File "C:\Users\Desktop\updatecursor2.py", line 4, in &lt;module&gt;     row[2] = row[0] + row[1] TypeError: unsupported operand type(s) for +: 'NoneType' and 'float'   The result of the script are the same results when I try to use the field calculator:    How do you use the data update cursor, or field calculator to calculate the sum of the values in a new field if you have null values?  I'm using ArcGIS 10.1 SP1 for Desktop.
123783
There is something that I am doing wrong in the exercise below and I would appreciate some help figuring it out.  Let $X_1, X_2, \ldots, X_5$ be a random sample from a $\Gamma \left(3,3 \right)$ distribution . Define $W=\sum_{i=1}^5 X_i $.    Obtain the distribution of $W$ and of $2W/3$ Find $c_1$ and $c_2$ such that $P \left( c_1 &lt; W &lt;c_2 \right)=0.95 $     Since $W$ is the sum of iid gamma variables, $W \sim \Gamma \left(15,3 \right)$ and now a straightforward transformation argument shows that $Y=\frac{2}{3} W \sim \chi^2 \left(30 \right) $. This completes question 1.  Now for question 2, my thoughts were to transform these probabilities using the result from question 1, namely that $\frac{2}{3} W \sim \chi^2 \left(30 \right) $. Hence, unless I have made a horrible mistake here:  $$P \left( \frac{2}{3} c_1 &lt; Y &lt; \frac{2}{3} c_2 \right)=0.95 $$  We can find the constants from a chisquare table for $30$ degrees of freedom. In my case, they are $16.791$ for the lower bound and $46.979$ for the upper one.   Then if we equate $\frac{2}{3}c_1=16.791$ and solve for $c_1$ we should be able to find the constant for the above gamma confidence interval, right?   The problem is that according to $R$, $P \left( W&lt;c_1 \right) =1 $, instead of  the desired $0.025$ which means I have gone wrong somewhere. I do not think it is in question though $1$ as I have verified the result many times.   Thank you.
94380
I've compared the logistic regression models on R (glm) and on Spark (LogisticRegressionWithLBFGS) on a dataset of 390 obs. of 14 variables.  The results are completely different in the intercept and the weights. How to explain this?  Here is the results of Spark (LogisticRegressionWithLBFGS) :  model.intercept  :   1.119830027739959 model.weights :  GEST    0.30798496002530473  DILATE  0.28121771009716895  EFFACE  0.01780105068588628  CONSIS -0.22782058111362183  CONTR  -0.8094592237248102  MEMBRAN-1.788173534959893  AGE    -0.05285751197750732  STRAT  -1.6650305527536942  GRAVID  0.38324952943210994  PARIT  -0.9463956993328745  DIAB   0.18151162744507293  TRANSF -0.7413500749909346  GEMEL  1.5953124037323745   Here is the result of R :               Estimate Std. Error z value Pr(&gt;|z|)    (Intercept)  3.0682091  3.3944407   0.904 0.366052     GEST         0.0086545  0.1494487   0.058 0.953821     DILATE       0.4898586  0.2049361   2.390 0.016835 *   EFFACE       0.0131834  0.0059331   2.222 0.026283 *   CONSIS       0.1598426  0.2332670   0.685 0.493196     CONTR        0.0008504  0.5788959   0.001 0.998828     MEMBRAN     -1.5497870  0.4215416  -3.676 0.000236 ***    AGE         -0.0420145  0.0326184  -1.288 0.197725     STRAT       -0.3781365  0.5860476  -0.645 0.518777     GRAVID       0.1866430  0.1522925   1.226 0.220366     PARIT       -0.6493312  0.2357530  -2.754 0.005882 **   DIAB         0.0335458  0.2163165   0.155 0.876760     TRANSF      -0.6239330  0.3396592  -1.837 0.066219 .   GEMEL        2.2767331  1.0995245   2.071 0.038391 *   Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
82720
I viewed in this post  a solution to my question, but...    Are there any easier way (or alternative) to do this using colors?  Thank you. ------------------ edited after showed than it´s duplicate ------------- I think that there are many sols, but for example this is easy using a normal table without pgf...     I´m trying to adapt to similar colors to apply to the table probabilitiess