I have 15,000 address strings which I have written a piece of code which assigns them to broad geographic region e.g. American address assigned to North America. I would like to check the accuracy of my code. However I must manually check the strings. To do this I've taken a sample (SRS). I set a margin of error of 5%, a CI of 95% and a response distribution of 50%. Therefore, I assumed that I would assign the correct region half the time.  The issue here is that I am much more accurate then I anticipated. Of my sample of 375,372 (99%) were assigned to the correct location. This means I am unable to calculate lower and upper confidence limits.  Is there any other types of formula I could use when your response distribution is skewed to a highly positive outcome? Or is the only remedy to simply increase my sample size?
15147
What is the best technique to calculate a confidence interval of a binomial experiment, if your estimate is that $p=0$ (or similarly $p=1$) and sample size is relatively small, for example $n=25$?
20761
Credit:   function getDirection(city1, city2, unit) {   var directions = Maps.newDirectionFinder()     .setOrigin(city1).setDestination(city2)     .setMode(Maps.DirectionFinder.Mode.DRIVING)     .getDirections();   var d = directions.routes[0].legs[0].distance.text;    var distance, value = d.split(" ")[0].replace(",", ""), text = d.split(" ")[1];   if(text == unit) {     distance = value;   } else if(text == "km" &amp;&amp; unit == "mi") {     distance = value / 1.6;   } else {     distance = value * 1.6;   }   return Math.round(distance); }   I have used this code snippet in one of my sheets. I am working on a dataset with roughly 6000 entries and I am getting some errors because of the large volume.      Script invoked too many times per second for this Google user account. (line 0).      Service invoked too many times for one day: route. (line 4).   I considered purchasing a Google Apps business account to increase my quota, but that might not even help. Do you maybe have any advice for me in this regard?
9388
I have taken a sample of $n$ data points from a population. Each of these points has a true value (known from ground truth) and an estimated value. I then calculate the error for each sampled point and then calculate the RMSE of the sample.  How can I then infer some sort of confidence interval around this RMSE, based upon the sample size $n$?  If I was using the mean, rather than the RMSE, then I wouldn't have a problem doing this as I can use the standard equation  $ m = \frac{Z \sigma}{\sqrt{n}} $  but I don't know whether this is valid for RMSE rather than the mean. Is there some way that I can adapt this?  (I have seen , but I don't have issues with whether my population is normally-distributed, which is what the answer there deals with)
60286
I have created two probit models, and even though p values are significant and sample spaces are large- 10000 samples in both- standard errors seem to be about 6.27e. What could be the reason for that? Here are the results: ============================================================================== Dep. Variable:                      y   No. Observations:                 9413 Model:                         Probit   Df Residuals:                     9412 Method:                           MLE   Df Model:                            0 Date:                Sat, 30 Jan 2021   Pseudo R-squ.:                 0.04258 Time:                        10:41:23   Log-Likelihood:                -6067.9 converged:                       True   LL-Null:                       -6337.8 Covariance Type:            nonrobust   LLR p-value:                       nan ==============================================================================                  coef    std err          z      P&gt;|z|      [0.025      0.975] ------------------------------------------------------------------------------ x1             0.0018   6.27e-05     28.368      0.000       0.002       0.002 ==============================================================================```  ```                          Probit Regression Results                            ============================================================================== Dep. Variable:                      y   No. Observations:                 9888 Model:                         Probit   Df Residuals:                     9887 Method:                           MLE   Df Model:                            0 Date:                Sat, 30 Jan 2021   Pseudo R-squ.:                 0.03962 Time:                        13:33:44   Log-Likelihood:                -6225.9 converged:                       True   LL-Null:                       -6482.7 Covariance Type:            nonrobust   LLR p-value:                       nan ==============================================================================                  coef    std err          z      P&gt;|z|      [0.025      0.975] ------------------------------------------------------------------------------ x1             0.0020   6.18e-05     32.747      0.000       0.002       0.002 ==============================================================================
41987
I am studying statistics and I am wondering when it comes to standard error or a sampling if the calculation changes when there are weights added.   I have a weighted mean:  $$\mu_{w} = \dfrac{\sum_{i=1}^Nw_ix_i}{\sum_{i=1}^Nw_i}$$ and a weighted variance calculated by:  $$s^2_{w}=\dfrac{\sum_{i=1}^Nw_i}{(\sum_{i=1}^Nw_i)^2-\sum_{i=1}^Nw_i^2}\cdot \sum_{i=1}^N(x_i-\mu)^2$$  is the sampling error still calculated as  $$\text{SE}=\sqrt{\dfrac{s^2_{w}}{n}}$$
10508
Problem: I have an address field from an Access database which has been converted to SQL Server 2005. This field has everything all in one field. I need to parse out the address's individual sections into their appropriate fields in a normalized table. I need to do this for approximately 4,000 records, and it needs to be repeatable. Assumptions:  Assume an address in the US (for now)  assume that the input string will sometimes contain an addressee (the person being addressed) and/or a second street address (i.e. Suite B)  states may be abbreviated  zip code could be standard 5 digits or zip+4  there are typos in some instances   UPDATE: In response to the questions posed, standards were not universally followed; I need need to store the individual values, not just geocode and errors means typo (corrected above) Sample Data:  A. P. Croll &amp; Son  2299 Lewes-Georgetown Hwy, Georgetown, DE 19947  11522 Shawnee Road, Greenwood DE  19950  144 Kings Highway, S.W. Dover, DE  19901  Intergrated Const. Services 2 Penns Way Suite 405 New Castle, DE  19720  Humes Realty 33 Bridle Ridge Court, Lewes, DE  19958  Nichols Excavation 2742 Pulaski Hwy Newark, DE  19711  2284 Bryn Zion Road, Smyrna, DE  19904  VEI Dover Crossroads, LLC  1500 Serpentine Road, Suite 100 Baltimore MD  21  580 North Dupont Highway Dover, DE  19901  P.O. Box 778 Dover, DE  19903
43054
I saw an example on a blog where a k nearest neighbors algorithm was run to predict responses on a test data set.The MSE(mean squared error) of the predictions was larger than the variance of the response variable in the test set. Shouldn't the MSE always be less than the variance of the response variable as SSTO(total sum of squares) = RSS(model sum of squares) + SSE(error sum of squares).
88671
I have a list of suburbs in Queensland, Australia along with their lat/longs.   However, when I try to read in the file in R and assign a CRS, R is giving the following error message:      Error in proj4string&lt;-(*tmp*, value = ) :     Geographical CRS given to non-conformant data: 153.53332931   Here are relevant lines of the code,  # Loading suburb list  qs &lt;- read.csv(file = '../data/Australian_Post_Codes_Lat_Lon.csv',                 header = TRUE,                stringsAsFactors = FALSE)  suburb &lt;- qs %&gt;%    filter(state == 'QLD') %&gt;%    select(postcode,          suburb,          lat,          lon)   coordinates(suburb) &lt;- c('lat', 'lon') # Assigning proper Coordinate Reference System (CRS) to Lat/Long. proj4string(suburb) &lt;- CRS("+init=epsg:4326")   I am using the csv file provided
28199
I just wrote an answer to , which was trying to find numeric strings with or without commas (e.g. 1,234,567.89). I wanted to be thorough because I've seen this problem a lot, both on SO and in real-world code.   When I finished, I found 7 other new answers waiting for me, mostly variations on each other. Which would be fine, except for one thing: They're garbage answers. Unless I'm severely mistaken, every answer but mine (so far) is deeply flawed, and I want to downvote all of them. Yeah, I'm a stickler on regex, but IMHO if your pattern lets stuff like  ...,.,,.,,..9 or even ... pass as "numeric input," then you've failed to meet even the bare minimum requirements.  Enough venting. The reason I'm whining to Meta about this is that I'm about to do something that'll REALLY look like tactical downvoting, and I'm mildly concerned about the consequences. If I answer a question and then rain downvotes on every competing answer, will I get flagged for abuse? That would certainly seem like suspicious behavior to me.  More of an opinion question: At what point does a semi-correct answer deserve a downvote? Obviously, if it were just a typo, I'd edit it myself. Where's the line between "close enough" and "this answer sucks?"  Basically, is it just me? You've seen the question; am I right that these answers deserve downvotes? Seven people thought that [0-9,.]+ was good enough to declare something a number, so why am I the only one who doesn't?    Note: I found some related questions, but nothing that seemed like a duplicate:    - More of an etiquette question, plus the voter hadn't posted his own answer.  - I guess this pretty much covers my second question.   Also, as some have noted, my answer ended up pretty long and thorough. I put in a lot of effort because, with all this talk about how the other answers sucked, I wanted to be extra sure mine didn't. I don't expect pages of detail in a normal SO answer.
116328
After a table of addresses is geocoded, ArGIS provides information about each geocoded address, among those the "match score" of the candidate to which the address was matched, which ranges from 0 to 100. According to their documentation "The match score is based on how well the locations found in the reference data match with the address data being searched."  It seems intuitive that 100 means an address with the exact name was found in the Address locator and 0 means no such address was found. However, I could not find any information about how exactly this score is calculated, particularly if values are somewhere between the extremes. I this known?  I found the pointer to  in the answer to , but I could not find any information in that paper that would answer the question.
57754
I know the "textbook" estimate of the standard error of a proportion is $SE=\sqrt{\frac{p(1-p)}{n}}$, but does this hold up when the data are weighted?
40862
I am writing an application to performance test all kinds of map services, primarily AGS 9.x, AGS 10, and WMS 1.x.  Part of the application involves generating random bounding boxes for individual requests, within the service's full extent. This part is working fine for geographic and projected coordinate systems when the service's full extent is known (e.g. through an AGS service's fullExtent property).  My problem is with WMS: each layer in a GetCapabilities response can define its bounding area in >= 1 CRSs. Some parts of the application need to know if a service's CRS is geographic or projected, so in order to remove ambiguity in WMS I am always using the layer's LatLonBoundingBox which is always defined and in EPSG:4326. I then have to calculate a full service bounding box based on all layers that go into an individual request (which are randomised). This is where it gets tricky.  I am getting lost because for each lat / lon bounding box the LLx (lower left longitude) could be a larger or smaller number than the URx (upper right longitude), depending on which meridians it spans. Every time I start drawing square or circular diagrams I think I have an approach figured out, and then find a case that ruins it and my brain turns to mush.  I am going to keep thrashing at it until it works, and if I get a solution post it here, but I'm sure there has to be an accepted and fully tested approach which would make my life easier. I just can't find it right now.
105583
I have done some interpolation in Geostatistical Analyst in ArcGIS and I got the Standard Deviation (SD) a bit different from the SD that I calculated in Excel Spreadsheet.   What algorithm does each package use? Why are the results different?
101191
I have a dataset with a few million rows and ~100 columns. I would like to detect about 1% of the examples in the dataset, which belong to a common class. I have a minimum precision constraint, but due to very asymmetric cost I am not too keen on any particular recall (as long as I am not left with 10 positive matches!)  What are some approaches that you would recommend in this setting? (links to papers welcome, links to implementations appreciated)
52094
I have over 400 open-ended responses. The total has been coded by one rater, and a 10% identical sample of the total has been coded by 2 raters. There is a choice of 14 nominal codes, of which the rater can use any number per case, though most cases have between 1-4 codes. Which statistical test is the most appropriate to determine if I have inter-rater agreement on the 10% sample?   Also am I right in assuming that I will need to treat each code as a variable and look for the level of agreement for each variable?   I'm using SPSS 21.  Much thanks, Kellie
39781
So I have a question with my teacher's solution below and I'm confused about one of the steps in their solution. I would appreciate some clarification.  Two brands of coffee were compared. Two independent random samples of 50 people each were asked to taste either Brand A or Brand B coffee, and indicate whether they liked it or not. Eighty four percent of the people who tasted Brand A liked it; the analogous sample proportion for Brand B was ninety percent.   (A) At α = 0.01, is there a significant difference in the proportions of individuals who like the two coffees? Use the p-value approach.   So I circled in blue a step where I'm confused about in the teacher's solution. I don't understand why P(Z&lt;-0.89) = 0.2061.   On the z-score table -0.89 is equal to 0.1867 so shouldn't it be that value instead? where did 0.2061 come from? Is it possible that the teacher made an error or am I not understanding something?
80674
I have a bunch of raw data values that are dollar amounts and I want to find a confidence interval for a percentile of that data.  Is there a formula for such a confidence interval?