I want to estimate percentile statistics for time-series pairs $(t, y)$ over varying granularities of time (hour, day, month, etc)  For example, if I have the following pairs (10:59, 3), (11:01, 2), (11:32, 5), the first point is part of a different hour window, but all three are part of the same day window.  When a new point arrives, I am recomputing the hour window statistics directly, by iterating over all data points in the window. Unfortunately, this does not scale to the longer time windows.  Is there a way to incrementally update the statistics for the longer time windows?
57619
I want to estimate the quantile of some data. The data are so huge that they can not be accommodated in the memory. And data are not static, new data keep coming. Does anyone know any algorithm to monitor the quantiles of the data observed so far with very limited memory and computation? I find  useful, but it does not work very well for my data, which are extremely heavy-tailed distributed.
9024
I have a question related to modeling short time-series. It is not a question , but how. What method would you recommend for modeling (very) short time-series (say of length $T \leq 20$)? By "best" I mean here the most robust one, that is the least prone to errors due the fact of limited numbers of observations. With short series single observations could influence the forecast, so the method should provide a cautious estimate of errors and possible variability connected to the forecast. I am generally interested in univariate time-series but it would be also interesting to know about other methods.
115278
I want to detect seasonality in data that I receive. There are some methods that I have found like the seasonal subseries plot and the autocorrelation plot but the thing is I don't understand how to read the graph, could anyone help?  The other thing is, are there other methods to detect seasonality with or without the final result in graph?
91469
Suppose I have following non-periodic time series. Obviously the trend is decreasing and I would like to prove it by some test (with p-value). I am unable to use classic linear regression due to strong temporal (serial) auto-correlation among values.  library(forecast) my.ts &lt;- ts(c(10,11,11.5,10,10.1,9,11,10,8,9,9,                6,5,5,4,3,3,2,1,2,4,4,2,1,1,0.5,1),             start = 1, end = 27,frequency = 1) plot(my.ts, col = "black", type = "p",      pch = 20, cex = 1.2, ylim = c(0,13)) # line of moving averages  lines(ma(my.ts,3),col="red", lty = 2, lwd = 2)     What are my options?
122301
When developing a general purpose time-series software, is it a good idea to make it scale invariant? How would one do that?  I took a time series of around 40 points, and then multiplied by factors ranging from 10E-9 to 10E3 and then ran through the ARIMA functions of Forecast Pro and Minitab. In Forecast Pro, all resulted in the same answer (automatic modeling), whereas in Minitab, they were not. Not sure what Forecast Pro does, but they might just scale up or down all the numbers to a certain scale (let's say 100s) before running the model. Is this good idea in general?
59745
I have a range of dates and a measurement on each of those dates.  I'd like to calculate an exponential moving average for each of the dates.  Does anybody know how to do this?  I'm new to python.  It doesn't appear that averages are built into the standard python library, which strikes me as a little odd.  Maybe I'm not looking in the right place.  So, given the following code, how could I calculate the moving weighted average of IQ points for calendar dates?  from datetime import date days = [date(2008,1,1), date(2008,1,2), date(2008,1,7)] IQ = [110, 105, 90]   (there's probably a better way to structure the data, any advice would be appreciated)
109448
I must clarify immediately that I am a practicing software developer, not a statistician, and that my college stats class was a very long time ago…  That said, I would like to know if there is a method for accumulating a set of descriptive statistics that could then be used to produce a boxplot, that does not entail storing a bunch of individual samples?  What I am trying to do is produce a graphical summary of queue service times within a complex multi-queue process. I have in the past used a package called tnftools that allowed large samples to be accumulated and then post-processed into a nice graph of response times and outliers… But tnftools are not available for my current platform.  Ideally I would like to be able to accumulate a set of descriptive statistics "on the fly" as the process runs, and then extract the data for analysis on demand. But I cannot simply have the process accumulate samples as the memory / IO involved in doing so would have an unacceptable impact on the performance of the system.
18245
I have samples from a highly skewed (looking like an exponential distribution) dataset about users' participation (e.g.: number of posts), that have different sizes (but not less than 200) and I want to compare their mean. For that, I'm using two-sample unpaired t-tests(and t-tests with the Welch’s factor, when the samples had different variances). As I have heard that, for really large samples, it doesn't matter that the sample are not normal distributed.  Someone, reviewing what I've done, said that the tests I am using were not suitable for my data. They suggested to log-transform my samples before using the t-tests.  I am a beginner, so it sounds really confusing to me to answer my research questions with "log of participation metric".  Are they wrong? Am I wrong? If they are wrong, is there a book or scientific paper which I could cite/show them? If I am wrong, which test should I use?
69363
Provided a sample size "N" that I plan on using to forecast data. What are some of the ways to subdivide the data so that I use some of it to establish a model, and the remainder data to validate the model?  I know there is no black and white answer to this, but it would be interesting to know  some "rules of thumb" or usually used ratios. I know back at university, one of our professors used to say model on 60% and validate on 40%.
80401
I have a novice background in time series (some ARIMA estimation/forecasting) and am facing a problem I don't fully understand.  Any help would be greatly appreciated.  I am analyzing multiple time series, all over the same time interval and all of the same frequency, all describing a similar type of data. Each series is just one variable, there are no other corresponding predictors that I'm looking at.   I have been asked to estimate a single model that describes ALL the series - for example, imagine I could find one ARIMA (p,d,q) with the same orders, coefficients, etc. that could fit all the series.  My supervisor does not want me to separately estimate each series, nor does he want me to do some sort of VAR model with dependencies between the series.  My question is:  what would I even call such a model, and how might I go about estimating / forecasting it?  If it's easier for you to use code examples, I speak both SAS and R.
113648
If i have this sorted data:  $1, 2, 2, 2, 3, 3, 4, 4, 5, 5, 5, 6, 6, 6, 6, 6, 7, 7, 8, 8, 8, 9, 9, 10, 10$  They are 25 numbers.  How should I calculate the quartiles and percentiles?  I try this:  $Q_2 = 2 * \frac{25}{4} = 12.5$  $Q_1 = 1 * \frac{25}{4} = 6.25$   $P_{50} = 50 * 25/100 = 12.5$  Well, but what should I do when it is decimal, for non-clustered data? In some part I have seen,     approach the nearest larger whole.   So, $12.5 \rightarrow 13$  So, $6.25 \rightarrow 7$  In other places I read:     the value will be the average between the data that is at the left of   the position with the data on the right.   So, What should I really do?
32635
Please pardon my ignorance, but...  I keep finding myself in a situation, where I'm faced with a bunch of new data I managed to find. This data usually looks something like this:  Date     Number1  Number2  Category1  Category2 20120125      11      101        Dog      Brown 20120126      21       90        Cat      Black 20120126      31      134        Cat      Brown (...)   Usually at first glance I can't really tell if there are any trends here. The correlations between the various columns may not be very significant, but I would be delighted if I didn't have to manually create a plot for every possible combination of columns/categories.  Is there a tool out there which would accept a table of data along with information which columns should be treated as numbers, dates and categories and then proceed to plot:   correlations between each two numerical columns correlations between each two numerical columns, with separate trend lines for each category each number column as a time series, each number column as a time series, separated by category, etc.   In the end this would generate a large number of plots, most of which would show only noise. Ideally, the tool could score the plots by correlation and in the end display a slideshow starting with the highest scoring plots. This would be a very imperfect, but useful first glance at the dataset.  So? Is there a tool everyone uses for this and I just don't know about it, or is this something we need to make?
109616
Suppose we have $N$ measurable variables, $(a_1, a_2, \ldots, a_N)$, we do a number $M &gt; N$ of measurements, and then wish to perform  on the results to find the axes of highest variance for the $M$ points in $N$-dimensional space.  (Note: assume that the means of $a_i$ have already been subtracted, so $\langle a_i \rangle = 0$ for all $i$.)  Now suppose that one (or more) of the variables has significantly different characteristic magnitude than the rest.  E.g. $a_1$ could have values in the range $10-100$ while the rest could be around $0.1-1$.  This will skew the axis of highest variance towards $a_1$'s axis very much.  The difference in magnitudes might simply be because of an unfortunate choice of unit of measurement (if we're talking about physical data, e.g. kilometres vs metres), but actually the different variables might have totally different dimensions (e.g. weight vs volume), so there might not be any obvious way to choose "comparable" units for them.  Question:  I would like to know if there exist any standard / common ways to normalize the data to avoid this problem.  I am more interested in standard techniques that produce comparable magnitudes for $a_1 - a_N$ for this purpose rather than coming up with something new.  EDIT:  One possibility is to normalize each variable by its standard deviation or something similar.  However, the following issue appears then: let's interpret the data as a point cloud in $N$-dimensional space.  This point cloud can be rotated, and this type of normalization will give different final results (after the SVD) depending on the rotation.  (E.g. in the most extreme case imagine rotating the data precisely to align the principal axes with the main axes.)  I expect there won't be any rotation-invariant way to do this, but I'd appreciate if someone could point me to some discussion of this issue in the literature, especially regarding caveats in the interpretation of the results.
46539
I've got an excerpt of my spreadsheet below. It's a time tracking application containing a start date/day, a session ID and the duration of the session. Column A is populated by adding the duration in Column D. I'm trying to total the durations, but keep getting incorrect values in cases where the total duration is > 24 hours.  Consider        A                   B        C                D          1  2012/05/22 01:00  |  Wed  |  Session 23341  |  9:00 2  2012/05/23 10:00  |  Wed  |  Session 23342  | 22:00 3  2012/05/25 08:00  |  Fri  |                 |  0:00   As you can see, the date differential between A2 and A3 demonstrates that the value of 22:00 in D2 represents a duration of 46 hours (05/23 10:00 + 46:00 = 05/25 08:00). However, Column D is only displaying the 22:00 in excess of a full 24 hour day.  My attempts to read D2 have included int(d2) = 0.00  /  int(d2)*24 = 0.00  /  hour(d2) = 22.00  /  int(a3-a2)*24 =0.00 etc. and have been unsuccessful in pulling a value of "46".   How can I accurately represent D2 to achieve accurate summary data?
58121
I'm using R(3.1.1), and ARIMA models for forecasting. I would like to know what should be the &quot;frequency&quot; parameter, which is assigned in the ts() function, if im using time series data which is:  separated by minutes and is spread over 180 days (1440 minutes/day) separated by seconds and is spread over 180 days (86,400 seconds/day).  If I recall right the definition, a &quot;frequency&quot; in ts in R, is the number of observations per &quot;season&quot;. Question part 1: What is the &quot;season&quot; in my case? If the season is &quot;day&quot;, then is the &quot;frequency&quot; for minutes = 1440 and 86,400 for seconds? Question part 2: Could the &quot;frequency&quot;  also depend on what I am trying to achieve/forecast? for example, in my case, I'd like to have a very short-term forecast. One-step ahead of 10minutes each time. Would it then be possible to consider the season as an hour instead of a day? In that case frequency= 60 for minutes, frequency = 3600 for seconds? I've tried for example to use frequency = 60 for the minute data and got better results compared to frequency = 1440 (used fourier see link below by Hyndman)  (The comparison was made by using MAPE for the measure of forecast accuracy) In case the results are complete arbitrary, and the frequency cannot be changed. What would be actually the interpretation of using freq = 60 on my data? I also think it's worth mentioning that my data contains seasonality at every hour and every two hours (by observing the raw data and the Autocorrelation function)
88563
I am having sales volume data for 100 stock-keeping units (SKUs). Is there a statistical way of identifying seasonal time series?
31294
I'm currently using R to predict a time series with these instructions:  X &lt;- ts(datas, frequency=24) X.arima &lt;- Arima(X, order=c(2,1,0), seasonal=c(1,1,1)) pred &lt;- predict(X.arima, n.ahead=24) plot.ts(pred$pred)   As you can see I've data each hour, and I chose the seasonal period of 24 (one day).  I would like to improve my forecasting using an additional seasonal period in order to include the seasonal component of the week (seasonal length of 7*24=168 data)  Is there any method for this? How do you do it?  UPDATE: I've read  (your) blog page, maybe can I use the external regressors to simulate a second seasonal period?