I have an excel dataset of 24-hour moving averages for PM10 air pollution concentration levels, and need to obtain the individual hourly readings from them. The moving average data is updated every hour, so at hour t, the reading is the average of the 24 readings from t-23 to t hours, and at hour t+1, the reading is the average of t-22 to t+1, etc. I do not have any known data points to extrapolate from, just the 24-hour moving averages.  Is there any way I can obtain the individual hourly readings for time t, t+1, etc, from the moving average?  The dataset contains data over 3 years, so with 24 readings a day (at every hour), the dataset has thousands of readings.  I have tried searching for a possible way to implement a simple excel VBA code to do this, but come up empty. Most of the posts I have seen on Stackoverflow and stackexchange, or other forums, involve calculating moving averages from discrete data, which is the reverse of what I need to do here.  The few I have seen involve using matrices, which I am not very sure how to implement. () ()  Any suggestions would be greatly appreciated!
123429
Is it possible to extract data points from moving average data?  In other words, if a set of data only has simple moving averages of the previous 30 points, is it possible to extract the original data points?  If so, how?
109448
I must clarify immediately that I am a practicing software developer, not a statistician, and that my college stats class was a very long time ago…  That said, I would like to know if there is a method for accumulating a set of descriptive statistics that could then be used to produce a boxplot, that does not entail storing a bunch of individual samples?  What I am trying to do is produce a graphical summary of queue service times within a complex multi-queue process. I have in the past used a package called tnftools that allowed large samples to be accumulated and then post-processed into a nice graph of response times and outliers… But tnftools are not available for my current platform.  Ideally I would like to be able to accumulate a set of descriptive statistics "on the fly" as the process runs, and then extract the data for analysis on demand. But I cannot simply have the process accumulate samples as the memory / IO involved in doing so would have an unacceptable impact on the performance of the system.
42478
I have a time series data which consist of 24 hours of the electricity demand. I transform the data into stationary from through differences, Now after forecasting for one year, I want to transform the data into its original form to check the accuracy of the model. I want to explain my question by an example     dat &lt;- c(5,24,23,12,14,15,16,17,18,22,24)     sat&lt;- diff(dat)  sat  [1]  19  -1 -11   2   1   1   1   1   4   2  ori &lt;- diffinv(sat) &gt; ori  [1]  0 19 18  7  9 10 11 12 13 17 19  By using the diffinv which is the inverse of the difference, but the result I obtain is the original data dat. Know can any one help me that how I obtain the original data dat, mean the result of dat and ori are different.
19945
I have a table that has just two columns, Date and Profit:  Date            Profit 24 - March      25.00 24 - March      19.85 25 - March      5.25 26 - March      6.36 26 - March      10.18 26 - March      35.63   I want to calculate and display the average daily profit in one cell.  Please keep in mind that when I enter data for new dates such as 27 - March and it's profit, it should update the average daily profit cell.  Some days I do not make any sale; for example, if I make sales and profit for march 24, march 25 and march 27, I did not make a sale on March 26, I want it to calculate it regardless of me making a sale on a date or not. Meaning in a month that has 31 days, it should divide by 31 regardless on whether I made a sale on a specific date or not. Would that be possible?
10661
I am trying to forecast time series data for next $n$ time periods and I am using various forecasting techniques like simple moving average, exponential smoothing (Single,double,triple) and auto.arima().  I am getting equal forecasting values for simple moving average and single exponential smoothing and at times, even auto.arima() function in R is giving me the same forecast values.  Here is the code that I am trying to use  mod_mva &lt;- SMA(data.train,n=2) forecast_mva &lt;-forecast(mod_mva,ts_freq)   Here is the output:
117954
I am having 'hour' field as my attribute, but it takes a cyclic values. How could I transform the feature to preserve the information like '23' and '0' hour are close not far.   One way I could think is to do transformation: min(h, 23-h)  Input: [0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23]  Output: [0 1 2 3 4 5 6 7 8 9 10 11 11 10 9 8 7 6 5 4 3 2 1]   Is there any standard to handle such attributes?  Update: I will be using superviseed learning, to train random forest classifier!
61712
SQL Server 2008 R2  PIVOT IO Stats    I have a script that collects IO stats from sys.dm_io_virtual_file_stats once a day and stocks them in a table.  I want to be able to use these stats to generate graphs.  Using PIVOT I have created columns based on the dates the stats were collected  select 'read io stall',[2012-10-17], [2012-10-18], [2012-10-19], [2012-10-20], [2012-10-21] from (      select io_stall_read_ms, date_snap     from dbo.tbl_io_stats     where database_name = 'mydb'     ) up PIVOT (max(io_stall_read_ms) FOR date_snap IN ([2012-10-17], [2012-10-18], [2012-10-19], [2012-10-20], [2012-10-21])) AS pvt UNION select 'write io stall', [2012-10-17], [2012-10-18], [2012-10-19], [2012-10-20], [2012-10-21] from (      select database_name, io_stall_write_ms, date_snap     from dbo.tbl_io_stats     where database_name = 'mydb'     ) up PIVOT (max(io_stall_write_ms) FOR date_snap IN ([2012-10-17], [2012-10-18], [2012-10-19], [2012-10-20], [2012-10-21])) AS pvt   Resulting table:  Type Stat   2012-10-17  2012-10-18  2012-10-19  2012-10-20  2012-10-21 read io stall   34449971    34499918    34504701    40383037    40852412 write io stall  20948385    20996323    21001665    24130053    24193110   As is, I have to add each new date as an additional column.    Question:  Is it possible to generate the PIVOT column list automatically or is there another way to do what I'm thinking of?    Instead of   FOR date_snap IN ([2012-10-17], [2012-10-18], [2012-10-19], [2012-10-20], [2012-10-21]   Something like:  FOR date_snap IN (SELECT DISTINCT date_snap FROM tbl_io_stats)   Update with RichardTheKiwi's answer  Here is the new working query, thank you RichardTheKiwi  DECLARE @cols AS NVARCHAR(MAX), @query  AS NVARCHAR(MAX);  SET @cols = STUFF((SELECT distinct ',' + QUOTENAME(ios.date_snap)              FROM tbl_io_stats ios             FOR XML PATH(''), TYPE             ).value('.', 'NVARCHAR(MAX)')          ,1,1,'')  set @query = 'SELECT database_name + '' read io stall'', ' + @cols + ' from              (                 select database_name, io_stall_read_ms, date_snap                 from dbo.tbl_io_stats                 where database_name = ''mydb''            ) x             pivot              (                 max(io_stall_read_ms)                 for date_snap in (' + @cols + ')             ) p              union SELECT database_name + '' write io stall'', ' + @cols + ' from              (                 select database_name, io_stall_write_ms, date_snap                 from dbo.tbl_io_stats                 where database_name = ''mydb''            ) x             pivot              (                 max(io_stall_write_ms)                 for date_snap in (' + @cols + ')             ) p '  --print @query execute(@query)
53968
I have a formula which is like this  =COUNTIFS('sheet1 '!W:W,">0",'sheet1 '!A:A,"Truck")  which is count all data in column W (sheet1) larger than 0, which are called Truck in column A.  Now, I want this formula to read the same information but by consider a range of dates in between 1st of October to 15th (I have dates in Column B). Any input is greatly appreciate. Thanks
122836
I'm making a worksheet where the wind speed defines the potential energy of the location. This value is the basis of a few sheets in the same Excel file. I'm trying to write a code that where I10 is the wind speed and I12 is the potential kWh on that location. I want both I10 as I12 to be input cells where you can either say: I want to know what wind speed I need to generate 4000 kwh i.e. or my wind speed is 4.2 m/s what will my potential energy be?  This is what I made so far, but I don't get the power function to work. At the moment I defined the cell I12 (The kWh's) without VBA as =(2208,5/(54,872))*I10^3 where I10 is the windspeed.  VBA code behind sheet which doesn't work:  Private Sub Worksheet_Change(ByVal Target As Excel.Range)     If Target.Address = "$I$10" Or Target.Address = "$I$12" Then          WIND = Range("I10")         KWH = Range("I12")          Application.EnableEvents = False          If Target.Address = "$I$10" Then             Range("I12").Value = (2208.5 / (54.872)) POWER(WIND,3)         Else             Range("I10").Value = POWER( KWH (2208.5 / (54.872)),(1/3)          End If     End If     Application.EnableEvents = True End Sub   Potential solution, but not working at the moment.  Function MyPower(Number As Double, Exponent As Double) As Double     MyPower = Number ^ Exponent End Function
119405
Does anyone know how to convert string time formats in ArcMap v10.2 from 12-hour to 24-hour using a simple query in the Field Calculator? i.e. NO Python, NO editing in Excel, NO changing my Windows PC settings, etc...  This is a successful calculation I used (which I found on a discontinued forum) to convert my dates from MM/DD/YYYY to YYYY/MM/DD:  year([DateFld]) &amp; "/" &amp; Month([DateFld]) &amp; "/" &amp; Day([DateFld])  Perhaps there is some sort of modification that I could make to the above calculation that would change my times to 24-hour?  I've searched for an answer everywhere and played around with it without any luck, but I would like for the times to be in h:mm:ss format.  Forgot to mention that this is a regular shapefile, not part of a GDB.
31294
I'm currently using R to predict a time series with these instructions:  X &lt;- ts(datas, frequency=24) X.arima &lt;- Arima(X, order=c(2,1,0), seasonal=c(1,1,1)) pred &lt;- predict(X.arima, n.ahead=24) plot.ts(pred$pred)   As you can see I've data each hour, and I chose the seasonal period of 24 (one day).  I would like to improve my forecasting using an additional seasonal period in order to include the seasonal component of the week (seasonal length of 7*24=168 data)  Is there any method for this? How do you do it?  UPDATE: I've read  (your) blog page, maybe can I use the external regressors to simulate a second seasonal period?
47837
I have an existing data set that comes from the same group of people before and after they received a treatment.  The data set comes from when participants tested their blood sugar values over a 30 day period before receiving an insulin pump and a 30 day period after receiving an insulin pump. This data was obtained from user logs (archival data) and was not controlled to ensure that they were tested at fixed intervals. Participants tested themselves when they needed to test throughout the day for a period of 30 days.  My goal is to determine whether the average blood sugar across 30 days was different for the before vs after group.   Normally this would be a paired samples t-test but unfortunately the before and after group have an unequal number of data points with the after group having significantly more. People are testing more often after receiving an insulin pump.  What is the correct way to handle this?  I can collapse the data to find a mean for each participant before treatment and after treatment (across all participants) so that the data matches up and then run a paired samples t-test on this data but I think this is not the ideal solution.   Would a one way within subjects ANOVA be the appropriate test to run in this case?
18594
What is the module/method used to get the current time?
51221
I have yearly maximum NDVI data (1982-2008) and extracted all value to Excel. Also calculated slope value for each point during the 27 years. All of you know the slope value represents changes per unit (per year). Now I would like to look for significant changes in NDVI over the 27 years. So I have to get "p value" for each point. How I calculate it?   I tried it in Excel. Firstly I have to estimate t-test, but i don't know which value should be in place of  'mu' for this case! After estimating it I can find p value using TDIST function in MS excel.
103784
Background I'm working on a time series data set of energy meter readings. The length of the series varies by meter - for some I have several years, others only a few months, etc. Many display significant seasonality, and often multiple layers - within the day, week, or year. One of the things I've been working on is clustering of these time series. My work is academic for the moment, and while I'm doing other analysis of the data as well, I have a specific goal to carry out some clustering. I did some initial work where I calculated various features (percentage used on weekends vs. weekday, percentage used in different time blocks, etc.). I then moved on to looking at using Dynamic Time Warping (DTW) to obtain the distance between different series, and clustering based on the difference values, and I've found several papers related to this. Question Will the seasonality in a specific series changing cause my clustering to be incorrect? And if so, how do I deal with it? My concern is that the distances obtained by DTW could be misleading in the cases where the pattern in a time series has changed. This could lead to incorrect clustering. In case the above is unclear, consider these examples: Example 1 A meter has low readings from midnight until 8AM, the readings then increase sharply for the next hour and stay high from 9AM until 5PM, then decrease sharply over the next hour and then stay low from 6PM until midnight. The meter continues this pattern consistently every day for several months, but then changes to a pattern where readings simply stay at a consistent level throughout the day. Example 2 A meter shows approximately the same amount of energy being consumed each month. After several years, it changes to a pattern where energy usage is higher during the summer months before returning to the usual amount. Possible Directions  I've wondered whether I can continue to compare whole time series, but split them and consider them as a separate series if the pattern changes considerably. However, to do this I'd need to be able to detect such changes. Also, I just don't know if this is a suitable way or working with the data. I've also considered splitting the data and considering it as many separate time series. For instance, I could consider every day/meter combination as a separate series. However, I'd then need to do similarly if I wanted to consider the weekly/monthly/yearly patterns. I think this would work, but it's potentially quite onerous and I'd hate to go down this path if there's a better way that I'm missing.  Further Notes These are things that have come up in comments, or things I've thought of due to comments, which might be relevant. I'm putting them here so people don't have to read through everything to get relevant information.  I'm working in Python, but have rpy for those places where R is more suitable. I'm not necessarily looking for a Python answer though - if someone has a practical answer of what should be done I'm happy to figure out implementation details myself. I have a lot of working &quot;rough draft&quot; code - I've done some DTW runs, I've done a couple of different types of clustering, etc. I think I largely understand the direction I'm taking, and what I'm really looking for is related to how I process my data before finding distances, running clustering, etc. Given this, I suspect the answer would be the same whether the distances between series are calculated via DTW or a simpler Euclidean Distance (ED). I have found these papers especially informative on time series and DTW and they may be helpful if some background is needed to the topic area:
89064
I'm trying to conduct a two sample t-test on  on food and drink workers in the Seattle MSA to see if there is a mean difference in levels of employment before and after the enactment of an increase in the minimum wage after April 1, 2015. The background for this project is that many people say that an increase in the minimum wage will force low-skill employers to cut their staff in order to minimize costs.  I've included my code below but some have indicated that my statistics aren't correct. Is there a better way of doing this with a t-test? How could I do a regression analysis on this same data? Would I have to create a new variable with the minimum wage for each corresponding observation date?  The data into two variables (observations before and after the minimum wage increase took effect),.  #seattleData &lt;- read.table(file=file.choose(), 2 # header=T, sep=",",)  food_drink_workers &lt;- seattleData$food_drink_workers  afterMinWageLaw &lt;- food_drink_workers[304:346]  beforeMinWageLaw &lt;- food_drink_workers[1:303]  boxplot(beforeMinWageLaw, afterMinWageLaw)  options(scipen = 100)  t.test(beforeMinWageLaw, afterMinWageLaw, mu=0,   alternative "less", conf=.95,   var.equal = FALSE, paired = FALSE)   Results:  Welch Two Sample t-test  data: beforeMinWageLaw and afterMinWageLaw  t = -36.92, df = 160.8, p-value &lt; 0.00000000000000022  alternative hypothesis: true difference in means is less than 0 95 percent confidence interval:  -Inf -38.87279  sample estimates:  mean of x mean of y  103.8059 144.5023
61610
I have a query which selects a max value over all the values of in a datetime column (dtimesent).  select dtimesent, max(ResponseTime) AS maxtime FROM allrequests  where Request like '%Requestsa%' group by dtimesent   I need to get the value of the request colmn for each max response time in each group by. Is there any easy way to do this?  There can be multiple rows per dtimesent value so I can't just include the request column in the group by,  Many thanks,