There must be a model for what I would like to do, because I have been spending far too long with this project for an answer not to exist.  For reference, I am using R.    I have a time series object, $X$, with $n$ observations and $p$ columns (indicator variables).  I also have a separate time series object $y$ which I would like to predict future values of, which has $n$ observations and $1$ column.    I would like to feed this data into a black box, and I would like the black box to spit out $h$ predicted future values of object $y$ using learned patterns from my independent time series object X.  Of course one could find a simple linear solution using regression, but this does not account for obvious autocorrelation.  It also requires me to forecast dummy observations for $X$, which is undesirable.  I would only like to forecast one series, not all of them.  I could also use an ARIMA model, which takes care of autocorrelation but has very poor predictive ability.  I looked into using bsts-type models, but these also require additional newdata when the object has a regression component.  I have been looking into keras-based models as well, but thus far have not found a solid tutorial to teach me what I am looking to learn.    To reiterate:    Time series $X_{n \times p}$ Time series $y_{n \times 1}$ Insert $X$ and $y$ into black box Supply value $h$, the desired time steps in the future I would like to predict See $h$ predicted values of $y$ based on features the machine learned during processing.      Tutorials and/or walkthrough links would be greatly appreciated, especially if they are coded in R!    This question was tagged as a duplicate of , which I had seen earlier.  However, the answer was not satisfying to me.  I feel this question is better worded, more nuanced, and asks specifically for some type of starter-code or tutorial, or information on how to begin implementation.  , but in some regards it falls short (it does not explain how to accomplish far-in-the-future prediction).  The theory of machine learning is interesting for all of us, but as noted in ,     Considering the complexity of financial time series, combining deep learning with financial market prediction is regarded as one of the most charming topics. However, this field still remains relatively unexplored.   In posing this question, I am hoping to assist other beginners with the same problem they face when attempting to incorporate time series forecasting with deep learning.
91053
I'm working on a project to predict the usage of all the files(rough frequency of usage) in a filesystem (a company server on which 100s of company employees are active) in near future (say the next 1 or 1.5 months) based on the metadata of the file system for past 6 months. I've got the following attributes about the files with me :   The temporal sequence of file usage for last 6 months(whenever the file was read/written/modified and by whom). All the users who are on the server and can access the files. Last modified/written/read epoch time and by whom. File creation epoch time and by whom. Any compliance regulations on the file(whether the file contains any confidential data). Size, name, extension, version, type of the file. The number of users who can access the file. File path. The total number of times accessed. Permitted users.   Now, I plan to use LSTM but for standard LSTMs, the input is temporal sequence only. However, all the attributes that I have seem significant in predicting the future usage of the file.   How should I also make use of the attributes of the file that I have? Should I train a Feedforward Neural Network, disregarding the fact that it usually fails on temporal sequences? How should I proceed? Does a variant of LSTM exist that can take into account the attributes of the file as well and predict the usage of the file in near future? Do I need to use MLP and LSTM together like a hybrid?
9024
I have a question related to modeling short time-series. It is not a question , but how. What method would you recommend for modeling (very) short time-series (say of length $T \leq 20$)? By "best" I mean here the most robust one, that is the least prone to errors due the fact of limited numbers of observations. With short series single observations could influence the forecast, so the method should provide a cautious estimate of errors and possible variability connected to the forecast. I am generally interested in univariate time-series but it would be also interesting to know about other methods.
10661
I am trying to forecast time series data for next $n$ time periods and I am using various forecasting techniques like simple moving average, exponential smoothing (Single,double,triple) and auto.arima().  I am getting equal forecasting values for simple moving average and single exponential smoothing and at times, even auto.arima() function in R is giving me the same forecast values.  Here is the code that I am trying to use  mod_mva &lt;- SMA(data.train,n=2) forecast_mva &lt;-forecast(mod_mva,ts_freq)   Here is the output:
83259
this is my first ever question on a website i use frequently!  This time series has given me much trouble over the last couple of days even after extensive googling, I suppose with TS theres no two series that look the same, hence my confusion!   I have the following ACF and PACF plots for my data:    After modelling NUMEROUS models in R (ARMA(p,q), with all combinations of p=0,1,2,3,4 and q=0,1,2,3,4) and looking at their theoretical acf plots, it seems R's auto.arima gave the best model with p=q=2.   But in my work, i don't want to just say i used R and inferred nothing from the ACF plots haha!  I understand that the slightly significant PACF at lag 1 indicates an AR component(?) and that as both plots 'slope off' it suggests a ARMA(p,q) model, but is there anything else im missing????  Also; the ts (in my eyes) seems seasonal, however ironically, no amount of differencing makes a difference, the plot still looks the same no matter what d equals. (R also tells me its not seasonal)  In short:   What does the ACF plots tell me (main) Am i right in not differencing the data   Thanks a bunch!!
103290
Ok, I have a logistic regression and have used the predict() function to develop a probability curve based on my estimates.   ## LOGIT MODEL: library(car) mod1 = glm(factor(won) ~ as.numeric(bid), data=mydat, family=binomial(link="logit"))  ## PROBABILITY CURVE: all.x &lt;- expand.grid(won=unique(won), bid=unique(bid)) y.hat.new &lt;- predict(mod1, newdata=all.x, type="response") plot(bid&lt;-000:1000,predict(mod1,newdata=data.frame(bid&lt;-c(000:1000)),type="response"), lwd=5, col="blue", type="l")   This is great but I'm curious about plotting the confidence intervals for the probabilities. I've tried plot.ci() but had no luck. Can anyone point me to some ways to get this done, preferably with the car package or base R.
117134
I am trying to figure out how R computes lag-k autocorrelation (apparently, it is the same formula used by Minitab and SAS), so that I can compare it to using Excel's CORREL function applied to the series and its k-lagged version. R and Excel (using CORREL) give slightly different autocorrelation values.  I'd also be interested to find out whether one computation is more correct than the other.
100160
We have three years of data for online visits at a daily level. We want to forecast the daily visits for the next 90 days. What would be the best method to capture weekday seasonality , holiday seasons, and also the drift.  Can this be successfully done in R? We are currently using R. We have considered ARIMA but it does not capture seasonality.  While converting the data to a time series in R,  what should be the "frequency"?  Should we use ARIMA with regressors?
38126
How do you estimate the appropriate forecast model for a time series by visual inspection of the ACF and PACF plots? Which one (i.e., ACF or PACF) tells the AR or the MA (or do they both)? Which part of the graphs tell you the seasonal and non-seasonal part for a seasonal ARIMA?  Consider the ACF and PCF functions displayed below.  They are from a log transformed series that's been differenced twice, one simple difference and one seasonal (, ).  How would you characterize the series? What model best fits it?
23122
I am familiar with "regular" cross-validation, but now I want to make timeseries predictions while using cross-validation with a simple linear regression function. I write down a simple example, to help clarify my two questions: one about the train/test split, one question about how to train/test for models when the aim is to predict for different n, with n the steps of prediction, in advance.  (1) The data  Assume I have data for timepoints 1,...,10 as follows:  timeseries = [0.5,0.3,10,4,5,6,1,0.4,0.1,0.9]   (2) Transforming the data into a format useful for supervised learning  As far as I understand, we can use "lags", i.e. shifts in the data to create a dataset suited for supervised learning:  input = [NaN,0.5,0.3,10,4,5,6,1,0.4,0.1] output/response = [0.5,0.3,10,4,5,6,1,0.4,0.1,0.9]   Here I have simply shifted the timeseries by one for creating the output vector. As far as I understand, I could now use input as the input for a linear regression model, and output for the response (the NaN could be approximated our replaced with a random value).  (3) Question 1: Cross-validation ("backtesting")  Say I want to do now 2-splits, do I have to shift the train as well as the test sets?    I.e. something like:  Train-set:  Independent variable: [NaN,0.5,0.3,10,4,5]  Output/response variable:[0.5,0.3,10,4,5,6]  Test-set:  Independent variable: [1,0.4,0.1]  Output/response variable:[0.4,0.1,0.9]  (ii) Question 2: Predicting different lags in advance:  As obvious, I have shifted dependent to independent variables by 1. Assuming now I would like to train a model which can predict 5 time steps in advance -- can I keep this lag of one, and nevertheless use the model to predict n+1,...,n+5,... or do I change the shift from independent to dependent variable to 5? What exactly is the difference?
20343
My final goal is to generate Impulse Response Functions in R.  I have variables that are non stationary when I set k = 5 in a Unit Root test, and they are cointegrated which to my understanding prompts the use of the VECM, from which the Vec2Var argument is used to then generate IRFs. However, my response functions from this methodology do not decay over time and mostly do not revert to the zero line.  Furthermore, I noticed that when I input first differenced variables into the VECM as opposed to level data above, response function do revert to the zero line. Therefore my first question is: 1) is it appropriate to use differenced variables in a VECM model?  Secondly, as an alternative I am considering restricting the number of lags used in the Unit Root tests so that the variables are not all non-stationary; in this case 2) would the use of VECM be void and the VAR with first differences be a more appropriate model? Again, using differenced data is giving me better response functions that revert to zero in the long run. However, 3) is it ok to use the VAR for stationary/non-stationary data in levels where these variables are still cointegrated?  Also, I am using growth rate variables, 4) should I still long transform all variables and use logs in all tests?  Thank you for the help!
98253
I have a multivariate time series data set having 6 variables. I have to predict the sixth variable at the next time step given the expected values of other five at the next time step. I am using LSTM for prediction. The data set is small (170*6), have 170 observations. Splitting the data set into three i.e. train/test/split doesn't seem a probable option in this case.   As far as I know, input sequences (of fixed length) over which the LSTM learns are assumed independent. So my question was, can one sequence be left out for validation just like in the case of a cross sectional data problem.
18394
Can you give some real-life examples of time series for which a moving average process of order $q$, i.e. $$ y_t = \sum_{i=1}^q \theta_i \varepsilon_{t-i} + \varepsilon_t, \text{ where } \varepsilon_t \sim \mathcal{N}(0, \sigma^2)   $$ has some a priori reason for being a good model? At least for me, autoregressive processes seem to be quite easy to understand intuitively, while MA processes do not seem as natural at first glance. Note that I am not interested in theoretical results here (such as  or invertibility).  As an example of what I am looking for, suppose that you have daily stock returns $r_t \sim \text{IID}(0, \sigma^2)$. Then, average weekly stock returns will have an MA(4) structure as a purely statistical artifact.
47868
I am surprised how often the auto.arima function from the "forecast" package in R returns straight linear forecasts when there appears to be fairly strong seasonality in the historic data.  I've attached an example:      tpits=ts(TPI,start=c(1986,1),end=c(2020,1),frequency=1)     fit=auto.arima(tpits)   The data is annual and varies significantly but the forecast is completely linear.  What am I doing wrong?
50188
So for the last few months I've been doing a lot of forecasting for my company and specifically I've been looking at monthly forecasts of total weight of different categories of products output's each month.  I've been using time series models such as Arima, ETS, and tslm() within R to do my forecasting, as well as using cross validation to select a model. Over the last two days I've been presenting my results and discussing implementation of my forecasts.  But, I've been asked the same question multiple times and I don't know the answer to it, so let me ask you guys.  If this has been asked before, I apologize.  I'll write out a few questions that hopefully will make clear what I'm trying to understand.  Also, I'd like to keep technical answers about the models in the context of R, since that's what I'm using.   Do time series models take into account the number of days in a month? Particularly, do time series models consider the number of business days in a month? (or is there a way to incorporate this?) Do we even need to worry about this when forecasting using a time series model or does the model account for this?   For instance, lets say in October of 2014 a certain category sold 35,000 lbs of a product, and that there were 31 days, 23 of which were business days.  Well for this year, 2015, there are still 31 days, but now there are only 22 business days.  Just some background on the data, I have monthly data that starts in August of 2008.    So would it possibly be better to average the weight per month with the number of business days and forecast out this way since the # of business days change month to month and year to year?
90364
I'm using the logistf package in R to perform Firth logistic regression on an unbalanced dataset. I have a logistf object:  fit = logistf(a~b)   Is there a predict() function like on that's used in the lm class to predict probabilities for future data points? Or do I have to manually input the estimated parameters from the Firth regression.
105865
Assume a p x n data matrix  (e.g. p independent samples of n variables). The n variables are correlated according to a correlation matrix $\rho$. Now if you consider the data matrix as some state t (t=time)  and based on this state I would like to calculate the next timestep (t+1). Therefore the correlation matrix would be allowed to have values &lt; 1 in the diagonal, which means that the diagonal elements would indicate the influence from each variable $x_{p,i}[t]$ on $x_{p,i}[t+1]$. So I would be interested to find a function f like this: $x_{p,i}[t+1] = f(x_{p,i}[t],\rho)$ so that the p x n data matrix  consisting of the $x_{p,i}[t+1]$ replicates the $\rho_{ij}$ (except the diagonal element). f should be something like this: $x_{i,j}[t+1] = \sum_{(k=1\ldots n)}\rho_{i,k} \cdot x_{kj}[t] + e_{i,j}.$ This means specifically, I not only want to calculate a data matrix, that replicates the correlation matrix as such, but also to let each observation be depending on its own state and linear dependent on the previous state of all the intercorrelated variables of its own state. For ease one can assume that the p variables are distributed normal and $e_{ij}$ are independent and distributed with mean$(e_{i,j}) = 0 $. Then for a numerical simulation it would be interesting  to find a formula to calculate the variance/standard deviation of each of the j variables (sd($e_{[.,j]} = ?).$ Thank you for any help!
27355
Let's take the following example:  set.seed(342) x1 &lt;- runif(100) x2 &lt;- runif(100) y &lt;- x1+x2 + 2*x1*x2 + rnorm(100) fit &lt;- lm(y~x1*x2)   This creates a model of y based on x1 and x2, using a OLS regression.  If we wish to predict y for a given x_vec we could simply use the formula we get from the summary(fit).  However, what if we want to predict the lower and upper predictions of y? (for a given confidence level).  How then would we build the formula?