Forecasting

When information is transferred across time, often to discrete points in time, it is called forecasting. Forecast of the future using temporal behavior of data need uncertainty quantification, so we stay aware of the level of our confidence in the forecasts or point estimates.

A time series is a set of data points (indexed in time), and is most commonly a sequence of successive equally spaced out points in time.

Time-series data can be continuous → ordinary differential equations (ODEs) and stochastic differential equations (SDEs) are continuous-time models. In the figure below, the horizon or period of forecast is 3 days and we’ve 80% confidence in the forecasts.

If we were to say that the timestamp is our feature and the value at that time is our dependent variable, we have a regression problem. The observed pattern in time-series data is the level - it’s the mean value over a specific period. The behavioral patterns in the data yield what are called components of time-series.

Time-series components

1. Trend → Persistent, long-term behavior (linear or non-linear)

2. Seasonality → Regular, periodic behavior within a year. A Fourier transform of time-series helps detect seasonality in the data, basically identifies the frequency peaks and corresponding amplitudes of the time-series signal.

3. Cyclicity → Repeated or recurring behavior over more than a year

4. Residual → Erratic or irregular behavior (noise). The randomness of residual/noise makes it a ‘white noise’ (contains all frequencies). The white noise does not influence the mean value of time-series.

Then there is autocorrelation - value at a certain point in time depends on prior values. An autoregressive (AR) characteristic of time-series, the sequential dependency of observations usually introduces structured errors that are correlated temporally on previous errors or observation(s).

The autocorrelation, also called the AR component models changes in the time-series that are not explained by trend or seasonality.

Forecasting Approaches

ARIMA (auto-regressive integrated moving average) handles data with trend.

SARIMA handles data with a seasonal component.

The trend, seasonality and noise in a time series are explained by model parameter set (p,d,q), also called the order. The auto-regressive parameter is p; d is difference parameter and q is the moving average parameter. Trend is the long-term change in the mean level of observations. Seasonality is the pattern that’s periodically repeated, and noise is the random variation in the data.

A time series is additive when the ‘trend’ is linear (changes in mean and variance are at linear rate) and ‘seasonality’ is constant in time. A time series is multiplicative when the ‘trend’ is non-linear. A stationary time series has constant mean over time and does not exhibit a trend. Having no trend means identically distributed random variables.

Y(t) = Level + Trend + Seasonality + Noise

📌 A stationary time-series does not exhibit a trend.

Stationarity means that the mean and variance are constant and do not change over time. The trend is how the mean value of a time-series changes over time, the seasonality is how the mean value of a time-series changes over seasons in a year.

For a non-stationary time series, there are two approaches to model the trend and seasonality - detrending, differencing.

In detrending, use a regression model, slope of the linear regression fit is the increase/decrease of y-value with x (time). Detrending only works for linear trends.

In the figure above showing ACF and PACF, there’s a strong residual seasonal effect upon detrending. The time-series has not become fully stationary.

In differencing, use difference between two consecutive values in time. Differencing can deal with both changes in the mean (first-order difference) as well as changes in the mean movement (second-order difference).

In the figure above, there is no correlation between the time steps. Stationarity has been achieved by differencing.

📌 Partial autocorrelation is the autocorrelation without the carry-over (correlation between non-consecutive points in the time-series).

---

Heteroscedasticity happens when the standard errors of a variable, monitored over a specific period of time are non-constant.

📌 Conditional heteroscedasticity identifies non-constant volatility (degree of variation of series over time) when future periods of high & low volatility cannot be identified.

📌 Unconditional heteroscedasticity is used when future high & low volatility periods can be identified.

About time-series data ⤵️

1) The uncertainty of forecast is just as important as the (point) forecast itself.

2) Model serving (deploying & scoring) is challenging.

3) Cross-validation (with sliding or expanding window as testing strategy) is tricky.

Uncertainty in forecasts

The uncertainty determines our confidence in the forecasts. There are several tools for quantification of this uncertainty, each with their drawbacks like not having guaranteed coverage, conformal prediction however fills the gap.

Read the book by Christoph Molnar for guidance on quantifying the uncertainty of machine learning models.

---

Some time-series exhibit ill-behaved uncertainty. The forecast errors do not follow known distributions. Such information is useful for making judgmental decisions, but cannot be modeled and used for forecasting. Such an uncertainty is coconut uncertainty - of unknown unknowns leading to unpredictability.

Other time-series exhibit well-behaved uncertainty. The forecast errors follow known distributions - normal, Poisson etc.. Such information is useful for modeling and predictions bound within a certain range. The window of uncertainty is subway uncertainty - of known unknowns.

Subway uncertainty is the uncertainty of known unknowns; events can be predicted assuming a window of uncertainty. It is like, even on a worst day trying to navigate through rush hour trains, your subway voyage almost definitely won’t take you more than around 30 minutes plus planned time.

Coconut uncertainty is an allusion to a coconut unexpectedly falling on one’s head while on a beach. It is the uncertainty of unknown unknowns due to ill-behaved forecast errors; events that could never be predicted no matter how hard we tried.

---

Prediction interval and confidence interval are often used in the same context, making it important to understand how they differ. The former is usually wider than the latter. The formulae for sample size n explains why.

where, y is the dependent variable or response and x the independent variable. The standard error (se) of prediction is given by,

A confidence interval is used to predict the mean response based on regression. A prediction interval is used to predict the future value based on regression. See this example for more.

Baseline models

Forecast of level + trend is a baseline forecast. Baseline forecasts with the persistence model (using an observation at the previous time step to learn what will happen in the next time step) quickly indicate whether you can do significantly better. Baseline forecasts quickly indicate whether you can have significantly better forecasts by utilizing advanced models. If there’s no recognizable pattern, you’re probably dealing with a random walk process (future is independent of past).

The human mind is hardwired to look for patterns everywhere and we must be vigilant if we’re developing elaborate models for random walk processes. A random walk having a step size that varies according to a normal distribution is used as a model for real-world time-series data such as financial markets.

Approaches to smoothing a time-series:

1. Holt’s method - there’re two parameters viz, level smoothing constant (alpha) and trend constant (beta).

2. Holt Winter’s method - there’s seasonal smoothing constant (delta) as well; considers seasonal baseline which is a regularly recurring pattern (day, week, month, quarter etc.) and baseline rises and falls at regular intervals. Deviation of each season from the baseline’s long-term or annual average is used for forecasts.

Smoothing models are for removal of noise.

1. Moving averages are smoothing models, they can be simple, exponential, and cumulative. Moving average models are used to model the effect of random fluctuations in the time-series. We can say the influence of random fluctuations is relatively small when the autocorrelation goes toward zero.

Simple Moving Average (SMA) uses a sliding window to take the average over a set number of time periods. It is an equally weighted mean of the previous data in this period. Unlike SMA which drops the oldest observation as the new one gets added, cumulative moving average (CMA) considers all prior observations.

Exponential moving average (EMA) gives more weight to the recent observations as a result of which, EMA can better capture the movement of the trend. Exponential smoothing defines trend as the difference between observed values in consecutive records in time.

There is a difference between SimpleExpSmoothing (SES in statsmodels acts as a basic, single-parameter model for stationary data) and ExponentialSmoothing (statsmodels Holt-Winters - a comprehensive framework handling level, trend, and seasonality). SES focuses on the most recent level, while ExponentialSmoothing uses components to handle complex trends and repeating seasonal patterns.

Forecasting Libraries

📌 Darts (by Unit8)

📌 statsforecast

📌 sktime

📌 skforecast

skforecast has backtesting strategies.

📌 tslearn

📌 autoTS

Outliers or anomalies in time-series data can be point or collective (subsequent).

📌 PROPHET (by Meta/FB)

📌 puncc for conformal prediction

📌 merlion (by Salesforce)

Multivariate time-series

For multivariate time-series data, one can follow VAR or vector autoregression approach.

One can develop dynamic models to investigate short-term and long-term causal effects. Short-term causality might be observed through Granger causality, while long-term causality is typically studied using techniques like ECM or VECM.

One can utilize deep learning algorithms like LSTM (with either tensorflow or pytorch) in multivariate time-series.

Probabilistic forecasting

For probabilistic time-series modeling, one can use AutoGluon’s gluonts from AWS labs.

Chronos is a family of pretrained time series forecasting models. Chronos models are based on language model architectures, and work by quantizing time-series into buckets which are treated as tokens.

There’s also Bayesian forecasting using statsmodels.

One may as well utilize skforecast for probabilistic forecasting. It uses conformal prediction, which provides distribution-free marginal coverage guarantees. Otherwise, traditional prediction intervals rely on Gaussian assumptions or expensive sampling.

Foundation model for forecasting

TimeGPT is production ready pre-trained time-series foundation model for forecasting & anomaly detection by NIXTLA, which democratizes access to SOTA open-source tools as well as enterprise APIs for anomaly detection and forecasting.

skforecast also lets you forecast using foundation models like Chronos (by Amazon), Moirai (by Salesforce), etc..