The Hidden Risks of Using Decoder-Only Models for Time Series Forecasting

As large language models (LLMs) like GPT continue to impress across a spectrum of tasks, it’s tempting to apply them everywhere—even to challenges like time series forecasting. Driven by their ability to handle complex temporal patterns at scale, decoder-only models are experiencing an explosion of use across diverse domains, including finance, healthcare, energy, and agriculture. At first glance, their ability to generate coherent sequences one token at a time seems promising for predicting future values in a data series. But there’s a subtle risk beneath the surface that traditional machine learning (ML) methods for time series forecasting manage more effectively.

Autoregressive Error Accumulation

Decoder-only models such as GPT produce outputs token by token, each conditioned on all previously generated outputs. In a forecasting context, this means every predicted data point sets the stage for the next. A small error early on—say, a mild underestimation of tomorrow’s sales—can compound over time, guiding the model into a trajectory that increasingly deviates from reality. With no native mechanism to correct past decisions, the inaccuracies can snowball. In practical terms, this can transform initially minor miscalculations into severely flawed long-term forecasts.

Heuristics, Not Guarantees

Strategies like temperature tuning or top-k sampling can reduce wild predictions, but they don’t provide any mathematical assurance that the forecast will remain stable and accurate over multiple steps. GPT and its variants lack an intrinsic verification loop. Once an error slips in, the model tends to commit to it, leading the subsequent predictions further astray.

The Traditional ML Edge

Traditional machine learning models like ARIMA, XGBoost, and LSTMs are not affected by autoregressive error accumulation because they predict future values directly from input data rather than iteratively chaining outputs. Instead, their primary source of error stems from their ability to capture the underlying data patterns, such as seasonality, trends, or noise. For example, ARIMA relies on fixed mathematical relationships, XGBoost maps historical features directly to forecasts, and LSTMs use input context in a single forward pass. This direct prediction approach ensures stability, with errors primarily tied to the model's representation of the input data rather than propagation through sequential dependencies.

Mitigating Risks for Decoder Models Over Long Horizons

To mitigate the risks of autoregressive error accumulation over longer time horizons, decoder models like TimeGPT employ techniques such as fine-tuning on domain-specific data, leveraging transformer-based attention mechanisms to capture long-term dependencies, and using strategies like temperature control or beam search to guide stable outputs. Additionally, hybrid approaches, where decoder models are combined with traditional forecasting methods, can improve robustness by blending generalization with domain-specific accuracy. However, these strategies only reduce the risk; they do not fully eliminate error propagation. As predictions remain sequentially dependent, any small error can still compound, making the challenge inherent to autoregressive modeling unavoidable over extended forecasts.

Conclusion

Decoder-only models shine at generating human-like text and handling unstructured queries. But when it comes to time series forecasting, their autoregressive design can amplify small errors over time, making them risky tools in scenarios where accuracy and long-term stability are paramount. Traditional forecasting methods or models explicitly designed for numeric sequences often provide more reliable, controlled results—ensuring that a small slip in day one’s forecast doesn’t derail your entire long-term projection.