Adaptive Forecasting Under Distribution Shift

Regime transitions routinely break stationarity in time series — hidden switches between latent operating modes change level, trend, volatility, or seasonality. This makes calibrated uncertainty as important as point accuracy: predictive intervals can badly under- or over-cover right at a regime boundary even when point error stays small, because classical approaches (Gaussian processes with stationary kernels, Bayesian models with smooth priors) implicitly assume a differentiability or stationarity that fails exactly there.

Lead researcher and implementer, as part of my PhD work. I designed the coupling of a Deep Switching State-Space Model (DS³M) with Adaptive Conformal Inference (ACI) and its aggregated variant (AgACI), built a unified residual-based conformal wrapper applied consistently across four baseline forecasters, and ran the experiments — culminating in a NeurIPS 2025 Workshop (BERT²S) paper.

1. 01DS³M forecaster: discrete Markov regimes dₜ, continuous latent states zₜ, and a learned history summary hₜ produce a one-step mean forecast ŷₜ and residual score sₜ = |yₜ − ŷₜ|.
2. 02Adaptive Conformal Inference (ACI): maintains a target miscoverage level αₜ, updated online after every observation (αₜ₊₁ = αₜ + γ(α − 1{yₜ ∉ Ĉₜ})), so intervals widen after misses and narrow after hits — no exchangeability assumption required.
3. 03Aggregated ACI (AgACI): runs a small set of ACI experts at different learning rates γ and combines their quantiles, removing the need to hand-tune γ.
4. 04A unified, model-agnostic conformal wrapper applies the same residual-based calibration on top of four different backbones — S4 (S4D encoder → blocks → decoder), change-point detection (CPD) with a light GRU refit per segment, MC-Dropout GRU, and a sparse Gaussian process (GPyTorch, inducing points) — so every baseline is calibrated the same way for a fair comparison.

• DS³M over kernel- or prior-based uncertainty

  Gaussian processes with stationary kernels and Bayesian models with smooth priors encode nonstationarity implicitly and assume differentiability that breaks at regime boundaries. DS³M instead models discrete regime switches explicitly while keeping continuous within-regime dynamics, which is both more interpretable and more robust right at a transition.

• ACI/AgACI over vanilla conformal prediction

  Vanilla conformal prediction is distribution-free but relies on exchangeability, which regime shifts violate outright. ACI relaxes this by correcting the target miscoverage online; AgACI further removes sensitivity to the single learning-rate choice by aggregating several experts.

• One wrapper, four backbones

  Rather than give each baseline (S4, CPD, MC-Dropout GRU, Sparse GP) a bespoke uncertainty method, the same residual-based conformal layer is applied to all of them. This isolates the effect of the calibration layer itself and makes the cross-model comparison fair.

• Centered, absolute-residual intervals by default

  Intervals are centered on ŷₜ using absolute residual scores — the simplest model-agnostic choice. Studentized or variance-aware scores would better handle skewed/heteroskedastic noise, and are left as explicit future work rather than added prematurely.

Evaluated with one-step rolling forecasts on held-out windows across three datasets: Lorenz (synthetic, chaotic, T=10,000), Sleep Apnea (real, 2Hz, T=2,000), and US Unemployment (real, monthly, T=879). Each series is standardized on training statistics with predictions inverse-transformed before scoring. Metrics are RMSE, empirical Coverage@90%, and median interval length, compared across five forecasters (CPD, Sparse GP, MC-Dropout GRU, S4, DS³M) all wrapped with the same conformal layer.

• Coverage guarantees are marginal, not conditional — they don't hold within specific regimes or covariate groups, which matters for high-stakes use.
• Absolute residuals give symmetric bands, which is inefficient under skewed or heteroskedastic noise.
• The calibration buffer introduces a short lag before intervals adapt right after a sudden regime shift.
• Comparing median interval width across datasets of very different scales can distort efficiency comparisons — it sometimes exaggerates interval size for scale-sensitive models.
• The evaluation is one-step-ahead only; it doesn't fully examine native probabilistic forecast quality or decision-oriented performance under asymmetric costs. Regime-aware conformal methods that explicitly condition on the inferred regime are the natural next step.

• PyTorch
• Deep Switching State-Space Models
• Adaptive Conformal Inference
• S4
• GPyTorch
• Time Series Forecasting