[ENH] Agentic Model Selection Benchmark: LLM vs AutoML baselines #9846

Summary

There are currently 5+ competing LLMForecaster implementations being developed for #9721, plus the Forecasting Autopilot proposal in #9841. However, there is no standardized benchmark to evaluate whether LLM-based model selection actually outperforms existing AutoML baselines — or under what conditions it does.

This issue proposes a benchmark framework to objectively compare LLM-driven forecasting pipelines against deterministic baselines on standard datasets.

Motivation

• [ENH] LLM-based forecasting agent #9721 has 5 open PRs with different LLM-based approaches (registry-based, RAG-enhanced, agent-translation, multi-provider), but no way to compare them
• [ENH] : Forecasting Autopilot — An Agentic Layer for Automatic Pipeline Selection, Evaluation & Optimization in sktime #9841 proposes a full agentic pipeline but evaluates only on the airline dataset
• Without a benchmark, design decisions (orchestration vs MCP tool-calling, prompt templates, candidate pool size) are based on intuition rather than evidence

Proposed approach

Datasets: M3 (monthly subset, 1428 series), M4 (monthly subset), Tourism (quarterly)

Baselines:

• AutoETS (sktime)
• AutoARIMA (sktime)
• MultiplexForecaster with grid selection (sktime)
• NaiveForecaster (seasonal, last value)

LLM candidates:

• LLMForecaster variants from [ENH] LLM-based forecasting agent #9721 PRs (once any are merged)
• Forecasting Autopilot ([ENH] : Forecasting Autopilot — An Agentic Layer for Automatic Pipeline Selection, Evaluation & Optimization in sktime #9841) when available
• Any sktime-mcp-based agentic pipeline

Metrics per series:

• Forecast accuracy: MASE, sMAPE
• Selection quality: % of series where LLM picks the best-performing model vs AutoML
• Cost: wall-clock time, LLM API token count, number of tool calls
• Robustness: variance across 3 runs (temperature=0 vs temperature>0)

Implementation sketch:

benchmarks/agentic/
├── config.yaml          # datasets, baselines, LLM backends, metrics
├── run_benchmark.py     # main entry point
├── baselines.py         # AutoETS, AutoARIMA, MultiplexForecaster wrappers
├── llm_candidates.py    # LLMForecaster / Autopilot wrappers
├── evaluate.py          # temporal CV + metric computation
└── report.py            # results tables + visualizations

Temporal CV via ExpandingWindowSplitter with initial_window=0.6*len(series), step_length=1.

Output

A results table like:

Scope

This could be:

• A standalone benchmarks/agentic/ directory in sktime
• A notebook in the sktime-mcp repo
• Part of an ESoC project

I'm happy to implement the baseline framework + evaluation harness as a first step. The LLM candidate wrappers would depend on which #9721 PRs get merged.

Related issues

• [ENH] LLM-based forecasting agent #9721 — LLM-based forecasting agent (5 competing PRs)
• [ENH] : Forecasting Autopilot — An Agentic Layer for Automatic Pipeline Selection, Evaluation & Optimization in sktime #9841 — Forecasting Autopilot (LangGraph-based)
• [ENH] Two Statistical Diagnostic MCP Tools sktime-mcp#96 — diagnostic MCP tools (my PR feat: add detect_seasonality and check_structural_break diagnostic tools (#96) sktime-mcp#114 adds pre-modeling diagnostics that could feed into the benchmark's LLM candidates)

Would the maintainers be open to this direction? Happy to scope it down or adjust the dataset/metric choices.