Four domain-specialised LLM agents — weather, traffic, social behaviour, public
safety — each see only their own sensor modality and predict how a trigger
event cascades through 311 complaints, traffic disruption and crime incidents
over the following seven days. A coordinator agent merges their predictions into
a single event chain with a strength label and a calibrated p_strong.
Code and data for the experiments reported in the manuscript. The directory runs
as-is: every script resolves data/ as a sibling of the code packages.
| Path | Contents |
|---|---|
prompts/ |
The method: 4 domain agents, the merger, the feedback block, the response schema, JSON extraction |
agents/ |
LangGraph pipeline — state, nodes, graph builder, the 5 configuration strategies, experiment runner, result writer |
pipeline/ |
Chain-label derivation and the LLM backend abstraction (AWS Bedrock + Ollama) |
evaluation/ |
Metrics (Event-Chain F1, Brier, ECE, Wilson CI, MCC), paired significance tests, and the 7 baselines |
scripts/ |
run_agents.py (one configuration), run_all.py (full sweep), run_baselines.py |
tests/ |
71 tests over prompt composition and the agent pipeline |
data/ |
Labelled chains, splits, aligned sensor tables, and every result the manuscript reports |
| Chicago | NYC | San Francisco | |
|---|---|---|---|
| Labelled chains | 209 | 205 | 207 |
| of which strong | 88 | 77 | 86 |
| Test chains | 36 | 17 | 32 |
| Daily observations | 1,096 | 1,096 | 1,096 |
621 labelled event chains (251 strong), 85 in the test split. Splits are chronological 80/10/10 per city.
| Path | What it is |
|---|---|
data/chains/{city}_chains.jsonl |
The 621 labelled chains — the ground truth every metric is computed against |
data/splits/ |
Chronological train/val/test manifests per city |
data/{city}/{city}_daily_aligned.csv |
The daily aligned sensor table the prompts and baselines read |
data/results/{model}-bedrock-{stamp}/ |
The nine-model grid: 3,825 per-chain predictions, all 3,825 scored |
data/results/baseline/ |
7 baselines × 3 cities, per-chain records plus summaries |
The manuscript asks one question — can frozen, instruction-tuned LLMs predict cross-modal urban event chains without any training, and does in-context feedback improve that — and decomposes it into three hypotheses plus an absolute-skill check.
| ID | Name | What it asks | How it is tested |
|---|---|---|---|
| H1 | Decomposition | Does splitting the task across four domain-specialised agents lower prediction error? | Primary: the four-agent ensemble against one agent given all four sensor streams, which holds information constant. Secondary: against a single-domain agent — reported, but it confounds decomposition with information access. Both leave a difference in inference budget (5 calls vs 1). |
| H2 | Self-consistency feedback | Does a second round whose prompts carry round-one self-signals — inter-agent disagreement, parse failures, uniformly low confidence — improve the prediction? | multi_feedback against multi_no_feedback, paired within model. On the held-out test split no ground-truth error signal can be supplied without leakage, so this is the only feedback regime evaluated. |
| H3 | Cross-domain interference (reported control) | Does giving one agent all four modalities degrade it relative to a clean single-domain agent? | noisy_single against single_agent. Reported because the ablation falls out of the implemented pipeline at no extra model calls — it is not advanced as a contribution. |
| — | Absolute skill | Does any configuration beat the trivial predictor that always answers the test-set majority class, and does any beat a per-city climatology forecast under the Brier score? | Every configuration and all seven baselines, scored by one shared code path. |
Each hypothesis is tested within model by McNemar's exact test and aggregated across the nine models by a sign test, so no single model can carry a conclusion.
| Configuration | Hypothesis | Role | LLM calls / chain | What it isolates |
|---|---|---|---|---|
single_agent |
H1 / H3 | control | 1 | One specialist (weather), merger bypassed |
multi_agent |
H1 | treatment | 5 | Four specialists + coordinator |
noisy_single |
H1 / H3 | treatment | 1 | One agent given all four streams |
multi_no_feedback |
H2 | control | 5 | Ensemble, single pass |
multi_feedback |
H2 | treatment | 10 | Two rounds; round 2 carries the feedback block |
multi_no_feedback and multi_agent are the same pipeline, run as separate
arms so the feedback comparison is against a control executed under identical
conditions. noisy_single serves both H1 (as the information-matched
comparison) and H3 (as the interference treatment).
Two regimes are recorded on every result record. error (train and validation
only) tells round two plainly that round one was wrong and what the outcome was.
self_consistency (test, and anything unrecognised, so it fails safe) carries
agent disagreement, unparseable output and uniformly low confidence — self
signals only.
Every feedback result shipped here is in the self_consistency regime,
because every result is on the test split. That is the intrinsic self-correction
setting in which Huang et al. report little or no benefit, so the null result is
the predicted outcome and says nothing about the error regime, which is not
evaluated.
All experiments run at the expert prompt level, and only that level is
included here: 135 of 135 shipped result cells are expert.
make install # venv + requirements
cp .env.example .env # then set SENSOR_BACKEND and credentials
make test # 71 tests
make baselines # 7 baselines x 3 cities
make agent-test # 5-chain smoke test
make agent-single CITY=chicago # one configuration, one city
make agent-all MODEL=<bedrock-id> PROVIDER=bedrock DRY_RUN=1 # cost it first
make agent-all MODEL=<bedrock-id> PROVIDER=bedrockResults are written to data/results/{run}/{city}/{config}/expert.jsonl with a
.meta.json manifest beside each one. The runner refuses to overwrite an
existing path, so a repeated run id fails loudly rather than destroying data.
Python 3.12+; packages in requirements.txt. AWS credentials for the Bedrock
backend, or a local Ollama install for the free
tier.
