Skip to content

About

Multi-Agent Prompting with In-Context Feedback for Cross-Modal Urban Event Chain Prediction

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Multi-Agent Prompting with In-Context Feedback for Cross-Modal Urban Event Chain Prediction

Four domain-specialised LLM agents — weather, traffic, social behaviour, public safety — each see only their own sensor modality and predict how a trigger event cascades through 311 complaints, traffic disruption and crime incidents over the following seven days. A coordinator agent merges their predictions into a single event chain with a strength label and a calibrated p_strong.

Methodology overview


What is in this package

Code and data for the experiments reported in the manuscript. The directory runs as-is: every script resolves data/ as a sibling of the code packages.

Path Contents
prompts/ The method: 4 domain agents, the merger, the feedback block, the response schema, JSON extraction
agents/ LangGraph pipeline — state, nodes, graph builder, the 5 configuration strategies, experiment runner, result writer
pipeline/ Chain-label derivation and the LLM backend abstraction (AWS Bedrock + Ollama)
evaluation/ Metrics (Event-Chain F1, Brier, ECE, Wilson CI, MCC), paired significance tests, and the 7 baselines
scripts/ run_agents.py (one configuration), run_all.py (full sweep), run_baselines.py
tests/ 71 tests over prompt composition and the agent pipeline
data/ Labelled chains, splits, aligned sensor tables, and every result the manuscript reports

The data

Chicago NYC San Francisco
Labelled chains 209 205 207
of which strong 88 77 86
Test chains 36 17 32
Daily observations 1,096 1,096 1,096

621 labelled event chains (251 strong), 85 in the test split. Splits are chronological 80/10/10 per city.

Path What it is
data/chains/{city}_chains.jsonl The 621 labelled chains — the ground truth every metric is computed against
data/splits/ Chronological train/val/test manifests per city
data/{city}/{city}_daily_aligned.csv The daily aligned sensor table the prompts and baselines read
data/results/{model}-bedrock-{stamp}/ The nine-model grid: 3,825 per-chain predictions, all 3,825 scored
data/results/baseline/ 7 baselines × 3 cities, per-chain records plus summaries

Hypotheses

The manuscript asks one question — can frozen, instruction-tuned LLMs predict cross-modal urban event chains without any training, and does in-context feedback improve that — and decomposes it into three hypotheses plus an absolute-skill check.

ID Name What it asks How it is tested
H1 Decomposition Does splitting the task across four domain-specialised agents lower prediction error? Primary: the four-agent ensemble against one agent given all four sensor streams, which holds information constant. Secondary: against a single-domain agent — reported, but it confounds decomposition with information access. Both leave a difference in inference budget (5 calls vs 1).
H2 Self-consistency feedback Does a second round whose prompts carry round-one self-signals — inter-agent disagreement, parse failures, uniformly low confidence — improve the prediction? multi_feedback against multi_no_feedback, paired within model. On the held-out test split no ground-truth error signal can be supplied without leakage, so this is the only feedback regime evaluated.
H3 Cross-domain interference (reported control) Does giving one agent all four modalities degrade it relative to a clean single-domain agent? noisy_single against single_agent. Reported because the ablation falls out of the implemented pipeline at no extra model calls — it is not advanced as a contribution.
— Absolute skill Does any configuration beat the trivial predictor that always answers the test-set majority class, and does any beat a per-city climatology forecast under the Brier score? Every configuration and all seven baselines, scored by one shared code path.

Each hypothesis is tested within model by McNemar's exact test and aggregated across the nine models by a sign test, so no single model can carry a conclusion.

Experiment configurations

Configuration Hypothesis Role LLM calls / chain What it isolates
single_agent H1 / H3 control 1 One specialist (weather), merger bypassed
multi_agent H1 treatment 5 Four specialists + coordinator
noisy_single H1 / H3 treatment 1 One agent given all four streams
multi_no_feedback H2 control 5 Ensemble, single pass
multi_feedback H2 treatment 10 Two rounds; round 2 carries the feedback block

multi_no_feedback and multi_agent are the same pipeline, run as separate arms so the feedback comparison is against a control executed under identical conditions. noisy_single serves both H1 (as the information-matched comparison) and H3 (as the interference treatment).

The feedback block carries no ground truth

Two regimes are recorded on every result record. error (train and validation only) tells round two plainly that round one was wrong and what the outcome was. self_consistency (test, and anything unrecognised, so it fails safe) carries agent disagreement, unparseable output and uniformly low confidence — self signals only.

Every feedback result shipped here is in the self_consistency regime, because every result is on the test split. That is the intrinsic self-correction setting in which Huang et al. report little or no benefit, so the null result is the predicted outcome and says nothing about the error regime, which is not evaluated.

All experiments run at the expert prompt level, and only that level is included here: 135 of 135 shipped result cells are expert.

Running it

make install                       # venv + requirements
cp .env.example .env               # then set SENSOR_BACKEND and credentials

make test                          # 71 tests
make baselines                     # 7 baselines x 3 cities
make agent-test                    # 5-chain smoke test

make agent-single CITY=chicago     # one configuration, one city
make agent-all MODEL=<bedrock-id> PROVIDER=bedrock DRY_RUN=1   # cost it first
make agent-all MODEL=<bedrock-id> PROVIDER=bedrock

Results are written to data/results/{run}/{city}/{config}/expert.jsonl with a .meta.json manifest beside each one. The runner refuses to overwrite an existing path, so a repeated run id fails loudly rather than destroying data.

Requirements

Python 3.12+; packages in requirements.txt. AWS credentials for the Bedrock backend, or a local Ollama install for the free tier.

About

Multi-Agent Prompting with In-Context Feedback for Cross-Modal Urban Event Chain Prediction

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Used by

Contributors

Languages