Agent Harness Paper

This private repository contains the manuscript and supporting artifacts for:

Thin Harness, Strong Contracts: Production-Oriented Agent Harnesses for Stateful AI Agents

The project studies agent harnesses as production infrastructure: runtime control, evaluation, observability, permission policy, replay, audit, and human review for stateful tool-using agents.

Status And License

This is a submission-ready, non-anonymous arXiv-style preprint artifact package for Song Luo. It uses the standard LaTeX article class rather than a conference proceedings template. The public artifact archive is published on Zenodo at https://doi.org/10.5281/zenodo.20907471 while the working GitHub repository remains private unless separately opened.

The repository is dual-licensed. Code is under the MIT License; the manuscript source, documentation, benchmark cases, evidence snapshots, derived tables, figures, and release metadata are under Creative Commons Attribution 4.0 International (CC BY 4.0). See LICENSE, LICENSE-MIT, and LICENSE-CC-BY-4.0.md.

Repository Layout

paper/          LaTeX manuscript and bibliography
paper/metadata.json  submission metadata
benchmark/      24 benchmark cases for routing, state, security, memory, replay
harness/        small replayable harness and provider-independent evaluation logic
scripts/        experiment, analysis, checks, and utility entrypoints
results/        regenerated tables and summaries
evidence/       artifact inventory and generated GitHub evidence snapshots
docs/           protocol, claim boundaries, reproducibility, reviewer guide
notes/          argument map, outline, related work notes
tests/          deterministic unit tests

Reproducibility

Single-command validation:

python scripts/run_checks.py --mode validate
python scripts/run_checks.py --mode regenerate

python -m pytest
python scripts/run_local_reference.py
python scripts/analyze_results.py
python scripts/analyze_uncertainty.py
python scripts/generate_figures.py
python scripts/collect_github_evidence.py
python scripts/audit_source_evidence.py --offline
python scripts/backfill_run_metadata.py
python scripts/check_tex_sources.py
python scripts/check_public_traces.py
python scripts/check_artifact_consistency.py
python scripts/secret_scan.py

Live model runs require environment variables and do not store credentials:

$env:DEEPSEEK_API_KEY="..."
python scripts/check_provider_credentials.py --provider deepseek
python scripts/run_model_experiment.py --provider deepseek --limit 24 --output logs/live-reruns/deepseek_full_smoke.jsonl

$env:MOONSHOT_API_KEY="..."
python scripts/check_provider_credentials.py --provider kimi
python scripts/run_model_experiment.py --provider kimi --limit 24 --output logs/live-reruns/kimi_full_smoke.jsonl

The live runner refuses to overwrite an existing output unless --resume or --force-overwrite is supplied. Use ignored logs/live-reruns/ paths for smoke tests before promoting any new raw file into experiments/raw/.

Parser-instrumentation repair for already recorded live rows also requires the relevant provider credential:

python scripts/rerun_parse_errors.py --provider deepseek

The paper should cite only regenerated results whose raw JSONL is present under experiments/raw/ and whose aggregate tables are regenerated by scripts/analyze_results.py.

PDF builds are handled by .github/workflows/latex.yml when GitHub Actions is available. Local PDF builds require pdflatex and bibtex. Artifact validation is handled by .github/workflows/validate.yml.

For review, start with docs/reviewer_guide.md and artifact_manifest.json.

For release state and final checks, see docs/artifact_release_index.md, docs/submission_readiness.md, docs/arxiv_submission.md, docs/zenodo_release_plan.md, CHANGELOG.md, and docs/final_cleanup.md. Submission, license, anonymization, artifact-sharing, and live-rerun decisions are tracked in docs/external_decision_register.md.