Skip to content

INTEGRATIONS.md

Latest commit

 

History

History
141 lines (111 loc) · 7.7 KB

INTEGRATIONS.md

File metadata and controls

141 lines (111 loc) · 7.7 KB
Raw

Integrations — a signed receipt of your eval or test run, automatically

proofbundle can auto-emit a signed, offline-verifiable receipt of an inspect_ai eval or a pytest run via each framework's native plugin API. Both are opt-in: they emit only when you explicitly turn them on (the PROOFBUNDLE_EMIT=1 environment variable, or a flag) — never silently, and never failing your run.

inspect_ai (end-of-task hook)

pip install "proofbundle[inspect,eval]"
PROOFBUNDLE_EMIT=1 PROOFBUNDLE_THRESHOLD=0.9 inspect eval task.py --model mockllm/model

Installed via the inspect_ai entry-point group; when enabled, the hook can write a receipt at the end of a task. The inspect extra installs inspect_ai>=0.3.112,<=0.3.288. Configure it through environment variables. PROOFBUNDLE_KEY is a path to a file containing a 32-byte raw Ed25519 seed; when unset, signing uses an ephemeral key with a warning. PROOFBUNDLE_OUT selects a file or directory. PROOFBUNDLE_METRIC selects the metric, PROOFBUNDLE_COMPARATOR defaults to >=, and PROOFBUNDLE_THRESHOLD is required for receipt emission since 5.0.0, with no default. The model and dataset stay salted commitments. proofbundle verify checks the signed payload of a receipt against the public key the receipt carries, not whose key that is, so a receipt someone changed and re-signed with their own key also verifies; pin the issuer you expect with proofbundle show-eval --expect-issuer ed25519:<key>, which then exits 1.

pytest (pytest11 plugin)

pip install "proofbundle[pytest,eval]"
PROOFBUNDLE_EMIT=1 PROOFBUNDLE_THRESHOLD=0.9 pytest     # or: PROOFBUNDLE_THRESHOLD=0.9 pytest --proofbundle

Installed via the pytest11 entry-point; when enabled, pytest_terminal_summary can emit a receipt of the run (metric pass_rate, with the per-outcome counts and exit status in provenance) from terminalreporter.stats. PROOFBUNDLE_THRESHOLD is required for receipt emission, with no default. If PROOFBUNDLE_KEY is unset, signing uses an ephemeral key with a warning. The metric is always pass_rate: PROOFBUNDLE_METRIC set to any other value skips the receipt and prints why, so no receipt names a metric the plugin did not compute.

GitHub Action

A composite action is prepared under action/action.yml (SHA-pinned). Usage:

- uses: b7n0de/proofbundle/[email protected]
  with:
    command: "verify receipt.json"

Optional, complementary — a GitHub-anchored SLSA provenance over the receipt (the receipt attests the run, attest-build-provenance attests the build). Add to the caller job:

permissions: { id-token: write, attestations: write, contents: read }
steps:
  - uses: b7n0de/proofbundle/[email protected]
    with: { command: "emit-eval --claim claim.json --out receipt.json --new-key signer.key" }
  - uses: actions/attest-build-provenance@<full-sha>  # v3
    with: { subject-path: receipt.json }
  - uses: actions/upload-artifact@<full-sha>          # v7
    with: { name: proofbundle-receipt, path: receipt.json }

promptfoo (results.json adapter, v1.4)

promptfoo already exports one JSON per run (promptfoo eval -o results.json, summary version 3). The adapter is file-based — no promptfoo dependency:

from proofbundle.adapters import from_promptfoo_results
from proofbundle.evalclaim import emit_eval_receipt
from proofbundle import generate_signer

claim, salts = from_promptfoo_results("results.json", comparator=">=",
                                      threshold="0.900000", timestamp="2026-07-02T14:00:00Z")
receipt = emit_eval_receipt(claim, generate_signer())

Metric: pass_rate = successes / (successes + failures + errors) from results.stats, as a fixed-point decimal. The model commitment pins the sorted provider-id set; the dataset commitment derives from the canonical config.tests JSON (the test suite IS the dataset). Legacy summary v1/v2 files are rejected with a clear message — re-export with a current promptfoo. In CI this composes with the GitHub Action: run promptfoo, emit the receipt, attach both to the run.

lm-evaluation-harness (results_*.json adapter, v0.6; sample-count provenance since v3.7.0)

EleutherAI's lm-evaluation-harness writes a results JSON per run. The adapter is file-based — no lm_eval import:

from proofbundle.adapters import from_lm_eval_results
from proofbundle.evalclaim import emit_eval_receipt
from proofbundle import generate_signer

claim, salts = from_lm_eval_results("results.json", "arc_easy", "acc", comparator=">=",
                                    threshold="0.30", timestamp="2026-07-01T12:00:00Z")
receipt = emit_eval_receipt(claim, generate_signer())

Metric keys resolve with the harness suffix convention (acc matches acc,none). The signed provenance carries git_hash, task version, n-shot, sibling stderr — and, since 3.7.0, the sample counts from the harness n-samples block: effective_samples, original_samples and skipped_samples, validated fail-closed (negative counts raise; effective > original clamps skipped_samples to 0 with both raw counts visible; effective = 0 keeps the honest n fallback). A silently subsetted run can no longer present itself as a full run under the same signed claim. See examples/lm_eval_receipt.py for a complete offline round-trip.

Hugging Face Community Evals (.eval_results/*.yaml, v1.4)

HF's Community Evals accept per-benchmark result entries with an optional string verifyToken ("a signature that can be used to prove that evaluation is provably auditable and reproducible"). Honest boundary: the Hub's verified badge is granted server-side by HF (currently HF Jobs + inspect-ai); that token format is not public, and proofbundle does not imitate it. What proofbundle provides is a self-contained, offline-verifiable token — pb1. + base64url(zlib(receipt JSON)) — that anyone can check without trusting the submitter:

from proofbundle.hf_evals import to_eval_results_entry, eval_results_yaml, verify_receipt_token

entry = to_eval_results_entry(receipt, dataset_id="Idavidrein/gpqa", task_id="gpqa_diamond",
                              value=0.412, date="2026-07-02",
                              source_url="https://example.com/my-eval-traces")
open(".eval_results/proofbundle.yaml", "w").write(eval_results_yaml([entry]))

result, bundle = verify_receipt_token(entry["verifyToken"])   # offline, exit-code style .ok

CLI: proofbundle hf-token receipt.json emits the token; proofbundle hf-token --verify <token> checks one. The entry builder is fail-closed (a non-verifying receipt is refused) and the YAML emitter is a strict, purpose-built serializer (JSON-escaped scalars — a valid-YAML subset — so dates and tokens can never be misparsed). Publishing a value is a disclosure decision the caller makes; a receipt may withhold the exact score via SD-JWT.

Where proofbundle sits (fairly)

proofbundle is the standards-native, offline receipt of an eval or test run, auto-emitted via the framework's own plugin API. It is complementary to its neighbours: ai-audit-trail records runtime agent Decision Receipts (FastAPI / LangChain / MCP, ISO 42001) — a different layer; ValiChord builds attestation bundles from inspect_ai logs post-hoc (its format is unsigned: JCS + RFC 6962 Merkle since v2, plus HMAC-derived audit challenges; no format version carries a signature). proofbundle's angle is the opt-in auto-emit via the native plugin (an inspect_ai hook + a pytest11 plugin) plus the standards stack (Ed25519 + RFC 6962 + SD-JWT + in-toto). Honest scope: it attests authenticity + integrity of the run, not the correctness of the computation.