The format of an eval claim, the signed statement inside an eval receipt. This page is for anyone who implements or checks eval receipts. It lists the fields, the canonical form and how the claim is bound to the key that signed it. Section 1 says what a receipt proves and what it does not.
An eval receipt is a regular proofbundle/v0.1 bundle (see SPEC.md)
whose payload is a canonical eval claim. This document specifies the claim.
A receipt is tamper-evident signed evidence of exactly one thing: on suite S, metric comparator
threshold held, passed=… — signed by a stated issuer and anchored in a
tamper-evident Merkle tree. It carries only salted commitments to the model
and dataset identifiers, never the weights, the raw data, or the plaintext names.
It does not prove that the evaluation was well designed, that the suite measures what it claims, or that the reported score is itself correct. Those are human judgements. What it removes is the need to simply trust the number.
The exact score is used only to compute passed and is then discarded, so a receipt carries a
threshold verdict, not an exact score. show-eval declares this explicitly, and the
machine-readable classifier proofbundle.evalclaim.eval_evidence_class returns one of:
THRESHOLD_VERDICT_VERIFIED— the signed claim provespassedfor the statedcomparator/threshold. This is the ONLY class the frozen v0.1 schema produces.EXACT_SCORE_VERIFIED— reachable only through the optional, additive exact-score profile (a signed decimal-stringscorewhose recomputedpassedagrees). Not part of the frozen 3.x core; EXPERIMENTAL.SCORE_COMMITMENT_PRESENT— a signed score commitment is a binding, not a range proof: it does not prove the hidden score crossed the threshold.SCORE_WITHHELD— the exact score is deliberately withheld; only the threshold verdict is signed.METHODOLOGY_NOT_EVALUATED— always present: a receipt never judges whether the suite is well designed.
The payload contains only salted commitments, the threshold, and passed — never
weights, raw data, or plaintext model/dataset names. Each identifier appears as a
salted commitment sha256:hex(salt ‖ utf8(identifier)) with a ≥16-byte high-entropy
salt that stays with the issuer and is never in the payload. Without the salt the
identifier cannot be recovered from the commitment — not even via a rainbow table over
known model names such as gpt-4o. The issuer may later disclose identifier + salt;
that is the path for selective disclosure in v0.5.
| field | required | type | notes |
|---|---|---|---|
schema |
yes | string | const proofbundle/eval-claim/v0.1 |
suite |
yes | string | eval suite name |
suite_version |
yes | string | |
metric |
yes | string | e.g. accuracy, refusal_rate |
comparator |
yes | string | one of >= > <= < |
threshold |
yes | string | a decimal string (e.g. "0.80"), never a JSON float |
passed |
yes | boolean | computed by the emitter from comparator+threshold, not trusted from the caller |
n |
yes | integer | sample size, 0 ≤ n ≤ 2^53-1 |
model_id_commit |
yes | string | sha256:<hex> salted commitment to the model identifier |
dataset_id_commit |
yes | string | sha256:<hex> salted commitment to the dataset identifier |
commit_alg |
yes | string | const sha256-salted-v1 |
issuer |
yes | string | ed25519:<base64 of the 32-byte public key> — part of the SIGNED payload; binds the receipt to the issuer |
timestamp |
yes | string | RFC 3339 |
assurance_level |
yes | enum | self_attested (default) · third_party · reproduced · enclave_attested — how much a PASS is worth; SIGNED (issuer-declared), always shown by show-eval. See THREAT_MODEL.md. enclave_attested is a STRING claim by itself — proofbundle.evalclaim.enclave_assurance_proven(claim, bundle, eat_jws=…, verifier_pubkey=…) optionally corroborates it against a real TEE Attestation Result (show-eval --eat/--verifier-key; EXPERIMENTAL v2.0, docs/EXPERIMENTAL_ENCLAVE.md) — additive, never force-promotes the signed field itself |
context_binding |
no | string | hash of an external context (e.g. a request id), against reuse in a foreign context |
ci95 |
no | array | exactly two decimal strings |
multiple_testing |
no | string | e.g. holm |
prereg_sha256 |
no | string | sha256 (hex) over the RAW bytes of the eval protocol file, committed BEFORE the run (proofbundle prereg); a verifier re-hashes the disclosed protocol and checks it |
evaluation_card_sha256 |
no | string | sha256 (hex) over the RAW bytes of an external, human-readable Eval Card document (Hugging Face EvalEval Coalition "Evaluation Cards", arXiv:2606.09809 — see src/proofbundle/evalcard.py); mechanically identical to prereg_sha256 (proofbundle evalcard / evalcard.verify_evaluation_card). Added in this revision: because the schema is additionalProperties: false, a receipt carrying this field is a one-way compatibility step — an older proofbundle build rejects it as an unknown field (mirrors anchors[], SPEC.md §7i) rather than silently ignoring it |
provenance |
no | object | traceability metadata (not a security commitment): harness, git_hash, harness_version, run_id, run_timestamp (log-native), config_hash (<alg>:<hex> over canonical config JSON), plus adapter-specific keys (e.g. task_hash, stderr) and the additive benchmark-hacking VISIBILITY keys run_attempts/aborted_runs (non-negative integers) and methodology_sha256/benchjack_audit_report_sha256 (plain sha256 references; see THREAT_MODEL.md — visibility only, never a guarantee against a gamed benchmark, BenchJack arXiv:2605.12673) |
Every provenance field that carries a version as reported by the harness — harness_version,
task_version, promptfoo_version — is accompanied by an explicit status:
| key | values | meaning |
|---|---|---|
<field>_status |
reported | not_reported | not_bound |
what the harness actually said |
<field>_status_reason |
free text | mandatory whenever the status is not reported |
Why a status and not a boolean. A boolean has three states of its own — true, false and absent — and would push the ambiguity one level up instead of closing it.
What it fixes. Before 5.0.0 a version field was simply ABSENT when the harness reported none, so absence carried two different meanings at once: the harness ran and reported no version and no harness was bound at all. A verifier could not tell them apart. For a product whose whole claim is that a receipt says what it means, that is the failure class it exists against.
The version field itself is unchanged. When nothing was reported the version key stays absent — writing a value the harness never reported would put a number into evidence that nobody measured. The status makes a statement about the REPORTING, never about the version.
Never derived. The status is written only from what the harness returned. It is not inferred
from neighbouring fields, and not_reported is never an all-clear: it does not fold to PASS and
no gate reads it as one.
A verifier rejects a provenance block whose status is outside the three literals, whose
non-reported status carries no reason, or whose status and field contradict each other in
either direction (reported without the field, or the field present while the status denies it).
Conformance vectors: conformance/provenance/version-status-* — one per status value plus one
per rejection class.
What the verifier checks, exactly. The rule judges the three fields named above and nothing
else: a member whose name ends in _status but whose field is not one of them (run_status,
scorer_status, or schema_version_status, whose schema_version is not a harness-reported
version) never produces an issue, whatever its value. A block that carries a version field and no
<field>_status is not rejected; the status is additive, and receipts issued before 5.0.0 carry
none. For each <field>_status of a judged field, in ascending order of the member names:
- a value outside the three literals is one issue, and nothing else is checked for that field;
- otherwise a status other than
reportedwhose<field>_status_reasonis absent,nullor empty after trimming whitespace is one issue; reportedwhile the field is absent,nullor the empty string is one issue;not_reportedornot_boundwhile the field is present (not absent,nullor the empty string) is one issue.
Steps 2 to 4 are each judged, so one status member can give two issues (step 2 beside step 4). Step 2
reads only a reason that is absent, null or a string that is empty after trimming; a present reason
that is not a string counts as given.
Each issue is reported as one sentence, built from these templates, where {key} is the name of
the status member, {field} the name of the version field, {status} the status value as written,
and {repr(x)} Python's repr of a value (a string in single quotes); the literal list is written
as Python writes a list of strings:
{key}={repr(status)} is not one of ['reported', 'not_reported', 'not_bound']{key}={status} without {field}_status_reason (reason is mandatory){key}=reported but {field} is absent{key}={status} but {field} is present ({repr(value)})
The conformance vectors compare the issue sentences as a set; an empty set means the block is consistent in this respect, never that the versions are trustworthy.
| samples | no | object | per-sample Merkle commitment {root_b64, n, leaf_alg} — SIGNED; samples.n MUST equal n; enables the forced-random-sample audit (SPEC §7g, proofbundle audit-challenge / verify-opening) |
Machine-readable: schemas/eval_claim_v0_1.schema.json.
The payload bytes are the claim canonicalized with RFC 8785 JCS. This profile, enforced on the emit path:
- object keys sorted by UTF-16 code units (not Python code points — otherwise it diverges on emoji and characters beyond the BMP);
- duplicate keys rejected when parsing claim JSON;
- all string values NFC-normalized (non-NFC rejected), the values inside arrays and nested objects included; member names are not checked for NFC;
- Python floats rejected (numbers with fractional parts are decimal strings);
- integers limited to the IEEE-754 safe range (
2^53-1); - compact separators, UTF-8.
The integer range is also part of the claim rule, so it holds at the verify boundary too:
decode_eval_claim refuses an integer beyond ±(2^53-1) anywhere in the claim (6.2.0). NFC
strings and the float ban stay emit-only, for the reason given below.
A real RFC 8785 library is used only on the emit path. The verify path never
canonicalizes — decode_eval_claim checks the exact stored bytes that
verify_bundle already authenticated — so the verifier stays dependency-free
(cryptography + stdlib only).
emit_eval_receipt sets issuer to the signer's fingerprint. decode_eval_claim
verifies the bundle, then checks that the bundle's signing key equals the claim's
issuer field; a mismatch fails decoding. A receipt is therefore bound to the key
that signed it — you cannot lift a claim under a different signature.