Skip to content

Latest commit

 

History

History
165 lines (130 loc) · 11.6 KB

EVAL_CLAIM.md

File metadata and controls

165 lines (130 loc) · 11.6 KB

Eval-claim specification — proofbundle/eval-claim/v0.1

The format of an eval claim, the signed statement inside an eval receipt. This page is for anyone who implements or checks eval receipts. It lists the fields, the canonical form and how the claim is bound to the key that signed it. Section 1 says what a receipt proves and what it does not.

An eval receipt is a regular proofbundle/v0.1 bundle (see SPEC.md) whose payload is a canonical eval claim. This document specifies the claim.

1. What a receipt proves (and does not)

A receipt is tamper-evident signed evidence of exactly one thing: on suite S, metric comparator threshold held, passed=… — signed by a stated issuer and anchored in a tamper-evident Merkle tree. It carries only salted commitments to the model and dataset identifiers, never the weights, the raw data, or the plaintext names.

It does not prove that the evaluation was well designed, that the suite measures what it claims, or that the reported score is itself correct. Those are human judgements. What it removes is the need to simply trust the number.

1a. Score evidence classes

The exact score is used only to compute passed and is then discarded, so a receipt carries a threshold verdict, not an exact score. show-eval declares this explicitly, and the machine-readable classifier proofbundle.evalclaim.eval_evidence_class returns one of:

  • THRESHOLD_VERDICT_VERIFIED — the signed claim proves passed for the stated comparator/threshold. This is the ONLY class the frozen v0.1 schema produces.
  • EXACT_SCORE_VERIFIED — reachable only through the optional, additive exact-score profile (a signed decimal-string score whose recomputed passed agrees). Not part of the frozen 3.x core; EXPERIMENTAL.
  • SCORE_COMMITMENT_PRESENT — a signed score commitment is a binding, not a range proof: it does not prove the hidden score crossed the threshold.
  • SCORE_WITHHELD — the exact score is deliberately withheld; only the threshold verdict is signed.
  • METHODOLOGY_NOT_EVALUATED — always present: a receipt never judges whether the suite is well designed.

2. Data minimization

The payload contains only salted commitments, the threshold, and passed — never weights, raw data, or plaintext model/dataset names. Each identifier appears as a salted commitment sha256:hex(salt ‖ utf8(identifier)) with a ≥16-byte high-entropy salt that stays with the issuer and is never in the payload. Without the salt the identifier cannot be recovered from the commitment — not even via a rainbow table over known model names such as gpt-4o. The issuer may later disclose identifier + salt; that is the path for selective disclosure in v0.5.

3. Fields

field required type notes
schema yes string const proofbundle/eval-claim/v0.1
suite yes string eval suite name
suite_version yes string
metric yes string e.g. accuracy, refusal_rate
comparator yes string one of >= > <= <
threshold yes string a decimal string (e.g. "0.80"), never a JSON float
passed yes boolean computed by the emitter from comparator+threshold, not trusted from the caller
n yes integer sample size, 0 ≤ n ≤ 2^53-1
model_id_commit yes string sha256:<hex> salted commitment to the model identifier
dataset_id_commit yes string sha256:<hex> salted commitment to the dataset identifier
commit_alg yes string const sha256-salted-v1
issuer yes string ed25519:<base64 of the 32-byte public key> — part of the SIGNED payload; binds the receipt to the issuer
timestamp yes string RFC 3339
assurance_level yes enum self_attested (default) · third_party · reproduced · enclave_attested — how much a PASS is worth; SIGNED (issuer-declared), always shown by show-eval. See THREAT_MODEL.md. enclave_attested is a STRING claim by itself — proofbundle.evalclaim.enclave_assurance_proven(claim, bundle, eat_jws=…, verifier_pubkey=…) optionally corroborates it against a real TEE Attestation Result (show-eval --eat/--verifier-key; EXPERIMENTAL v2.0, docs/EXPERIMENTAL_ENCLAVE.md) — additive, never force-promotes the signed field itself
context_binding no string hash of an external context (e.g. a request id), against reuse in a foreign context
ci95 no array exactly two decimal strings
multiple_testing no string e.g. holm
prereg_sha256 no string sha256 (hex) over the RAW bytes of the eval protocol file, committed BEFORE the run (proofbundle prereg); a verifier re-hashes the disclosed protocol and checks it
evaluation_card_sha256 no string sha256 (hex) over the RAW bytes of an external, human-readable Eval Card document (Hugging Face EvalEval Coalition "Evaluation Cards", arXiv:2606.09809 — see src/proofbundle/evalcard.py); mechanically identical to prereg_sha256 (proofbundle evalcard / evalcard.verify_evaluation_card). Added in this revision: because the schema is additionalProperties: false, a receipt carrying this field is a one-way compatibility step — an older proofbundle build rejects it as an unknown field (mirrors anchors[], SPEC.md §7i) rather than silently ignoring it
provenance no object traceability metadata (not a security commitment): harness, git_hash, harness_version, run_id, run_timestamp (log-native), config_hash (<alg>:<hex> over canonical config JSON), plus adapter-specific keys (e.g. task_hash, stderr) and the additive benchmark-hacking VISIBILITY keys run_attempts/aborted_runs (non-negative integers) and methodology_sha256/benchjack_audit_report_sha256 (plain sha256 references; see THREAT_MODEL.md — visibility only, never a guarantee against a gamed benchmark, BenchJack arXiv:2605.12673)

Reported-version status (5.0.0, additive)

Every provenance field that carries a version as reported by the harness — harness_version, task_version, promptfoo_version — is accompanied by an explicit status:

key values meaning
<field>_status reported | not_reported | not_bound what the harness actually said
<field>_status_reason free text mandatory whenever the status is not reported

Why a status and not a boolean. A boolean has three states of its own — true, false and absent — and would push the ambiguity one level up instead of closing it.

What it fixes. Before 5.0.0 a version field was simply ABSENT when the harness reported none, so absence carried two different meanings at once: the harness ran and reported no version and no harness was bound at all. A verifier could not tell them apart. For a product whose whole claim is that a receipt says what it means, that is the failure class it exists against.

The version field itself is unchanged. When nothing was reported the version key stays absent — writing a value the harness never reported would put a number into evidence that nobody measured. The status makes a statement about the REPORTING, never about the version.

Never derived. The status is written only from what the harness returned. It is not inferred from neighbouring fields, and not_reported is never an all-clear: it does not fold to PASS and no gate reads it as one.

A verifier rejects a provenance block whose status is outside the three literals, whose non-reported status carries no reason, or whose status and field contradict each other in either direction (reported without the field, or the field present while the status denies it). Conformance vectors: conformance/provenance/version-status-* — one per status value plus one per rejection class.

What the verifier checks, exactly. The rule judges the three fields named above and nothing else: a member whose name ends in _status but whose field is not one of them (run_status, scorer_status, or schema_version_status, whose schema_version is not a harness-reported version) never produces an issue, whatever its value. A block that carries a version field and no <field>_status is not rejected; the status is additive, and receipts issued before 5.0.0 carry none. For each <field>_status of a judged field, in ascending order of the member names:

  1. a value outside the three literals is one issue, and nothing else is checked for that field;
  2. otherwise a status other than reported whose <field>_status_reason is absent, null or empty after trimming whitespace is one issue;
  3. reported while the field is absent, null or the empty string is one issue;
  4. not_reported or not_bound while the field is present (not absent, null or the empty string) is one issue.

Steps 2 to 4 are each judged, so one status member can give two issues (step 2 beside step 4). Step 2 reads only a reason that is absent, null or a string that is empty after trimming; a present reason that is not a string counts as given.

Each issue is reported as one sentence, built from these templates, where {key} is the name of the status member, {field} the name of the version field, {status} the status value as written, and {repr(x)} Python's repr of a value (a string in single quotes); the literal list is written as Python writes a list of strings:

  • {key}={repr(status)} is not one of ['reported', 'not_reported', 'not_bound']
  • {key}={status} without {field}_status_reason (reason is mandatory)
  • {key}=reported but {field} is absent
  • {key}={status} but {field} is present ({repr(value)})

The conformance vectors compare the issue sentences as a set; an empty set means the block is consistent in this respect, never that the versions are trustworthy.

| samples | no | object | per-sample Merkle commitment {root_b64, n, leaf_alg} — SIGNED; samples.n MUST equal n; enables the forced-random-sample audit (SPEC §7g, proofbundle audit-challenge / verify-opening) |

Machine-readable: schemas/eval_claim_v0_1.schema.json.

4. Canonicalization (RFC 8785 JCS) — emit path only

The payload bytes are the claim canonicalized with RFC 8785 JCS. This profile, enforced on the emit path:

  • object keys sorted by UTF-16 code units (not Python code points — otherwise it diverges on emoji and characters beyond the BMP);
  • duplicate keys rejected when parsing claim JSON;
  • all string values NFC-normalized (non-NFC rejected), the values inside arrays and nested objects included; member names are not checked for NFC;
  • Python floats rejected (numbers with fractional parts are decimal strings);
  • integers limited to the IEEE-754 safe range (2^53-1);
  • compact separators, UTF-8.

The integer range is also part of the claim rule, so it holds at the verify boundary too: decode_eval_claim refuses an integer beyond ±(2^53-1) anywhere in the claim (6.2.0). NFC strings and the float ban stay emit-only, for the reason given below.

A real RFC 8785 library is used only on the emit path. The verify path never canonicalizes — decode_eval_claim checks the exact stored bytes that verify_bundle already authenticated — so the verifier stays dependency-free (cryptography + stdlib only).

5. Issuer binding

emit_eval_receipt sets issuer to the signer's fingerprint. decode_eval_claim verifies the bundle, then checks that the bundle's signing key equals the claim's issuer field; a mismatch fails decoding. A receipt is therefore bound to the key that signed it — you cannot lift a claim under a different signature.