feat(reporting): record the evaluated source identity on benchmark cards - #131

…ntities

Addresses three review findings on the evaluated-source contract.

Contradictory carriers no longer resolve by position. The renderer read the
Tier 3 payload, that payload's summary and every result's metadata as an
ordered candidate list and published the first that validated, so a card whose
Tier 3 payload named one repository and whose Tier 1 result named another
published whichever the iteration order reached first. Result ordering follows
tier execution rather than orchestration input, so it must not decide what a
card claims to describe. All populated carriers are now folded together and a
field two carriers disagree about raises EvaluatedSourceConflict. The existing
two-channel resolver is the two-item case of that fold, so its per-field merge
and its error wording are unchanged. Because the card is generated compulsorily
for skills, `validate` reports that conflict as a CLI error naming the values
that disagreed, rather than an unhandled traceback.

Revisions are accepted only in canonical shapes. The commit pattern admitted a
7-character prefix, which can grow a second match as a tree grows, and the
digest pattern admitted any algorithm followed by 32 to 128 hex characters, so
sha512 with 32 hex characters rendered as immutable provenance. A commit must
now be a full Git object id (40 or 64 hex) and a digest must match the width of
the algorithm it names. The stdlib-only publication gate mirrors both rules.

Strict publication mode now requires the evaluator container revision, and
requires it to be pinned by digest. Blanking the golden card's container digest
previously produced no offenders, which contradicted both the flag's
documentation and the confusion issue #72 reported, where the only recorded
revision was a shared evaluator image.

Tests cover each rule, including the reviewer's reproduction: a Tier 3 payload
and a Tier 1 result naming different repositories now fails closed in either
result order, rather than publishing one of them.

Signed-off-by: Max Freedom Pollard <[email protected]>

…n input

Nothing in the tree populated the identity, so a normal validate run rendered
all three provenance fields as not recorded and the confusion issue #72
reported stayed reproducible.

`validate` now accepts --evaluated-source-repository,
--evaluated-source-revision and --evaluator-container-revision. One revision
option covers both immutable shapes the card accepts, matching its single
"Evaluated source revision" line: a full Git object id, or a content digest for
a run with no upstream commit. A value that is not canonical is refused at the
CLI boundary rather than dropped, so a card never reports not recorded for a
field the operator believed they had supplied.

The identity reaches the card through the result metadata carrier, which is the
seam that works for every card shape including a Tier 1-only PASS. It is
attached with setdefault, so an identity a producer already recorded stays in
place and a disagreement between the two is reported rather than resolved.
For a Tier 3 run it also travels EvaluationOptions to tier3.commands.evaluate
to the Harbor runner, which persists it in run_config.json, so a card rendered
later from that run directory records the same source tree.

CI now runs the publication scan with --require-source-provenance. The flag
remains opt-in for trees whose cards predate the contract.

Without these options the card renders not recorded exactly as before, which
the default-path test asserts.

Signed-off-by: Max Freedom Pollard <[email protected]>

One ordering-dependent path survived the previous fix. The card resolved the
evaluated source from the first result carrying an `agent_eval` payload, so a
run that produced two agent-eval results published whichever payload came
first: reversing the result list flipped the card to the other source tree
instead of raising EvaluatedSourceConflict. Result order follows how a run was
scheduled, so it must not decide what a published card claims to describe.

`source_identity` now owns the whole set of carriers. `evaluated_source_carriers`
names the three places one result can record the identity: its own metadata, the
agent-eval payload, and that payload's summary. `recorded_evaluated_source`
folds every carrier of every result through the existing per-field merge, so a
second payload can no longer be shadowed and a disagreement anywhere fails
closed in either order. Keeping the set in one place also keeps the card and the
machine-readable report reading the same carriers, rather than honouring one in
a report and ignoring it in another.

The renderer still selects the first payload for the score and run-detail lines,
which describe a single run, and no longer uses that selection to resolve
provenance.

The regression covers the reviewer's reproduction in both orders: two full
payloads naming different repositories, and a first payload against a
summary-only carrier on a second result. Two payloads that agree still render
the identity rather than reading as a conflict.

Signed-off-by: Max Freedom Pollard <[email protected]>

`validate` attached and resolved the identity after `emit_reports`, so the
machine-readable report disagreed with the card published beside it. A
Tier 1-only run wrote a JSON report carrying neither the source repository nor
its revision while BENCHMARK.md recorded both, and a run whose carriers
contradicted each other wrote JSON and HTML naming one of the two before
benchmark generation failed the run.

The attach and resolve step now runs immediately after `apply_policy`, for
every content type rather than only for skills, so a run that cannot say what
it evaluated writes nothing at all. The supplied identity is folded in as its
own carrier rather than only attached: `setdefault` is silent where it declines
to overwrite, so a result that already recorded a different source tree would
otherwise publish that one and drop the operator's orchestration input
unreported. The BENCHMARK.md block keeps its own conflict handling as a last
line of defence, because the renderer re-validates whatever carriers it is
handed.

The JSON report now always carries a top-level `evaluated_source` resolved from
the same fold, so CI can verify provenance from the file it already parses. The
key is always present: `null` says the orchestration input supplied none,
rather than leaving a consumer to guess whether an older reporter omitted it.

Tests cover a Tier 1-only run whose JSON report and card record the same
identity, the JSON contract for a recorded, absent and contradictory identity,
and a payload/CLI conflict that leaves no JSON, HTML or BENCHMARK.md behind.

Signed-off-by: Max Freedom Pollard <[email protected]>

…llel catalog children

The single-skill `validate` path supplied the identity, but the two other
producers still dropped it, so the contract held only for the narrowest run
shape.

`tier3 evaluate` exposed no provenance options at all. It owns its own run
directory, so a standalone Tier 3 run persisted `evaluated_source: null` into
`run_config.json` and any card rendered later from that directory reported the
source tree as not recorded, which is the confusion the contract exists to
prevent. The command now accepts the same three options as `validate`, with the
same names and the same canonical-shape refusal at the CLI boundary, and passes
the resolved identity through `EvaluationOptions` to the engine. The visible
`tier3 evaluate` re-lists the hidden command's params, so both spellings pick
the options up.

`validate <catalog> --workers > 1` rebuilds each child's argv from an explicit
allowlist, and that allowlist omitted all three options. Every child renders its
own BENCHMARK.md, so a parallel catalog run published a whole directory of cards
saying the evaluated source was never recorded while the serial run of the same
catalog recorded it. The three options are now forwarded with the rest.

A drift guard covers the allowlist itself: every `validate` parameter must
either be forwarded by name or appear in the test's not-forwarded map with the
reason it must stay behind, so the next option added to `validate` cannot be
silently dropped on the way to a child.

The standalone path is covered down to disk rather than by signature: a run
driven through `tier3.commands.evaluate` with a failing runtime preflight still
writes `run_config.json`, and the test asserts the identity is in it. The
parallel path is covered end to end on a two-skill catalog, asserting both
children's cards name the repository and the supplied revision.

Signed-off-by: Max Freedom Pollard <[email protected]>

The entry now covers the producers that carry the identity (both commands,
parallel catalog children and the JSON report), that conflicting carriers
fail before any report is written, and the component rules a container
revision is validated against.

Signed-off-by: Max Freedom Pollard <[email protected]>

`_finalize_evaluated_source` attached the supplied identity with `setdefault`
and then folded the carriers only to detect a conflict, discarding the merged
value. A result that had already recorded part of the identity therefore
shadowed the rest of it. A Tier 1 result carrying just
`metadata["evaluated_source"] = {"repository": "NVIDIA/NVFlare"}`, which is the
documented seam for a card published without a completed Tier 3 run, published
a card and a JSON report calling the revision and the evaluator container not
recorded, though the operator had supplied both on the command line. Nothing
said so: the fold saw no disagreement, because a partial carrier does not
conflict with the fields it omits.

The fold's result is now what gets written. Every carrier, the orchestration
input included, is merged into one identity, and that union is written back to
each result whose metadata is a mapping, as a fresh dict per result so no two
results and no caller share one mutable identity. A run that recorded no
identity anywhere still has none written, so absent stays absent rather than
becoming empty. Conflicts keep failing closed with nothing on disk.

The hidden `evaluate` command advertised its repository option as "recorded on
BENCHMARK.md", which that command never writes. It says where the value
actually lands instead: the run's `run_config.json`, which a card rendered
later reads.

Signed-off-by: Max Freedom Pollard <[email protected]>

The reference grammar read any single first label as a registry host, so
`NVIDIA/skillevaluator@sha256:...` was accepted while the same repository under
a registry, `ghcr.io/NVIDIA/skillevaluator@sha256:...`, was refused. The OCI
grammar has no such asymmetry. A first component is a host only when it is
`localhost`, when it carries a dot, or when it carries a port; anything else
begins the path, whose components are lower case. Both spellings above are an
uppercase path component, so both are refused now, and a host is recognised in
the three shapes the grammar names, in any case, because DNS is
case-insensitive.

The 255-character bound was applied to the host and the path together. The
grammar's RepositoryNameTotalLengthMax measures the path once the host has been
split off it, so a long registry name was spending a budget that belongs to the
path. The path is now its own capture group and the bound measures that group,
in the library and in the gate alike. The constant says path rather than name,
so the two cannot be read as the same thing again.

The gate's `_GIT_OBJECT_ID` was a hand copy of `_SOURCE_COMMIT` that the drift
test never compared, so it could have drifted while the reference grammar still
matched. Its text is now identical, anchors included, and the drift test
compares it alongside the reference pattern and the bound.

Adversarial inputs were measured through both copies of the grammar before this
landed: 5,000-character runs of `-`, `.`, `_`, `/` and alternating `a-`
complete in about 1.2 ms, so the added alternation costs no catastrophic
backtracking.

The docs and the changelog now state the rule the code enforces, including the
lower-case path components, which were written down nowhere.

Signed-off-by: Max Freedom Pollard <[email protected]>

`_PUBLISHED_PASS` matched any line that contained "Overall verdict: PASS",
because its `.*` ran straight past the list marker that opened the line. Every
card carries the policy section, which restates the rule as a bullet, "- Overall
verdict: PASS only when every configured dimension passes for at least one
supported agent", so an INCOMPLETE or a FAIL card read as a published PASS.
Under `--require-source-provenance` that card was then asked for the three
provenance fields a publication claims, and the scan reported three offenders
against a card that published nothing. A local INCOMPLETE card failed the strict
scan on exactly those three lines, none of which it was answerable for.

Both detectors now read a verdict line and nothing else. A list item is excluded
at the line start, and the verdict has to close its line or introduce the
wording that explains it, so the rendered callout, `> **Overall verdict: PASS`
followed by the recommendation, still matches while the policy bullet does not.
The two are built from one shared tail, so they cannot drift into disagreeing
about what a verdict line is. `_PUBLISHED_PASS` still accepts a missing
blockquote marker, which is how a hand-authored backfill card states its
verdict, and `_OVERALL_PASS` still requires one.

The strict scan of a tree of cards is the only place this was visible, so the
tests scan the two shapes that matter: a non-PASS card carrying the policy
bullet, which must raise no publication offence, and the published card, which
carries the same bullet and must still be charged for provenance it omits.

Signed-off-by: Max Freedom Pollard <[email protected]>

`_catalog_child_argv` had no caller anywhere in the tree, and the only reason
`_catalog_child_argv_from_sys` existed was to be chosen by it. The parallel
catalog path calls `_catalog_child_argv_from_ctx` directly, so both functions
were a second answer to a question the code stopped asking.

Keeping them was not free. The sys builder rebuilt a child command by scraping
the parent argv, and it kept only the flags: anything that did not start with
`-` was skipped, option values included. An option that takes a value therefore
came out bare. Reinstated on this branch, `--evaluated-source-repository
NVIDIA/demo` would have reached the child as the option alone, and Click would
have read the appended `-o` as its value, leaving the output directory a stray
argument. The identity this branch forwards to every catalog child would have
gone missing on the way, and the cards would have said the evaluated source was
never recorded. Deleting the builder closes that path rather than leaving it in
reach of the next person who needs one.

Signed-off-by: Max Freedom Pollard <[email protected]>