Skip to content

feat(fingerprint): cluster identity projection from snapshot measurements#844

Merged
njhensley merged 13 commits into
NVIDIA:mainfrom
njhensley:feat/snapshot-fingerprint
May 11, 2026
Merged

feat(fingerprint): cluster identity projection from snapshot measurements#844
njhensley merged 13 commits into
NVIDIA:mainfrom
njhensley:feat/snapshot-fingerprint

Conversation

@njhensley

@njhensley njhensley commented May 11, 2026

Copy link
Copy Markdown
Member

Summary

Add a new pkg/fingerprint package that projects snapshot measurements
into a typed cluster-identity record (service, accelerator, OS, region,
k8s version, node counts) and offers a three-way per-dimension match
against recipe.Criteria. Surface it as a top-level fingerprint:
block on aicr snapshot output and unify the existing snapshot →
criteria extraction through the same projection layer.

Motivation / Context

ADR-007 (verifiable recipe test evidence) needs the bundler and
verifier to confirm that the cluster a recipe was tested on actually
matched the recipe's declared criteria — without re-running
validation. Today the snapshot YAML carries only raw measurements; a
reviewer can't tell at a glance "is this an H100 EKS cluster" without
hand-parsing every collector's output. This PR is the foundational
piece that #753 (aicr evidence verify) and #754 (aicr validate --emit-attestation) build on.

A second motivation surfaced during implementation: pkg/recipe
already had ExtractCriteriaFromSnapshot doing a near-identical
projection with its own GPU/OS normalization. The two were drifting.
This PR consolidates both paths through fingerprint.FromMeasurements

  • Fingerprint.ToCriteria and inverts the recipe → snapshotter
    import edge so the cycle the original adapter dodged is no longer
    needed.

Fixes: #752
Related: #750, #753, #754, #739, ADR-007 (docs/design/007-recipe-evidence.md)

Type of Change

  • New feature (non-breaking change that adds functionality)
  • Refactoring (no functional changes) — recipe.ExtractCriteriaFromSnapshot consolidated into fingerprint

Component(s) Affected

  • CLI (cmd/aicr, pkg/cli) — aicr recipe --snapshot reads from the new path
  • Recipe engine / data (pkg/recipe) — drops ExtractCriteriaFromSnapshot; cycle inversion
  • Collectors / snapshotter (pkg/collector, pkg/snapshotter) — Snapshot.Fingerprint field, populated post-collection
  • Core libraries — new pkg/fingerprint
  • Docs/examples — new "Cluster Fingerprint" section in docs/contributor/data.md

Implementation Notes

Dimensions captured

Dimension Source Recipe criteria participant?
service k8s.node.provider (parsed providerID) yes
accelerator gpu.smi.gpu.model reconciled against nvidia.com/gpu.product topology labels yes
os.value + os.version /etc/os-release ID and VERSION_ID value only (criteria has no OS version)
k8sVersion k8s.server.version (leading v stripped) no (audit only)
region topology.kubernetes.io/region aggregated label no (audit only)
nodeCount nodeTopology.summary.node-count (all nodes) yes (nodes)
gpuNodeCount union of nodes carrying nvidia.com/gpu.product no (audit only)

Match semantics

Fingerprint.Match(*recipe.Criteria) returns MatchResult{Matched bool, PerDimension []DimensionDiff} with three-way per-dimension outcomes
(matched | mismatched | unknown). unknown covers two cases: the
fingerprint deliberately doesn't capture this dimension (intent,
platform — recipe-author choices, not cluster facts) and the
fingerprint failed to detect it (e.g., no GPU collector output). The
overall Matched flag stays true unless a dimension is mismatched;
unknowns surface for human review without flipping the overall outcome.

PerDimension is an ordered slice with typed DimensionName keys and
a Find() helper — deterministic iteration and serialization, no
magic-string map keys.

Heterogeneity handling

Two cases where naïve detection would lie:

  • Multi-region clusters. The topology collector emits
    disambiguated topology.kubernetes.io/region.<value> keys when
    nodes have differing region labels. extractRegion returns empty
    with Note: "multi-region" rather than picking arbitrarily.
  • Heterogeneous GPU clusters. The smi collector only inspects
    the snapshotter's local node, so an 8-node cluster split between
    H100 and L40 would otherwise be claimed as homogeneous in whichever
    SKU the agent landed on. A reconciliation pass cross-references the
    nvidia.com/gpu.product topology labels; disagreeing labels →
    Note: "multi-gpu" with empty Value.

Cycle inversion

Before this PR: pkg/recipe imported pkg/snapshotter for
ExtractCriteriaFromSnapshot, forcing the new pkg/fingerprint to
use a flat-string CriteriaInput adapter to avoid closing a cycle
(pkg/snapshotter → pkg/fingerprint → pkg/recipe → pkg/snapshotter).

After this PR: Fingerprint.ToCriteria() owns the snapshot→criteria
projection, Fingerprint.Match takes *recipe.Criteria directly, the
CriteriaInput/ToFingerprintInput adapter pair is removed, and
pkg/recipe/snapshot.go is deleted. The import direction is now
pkg/recipe → pkg/fingerprint only.

Quiet correctness fix in pkg/cli/recipe_test.go

The test-data update in pkg/cli/recipe_test.go is not a pure rename.
The deleted recipe.ExtractCriteriaFromSnapshot was reading
subtype="server" key="service" and subtype="device" key="model",
but the k8s collector emits subtype="node" key="provider"
(pkg/collector/k8s/node.go:56) and the GPU collector emits
subtype="smi" key="gpu.model" (pkg/collector/gpu/gpu.go:220). The
old function was reading keys that don't exist in real collector
output — the tests passed only because the synthetic fixtures had
been fabricated to match the broken reads. Consolidating through
fingerprint.FromMeasurements quietly fixes that bug; the
pkg/cli/recipe_test.go data update brings the fixtures back in line
with what real collectors produce. Treat this PR as a correctness
improvement on top of the refactor, not a pure refactor.

Snapshot.Fingerprint is advisory, not authoritative

A field comment on Snapshot.Fingerprint explicitly states it is a
convenience for humans reading the YAML, not a trust-bearing claim.
The snapshot file isn't signed at this layer — an attacker controlling
it could swap the embedded fingerprint without touching the
measurements. ADR-007 consumers (bundler at #754, verifier at #753)
MUST recompute via fingerprint.FromMeasurements(snap.Measurements).

Persona-review pass

After the initial implementation, a five-persona review (security,
downstream-consumer, cluster-operator, recipe-contributor DX, Go API
design) surfaced eight follow-up items, all addressed in commits 3-4:

Item Disposition
B1 Predicate body keeps three-way match: state, not bool collapse Documented in data.md
B2 Snapshot.Fingerprint marked advisory + recompute expectation Field godoc
B3 PerDimension ordered slice with typed DimensionName keys API change
M1 Unify ExtractCriteriaFromSnapshot into fingerprint.ToCriteria Refactor
M2 Distinguish multi-region from no-label Dimension.Note field
R1 Multi-GPU heterogeneity reconciliation Reconciliation pass
R2 GPUNodeCount separate from NodeCount New dimension
R3 Gate OS.Version on a recognized OS.Value Avoid version-without-kind audit confusion

Per-signal provenance (signals[], confidence per dimension) is
ADR-007 V2 — additive, no schema break.

Agent-mode parity

Local and agent-mode share the same measure() function and the same
Serializer.Serialize path; the fingerprint is computed inside the
agent pod and emitted via the existing ConfigMap writer. The
tests/chainsaw/snapshot/deploy-agent assertion file doesn't yet pin
the fingerprint block; it could be a small follow-up.

Testing

unset GITLAB_TOKEN
make qualify

Local coverage

  • pkg/fingerprint: 98.0% (new package)
  • pkg/snapshotter: 51.1% → 51.3% (+0.2%)
  • pkg/recipe: 90.5% → 90.5% (unchanged; snapshot.go's 472 test lines deleted along with the function)
  • pkg/cli: unchanged (test data updated to canonical collector subtype/key shapes)

What was exercised locally

  • go build ./... — clean
  • go test ./pkg/fingerprint/... ./pkg/snapshotter/... ./pkg/recipe/... ./pkg/cli/... — all pass
  • golangci-lint -c .golangci.yaml run on the affected packages — 0 issues
  • tools/check-agents-sync, check-docs-sidebar, check-docs-filenames, check-docs-mdx — all pass
  • go vet — clean

What CI will exercise that I couldn't locally

  • make test with -race (sandbox lacks cgo/gcc; non-race pass clean)
  • make e2e, make scan (grype), lychee link check
  • tests/chainsaw/snapshot/deploy-agent against a real Kind cluster

Risk Assessment

  • Low — Additive top-level field, advisory by design, isolated package boundary

Rollout notes:

  • Snapshot YAML is forward-compatible. Old controllers reading
    new snapshots silently ignore the unknown fingerprint: field
    (Go yaml.v3 default behavior).
  • Image-version skew degrades gracefully. A stale agent image
    emits snapshots without the fingerprint block; new controllers
    recompute via fingerprint.FromMeasurements rather than trust the
    embedded value.
  • recipe.ExtractCriteriaFromSnapshot removed — internal-only
    function; the one external caller in pkg/cli/query.go is migrated
    in the same commit.
  • pkg/recipe.Criteria.ToFingerprintInput() and
    pkg/fingerprint.CriteriaInput removed
    — internal-only adapter
    types added then removed in the same PR; no external surface change.

Checklist

  • Tests pass locally (make test with -race) — non-race pass locally; -race deferred to CI (no gcc in sandbox)
  • Linter passes (make lint)
  • I did not skip/disable tests to make CI green
  • I added/updated tests for new functionality
  • I updated docs if user-facing behavior changed
  • Changes follow existing patterns in the codebase
  • Commits are cryptographically signed (git commit -S)

njhensley added 4 commits May 11, 2026 10:57
Adds pkg/fingerprint, a structured cluster identity (service,
accelerator, OS, k8s version, node count) derived from the existing
collector outputs and emitted alongside Snapshot.Measurements. Each
dimension records its source signal so an evidence bundle can prove
the recipe was tested on hardware matching its declared criteria,
per ADR-007.

- pkg/fingerprint: types, FromMeasurements, Match (per-dimension
  matched/mismatched/unknown diff), GPU SKU normalization
- pkg/snapshotter: populate Snapshot.Fingerprint after collectors
  finish so missing signals surface as zero-value dimensions
- pkg/recipe: Criteria.ToFingerprintInput adapter; switch OS aliases
  to oskind constants
- docs/contributor/data.md: schema, detection sources, match
  semantics, worked example
Collapses matchString/matchUnknownDim/matchNodes into a single
matchDim that takes a fingerprintCaptured flag for the
"deliberately not captured" vs "signal absent" distinction. Nodes
remap 0 to "" to reuse the same wildcard path.

No behavior change; all existing match tests pass unchanged.
Reads topology.kubernetes.io/region from the node-topology label
subtype. Multi-region clusters surface as empty rather than picking
one arbitrarily.
…rder match diffs

Lift snapshot→criteria responsibility from pkg/recipe into the
fingerprint package via Fingerprint.ToCriteria, dropping the
CriteriaInput indirection that previously bridged the import cycle.
Match now takes *recipe.Criteria directly.

Add GPUNodeCount sourced from nvidia.com/gpu.product topology labels
and reconcile per-node smi accelerator against cluster-wide labels —
heterogeneous clusters surface as multi-gpu via a new Dimension.Note
field rather than misreporting whichever SKU the snapshotter landed
on. Same Note path records multi-region when region labels disagree.

Change MatchResult.PerDimension from map to ordered slice with a
typed DimensionName enum and MatchResult.Find lookup so serialization
is byte-stable. Drop OS Version when the ID is unrecognized, and
flag Snapshot.Fingerprint as advisory — trust-bearing consumers must
recompute from Measurements.
@njhensley njhensley requested a review from a team as a code owner May 11, 2026 20:43
@njhensley njhensley self-assigned this May 11, 2026
@coderabbitai

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

njhensley added 3 commits May 11, 2026 14:06
Reframe the fingerprint description in data.md and the package
godoc as the cluster-identity dimensions used to bind a snapshot
to a recipe (not just a mirror of recipe criteria), and list the
full current set: service, accelerator, OS, K8s version, region,
total node count, and GPU node count.
NodeCount.Value == 0 is the "not captured" sentinel (e.g. when
the topology summary subtype is missing). The match diff was
rendering it as a literal "0" in FingerprintProvides while
the recipe side was already wildcarded to "" — the asymmetry
made the diff misleading. Remap both sides to "" so isAny in
matchDim treats them as wildcards consistently.
Exercise the topology-label backfill path used when smi did not
run (agent landed on a non-GPU node) but the GPU operator labels
nodes: an unrecognized product string must yield empty rather
than fabricating a SKU, matching the smi-side ParseGPUSKU
behavior.
coderabbitai[bot]

This comment was marked as resolved.

Add a callout warning that the fingerprint embedded in the
snapshot YAML is a human-readability convenience, not a
trust-bearing claim — the snapshot file is not signed at this
layer, so trust-bearing consumers (evidence bundler NVIDIA#754,
verifier NVIDIA#753, downstream policy gates) must recompute via
fingerprint.FromMeasurements before acting on it.
coderabbitai[bot]

This comment was marked as resolved.

Add a subsection explaining the multi-gpu / multi-region notes
the fingerprint emits when nodes disagree on accelerator or
region, and update the schema paragraph to cover the optional
note field alongside value and source. Distinguishes "nodes
disagreed" from "signal not captured" so readers know how
the verifier will render each case.
coderabbitai[bot]

This comment was marked as resolved.

@mchmarny mchmarny left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean implementation — the Fingerprint type with three-way DimensionMatch semantics and the Note-for-heterogeneity pattern are exactly the right shapes for an audit-trail data type. The advisory godoc on Snapshot.Fingerprint and the cycle inversion (consolidating two snapshot→criteria projections through one path) are both solid moves.

One medium nit: populateFromGPU and the topology-backfill path in reconcileAccelerator silently drop unrecognized GPU SKUs with no audit signal — would be more consistent with the multi-region/multi-gpu treatment to surface a note: unknown-sku so registry staleness shows up in the snapshot rather than masquerading as "no GPU." The remaining items are nits (label-value parsing duplicated three times, second linear scan in reconcileAccelerator, criteriaAnyValue const duplicated from pkg/recipe).

The k8s/GPU subtype/key renames in pkg/cli/recipe_test.go are a quiet bug fix — the deleted ExtractCriteriaFromSnapshot was reading keys that don't exist in real collector output. Worth a line in the PR description so reviewers don't read it as a pure refactor.

CI: most checks green, tests / E2E, analyze, and the GPU H100 jobs still in-flight at review time. mergeable_state: behind — needs a rebase before merge.

Not blocking on any of this.

Comment thread pkg/fingerprint/from_measurements.go
Comment thread pkg/fingerprint/from_measurements.go Outdated
Comment thread pkg/fingerprint/from_measurements.go Outdated
Comment thread pkg/fingerprint/match.go Outdated
Comment thread pkg/snapshotter/types.go
Comment thread pkg/cli/recipe_test.go
njhensley added 3 commits May 11, 2026 15:07
The fingerprint package's matchDim needs the same "any" wildcard
literal that pkg/recipe's parsers already use, and was carrying
its own duplicate const to avoid an import. Export
CriteriaAnyValue with godoc explaining its relationship to the
typed *Any enums, and drop the duplicate in pkg/fingerprint/match.go
in favor of recipe.CriteriaAnyValue.
…lookup

Two small refactors with no behavior change:

* Pre-find the NodeTopology measurement once in FromMeasurements
  and hand it to reconcileAccelerator, replacing reconcile's
  internal scan-and-nil-check loop. The two passes were redundant
  because populateFromTopology already iterates the measurement.

* Factor the topology collector's "<value>|<node1,node2,...>"
  label-value parsing into a parseLabelEncoding helper. Three
  call sites (reconcileAccelerator, countGPUNodes, extractRegion)
  open-coded the same strings.Index split with slightly different
  branching; folding them into one function makes the encoding
  contract obvious in one place.
…istry

When nvidia-smi or the GPU operator's topology label reports a
product string ParseGPUSKU does not recognize, the fingerprint
was silently dropping the Accelerator dimension — indistinguishable
from "no GPU detected" to downstream consumers. A reviewer
flagged this as a likely registry-staleness signal that should
surface, not hide.

Emit a new "unknown-sku" note on Accelerator in both paths:
  * populateFromGPU sets it from sourceAcceleratorSMI when smi
    reports a product outside the registry.
  * reconcileAccelerator sets it from sourceTopologyGPU when the
    topology label is present but unrecognized, unless smi
    already marked it (avoids overwriting an identical signal
    from a less-specific source). When topology resolves to a
    known SKU after smi marked unknown-sku, topology wins and
    clears the note (covered by the new
    SMIUnknownPlusTopologyRecognized test).

The raw model string stays in the underlying GPU measurement for
forensics; only the normalized fingerprint surface gains the
audit hint. docs/contributor/data.md picks up the third note
alongside the existing multi-region / multi-gpu cases.
@njhensley njhensley enabled auto-merge (squash) May 11, 2026 22:26
@njhensley njhensley merged commit 10da4f9 into NVIDIA:main May 11, 2026
34 checks passed
@njhensley njhensley deleted the feat/snapshot-fingerprint branch May 11, 2026 22:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Capture cluster fingerprint in snapshot for recipe-criteria binding

2 participants