Skip to content

refactor(recipe): add maximal leaf candidate selection to overlay resolver#496

Merged
yuanchen8911 merged 1 commit into
NVIDIA:mainfrom
yuanchen8911:refactor/maximal-leaf-candidate-selection
Apr 6, 2026
Merged

refactor(recipe): add maximal leaf candidate selection to overlay resolver#496
yuanchen8911 merged 1 commit into
NVIDIA:mainfrom
yuanchen8911:refactor/maximal-leaf-candidate-selection

Conversation

@yuanchen8911

@yuanchen8911 yuanchen8911 commented Apr 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Add maximal leaf candidate selection to FindMatchingOverlays — filter out any matching overlay that is an ancestor (via spec.base chain) of another matching overlay. Only the most-specific leaves survive as candidates.

Motivation / Context

The overlay resolver currently applies all matching overlays independently, then expands each match's inheritance chain. This works correctly with the current linear tree structure, but breaks when composition mechanisms (mixins, intermediates) introduce overlays at the same specificity level — ancestor overlays compete with their descendants, producing non-deterministic merge results.

This is Phase 2 of the revised ADR-005 (#439): stabilize resolver semantics before adding composition abstractions.

Fixes: N/A
Related: #439, #305, #492, #493

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update
  • Refactoring (no functional changes)
  • Build/CI/tooling

Component(s) Affected

  • CLI (cmd/aicr, pkg/cli)
  • API server (cmd/aicrd, pkg/api, pkg/server)
  • Recipe engine / data (pkg/recipe)
  • Bundlers (pkg/bundler, pkg/component/*)
  • Collectors / snapshotter (pkg/collector, pkg/snapshotter)
  • Validator (pkg/validator)
  • Core libraries (pkg/errors, pkg/k8s)
  • Docs/examples (docs/, examples/)
  • Other: ____________

Implementation Notes

filterToMaximalLeaves() is a shared helper called by FindMatchingOverlays, which is the entry point for both BuildRecipeResult and BuildRecipeResultWithEvaluator. This ensures consistent candidate selection regardless of call site.

The filter builds a set of all ancestors of matching overlays by walking each match's spec.base chain (with cycle detection), then removes any match whose name appears in that ancestor set.

Evaluator behavior change: With maximal leaf selection, ancestor overlays are no longer independent candidates in BuildRecipeResultWithEvaluator. When a leaf's constraints fail evaluation, the entire candidate is excluded — ancestors do not silently fill in as fallback candidates. The result falls back to base plus any other non-excluded leaf candidates (e.g., monitoring-hpa which matches intent: any and is not an ancestor of any other overlay). This is intentional and covered by TestEvaluatorFailingLeafExcludesCandidate.

Current tree impact: The existing overlay tree is linear — every matching overlay is an ancestor of the most-specific leaf. For BuildRecipeResult (no evaluator), the filter is a no-op: hydrated recipe content is unchanged for all current overlays. For BuildRecipeResultWithEvaluator, the fallback behavior changes as described above, but this path is only exercised when constraints fail against a snapshot.

appliedOverlays metadata change: 3 of 28 overlay combinations show minor ordering shifts in the appliedOverlays list. Hydrated recipe content is unchanged.

Testing

go test -race -v ./pkg/recipe/... -run 'TestMetadataStore_FindMatchingOverlays|TestBothBuildPathsProduceIdenticalContent|TestEvaluatorFailingLeafExcludesCandidate'
  • Unit tests for filterToMaximalLeaves: ancestor filtering with cycle detection, multiple branches, single match
  • Characterization test: all 14 leaf overlays discovered from recipes/overlays/ produce identical hydrated content through both BuildRecipeResult and BuildRecipeResultWithEvaluator (pass-all evaluator)
  • Failing-leaf test: verifies excluded leaf's ancestors are not used as fallback candidates, monitoring-hpa remains applied as independent non-ancestor leaf
  • Golden-file comparison of all 28 overlay combinations: zero content change

Risk Assessment

  • Medium — Touches multiple components or has broader impact

Rollout notes: The evaluator-path fallback behavior changes: when a leaf fails constraint evaluation, its ancestors no longer appear as independent candidates. This only affects BuildRecipeResultWithEvaluator when constraints fail. Revert by removing the filterToMaximalLeaves call in FindMatchingOverlays.

Checklist

  • Tests pass locally (make test with -race)
  • Linter passes (make lint)
  • I did not skip/disable tests to make CI green
  • I added/updated tests for new functionality
  • I updated docs if user-facing behavior changed
  • Changes follow existing patterns in the codebase
  • Commits are cryptographically signed (git commit -S) — GPG signing info

@yuanchen8911 yuanchen8911 requested a review from a team as a code owner April 6, 2026 21:20
@yuanchen8911 yuanchen8911 force-pushed the refactor/maximal-leaf-candidate-selection branch 2 times, most recently from 869f74f to e8e511b Compare April 6, 2026 21:27
@mchmarny mchmarny self-assigned this Apr 6, 2026

@mchmarny mchmarny left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean implementation — algorithm is correct and the behavior change is well-documented. One blocker (missing observability on the filter) and two small nits inline.

Comment thread pkg/recipe/metadata_store.go
Comment thread pkg/recipe/metadata_store_test.go
Comment thread pkg/recipe/metadata_store_test.go
@yuanchen8911 yuanchen8911 force-pushed the refactor/maximal-leaf-candidate-selection branch from e8e511b to 9fde20c Compare April 6, 2026 21:55
@yuanchen8911

Copy link
Copy Markdown
Contributor Author

All three items addressed:

  1. Blocker (observability): Added slog.Debug after filterToMaximalLeaves — logs removed/remaining counts when ancestors are pruned.
  2. Nit (cleanup safety): Switched to t.Cleanup for store mutation in the multi-branch test.
  3. Nit (comparison depth): reflect.DeepEqual on full ComponentRefs structs instead of name-only comparison.

All tests pass with -race.

@mchmarny — ready for another look.

@yuanchen8911 yuanchen8911 requested a review from mchmarny April 6, 2026 21:56
mchmarny
mchmarny previously approved these changes Apr 6, 2026

@mchmarny mchmarny left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@yuanchen8911 yuanchen8911 force-pushed the refactor/maximal-leaf-candidate-selection branch from 77819a6 to b14b49f Compare April 6, 2026 22:11
…gOverlays

Add filterToMaximalLeaves() to FindMatchingOverlays that removes any
matching overlay that is an ancestor (via spec.base chain) of another
matching overlay. Only the most-specific leaves survive as candidates;
their full inheritance chains are still resolved during merging.

This is a shared semantic change under both BuildRecipeResult and
BuildRecipeResultWithEvaluator — both use FindMatchingOverlays as the
entry point for candidate selection.

Evaluator behavior change: with maximal leaf selection, ancestor
overlays are no longer independent candidates. When a leaf's constraints
fail evaluation, the entire candidate is excluded — ancestors do not
silently fill in as fallback candidates. The result falls back to
base-only output plus any other non-excluded leaf candidates.

The current overlay tree is linear (every matching overlay is an
ancestor of the most-specific leaf), so this is a no-op for existing
recipes. It becomes load-bearing when composition mechanisms (mixins,
intermediates) introduce overlays at the same specificity level.

Add unit tests for the maximal leaf filter, a characterization test
that verifies all leaf overlays discovered from recipes/overlays/
produce identical hydrated content through both build paths, and a
failing-leaf test that verifies excluded candidates do not include
ancestors.

Signed-off-by: Yuan Chen <[email protected]>
@yuanchen8911 yuanchen8911 force-pushed the refactor/maximal-leaf-candidate-selection branch from b14b49f to 25762ee Compare April 6, 2026 22:18
@yuanchen8911 yuanchen8911 requested a review from mchmarny April 6, 2026 22:42
@yuanchen8911 yuanchen8911 merged commit abeea2b into NVIDIA:main Apr 6, 2026
21 of 22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants