Skip to content

Defer value loading, templating and inheritance by adding a third stage via a release value-template (#20)#21

Merged
max06 merged 1 commit into
mainfrom
worktree-stage3-values-prototype
May 1, 2026
Merged

Defer value loading, templating and inheritance by adding a third stage via a release value-template (#20)#21
max06 merged 1 commit into
mainfrom
worktree-stage3-values-prototype

Conversation

@max06

@max06 max06 commented May 1, 2026

Copy link
Copy Markdown
Owner

v0.3.0 (commit fad007e on add-selectors-to-subhelmfiles, merged via #16) added a hardcoded selectors: field to every emitted sub-helmfile entry on the assumption that helmfile/helmfile#2545 was available. In stock helmfile (1.4.4 and earlier — including the pre-1.0 versions consumers still run), child selectors: REPLACE the inherited CLI selector when filtering releases inside that sub-helmfile, so --selector became a no-op and every render produced manifests for every deployment in the repo. Verified empirically: 1.4.4 + --selector emitted 40 release dirs instead of the targeted 1.

The same change deleted ATLAS's stage-1 atlas.filter.* env path on the same wrong assumption, leaving consumers (especially the ArgoCD ApplicationSet pattern) with no efficient way to render a single deployment.

Restore both:

  • Drop the hardcoded sub-helmfile selectors: block — CLI --selector now reaches release-level commonLabels (cluster, clusterName, clusterGroup, deploymentName, template, instance, variant) again.
  • Re-add stage-1 atlas.filter.{cluster,deploymentName} short-circuit in helmfile.all.yaml.gotmpl — non-matching clusters skip FS scan, deployment collection, and sub-helmfile emission entirely. Wired from env (ATLAS_FILTER_CLUSTER / ATLAS_FILTER_DEPLOYMENT_NAME) at the top-level helmfile.yaml.gotmpl so consumers can pass it via helmfile -e ... or directly from an ArgoCD Application's container env.
  • selectorsInherited: true removed — pointless without child selectors to inherit through.

Tests: tests/bats/filtering.bats covers both contracts (CLI --selector

  • env-var stage-1 filter) with positive and negative assertions, so the v0.3.0 regression cannot recur silently. helmfile_list helper drops stderr because bats's run captures both streams and helmfile's "err: no releases found..." would otherwise corrupt the JSON-parse assertion for empty-result selectors.

Full suite: 290/290.

snapshot-review: leverage stage-1 atlas.filter.* env vars in fast path

The reusable PR-review workflow's fast path renders one (cluster, deploymentName) pair at a time with --selector. Pre-fix, every invocation parsed every sub-helmfile and decrypted SOPS for unrelated deployments — the --selector only filtered AFTER the work was done.

With the stage-1 atlas.filter.* env path restored, ATLAS_FILTER_CLUSTER and ATLAS_FILTER_DEPLOYMENT_NAME short-circuit in helmfile.all so non-matching sub-helmfiles never get emitted in the first place. helmfile.single only sees one entry, SOPS only loads for that one deployment.

Both env vars AND --selector are passed:

  • Current ATLAS: env vars short-circuit at stage-1, --selector is a redundant safety net.
  • Older ATLAS pinned by consumer (no stage-1 filter): env vars are no-op, --selector handles release-level filtering.

Probe + bulk fallback path is untouched — still detects pinned ATLAS refs that carry the broken hardcoded sub-helmfile selectors and falls back to a single bulk render. Probe diagnostic and warning text updated to reflect that the broken behavior is now historical, not the helmfile binary's fault.

Two near-identical edits: the merge-result render (PR side) and the baseline render (target branch HEAD). Both fast paths now export the env vars per pair.

Add stage-3 release-aware values pipeline (prototype)

Introduces a third pipeline stage where release values are resolved at helmfile's release-evaluation time rather than at stage-2. The new path makes .Release.Name / .Release.Namespace available inside values gotmpls, which the existing stage-2 progressive-merge cannot offer.

Pipeline shape:
stage 1 (helmfile.all) -> per-deployment sub-helmfiles
stage 2 (helmfile.single) -> per-instance sub-helmfiles (when engine: stage3)
stage 3 (helmfile.instance) -> emits releases with values: []
stage 3' (values-loader) -> resolves values per-release with .Release.* in scope

Each app instance must live in its own helmfile file because multiple environments: blocks in one file collapse to a single merged env-values (verified by spike). Splitting by instance keeps atlas.instance unambiguous and removes the need for a release-name lookup table.

Deployments opt in via top-level engine: stage3 in deployment.yaml. Existing deployments without the flag run through the unchanged stage-2 path; all 284 prior assertions stay green.

Prototype scope (deferred to follow-ups):

  • SOPS decryption in the loader
  • Redaction twin-load + post-renderer wiring
  • applyListOverride for patches/transformers in stage-3
  • .yaml.gotmpl + group-level hierarchy variants in the loader
  • Progressive accumulator across the values: list

Stage-3 loader: complete hierarchy walk (.sops.yaml + .yaml + .yaml.gotmpl)

Extends the values loader's hierarchy walk to mirror stage-2's full source order at every level (global, group, cluster, deployment). Each level now loads .sops.yaml -> .yaml -> .yaml.gotmpl with later sources overriding earlier ones. .yaml.gotmpl files are tpl-rendered with a fresh ctx that exposes the running merge so a .gotmpl at one level can reference any key defined earlier — including SOPS-decrypted values.

SOPS at the hierarchy level was bundled in (rather than deferred to the template/instance SOPS milestone) because every existing hierarchy .yaml.gotmpl fixture references SOPS keys; .yaml.gotmpl is unusable without it. Template- and instance-level SOPS files remain in the next milestone.

deployment31 (group1/cluster2 + engine: stage3) covers:

  • plain .yaml at all four hierarchy levels
  • .yaml.gotmpl variants at every level
  • .yaml.gotmpl chaining through earlier .yaml and earlier .gotmpl
  • override precedence (group beats global, cluster beats group)
  • nested deep merge across the four levels

309/309 bats green.

Stage-3 loader: progressive accumulator across the values: list

Reorders the loader so hierarchy is computed first as a baseline ctx for the template + instance walks, then overlaid at the end so it still wins precedence (chart < template < instance < hierarchy is preserved). Each .yaml.gotmpl entry in a release's values: list is now tpl-rendered with a fresh ctx that exposes:

  • .Values (atlas baseline)
  • the hierarchy as already loaded
  • the running merge of all earlier entries in the same values: list
  • .Release.* (Name, Namespace, Chart, Labels)
  • .Values.foo self-ref so wrapped form ({{ .Values.foo }}) keeps working

A later entry can therefore reference any key defined by an earlier inline map, plain .yaml, or .yaml.gotmpl — same semantics stage-2 already provides via its progressive twin-load merge, only this version is release-aware.

deployment32 (cluster1 + engine: stage3) covers:

  • inline + plain .yaml + .yaml.gotmpl in one values: list
  • .gotmpl reading earlier inline keys
  • .gotmpl reading earlier plain .yaml keys
  • .gotmpl mixing hierarchy + earlier-list keys
  • .gotmpl using .Release.Name alongside earlier keys
  • .gotmpl using .Values.foo (wrapped) form

317/317 bats green.

Stage-3 named instances: cover via test, no loader change needed

deployment33 instantiates the same app template twice in one deployment via apps[].name (primary, secondary). Stage-2's per-instance fan-out already gives each instance its own helmfile state file with a fully populated atlas.instance.name in env-values; the loader's existing instance-matcher already keys on (template, instance-name). The whole flow worked on the first run with no loader edit — adding the fixture locks in the behaviour as a regression guard.

deployment33 covers:

  • both releases render (primary, secondary)
  • each release sees its own atlas.instance.name in the values loader
  • release name correctly embeds the instance name
  • shared template name surfaces identically on both

325/325 bats green.

Stage-3 loader: SOPS in template + instance values: lists

Adds a .sops.yaml branch to the template-level and instance-level walks in the values loader. fetchSecretValue with the ref+sops:// protocol performs whole-document decryption — same primitive used at the hierarchy level (commit 38500a4) and by stage-2. List-order precedence is preserved: a .sops.yaml listed after an inline map overrides the inline keys.

deployment34 reuses the existing app-tplsops template (values.yaml.gotmpl

  • inline map + values.sops.yaml) through engine: stage3. Asserts:
    • plain template-include value reaches output
    • template-inline default reaches output
    • SOPS scalar decrypts
    • SOPS nested map decrypts (preserves nested structure)
    • SOPS value listed last overrides earlier inline of the same key

The instance-level SOPS branch is added symmetrically to the loader but no existing fixture exercises it; covered when a test surfaces.

Redaction of these template-level SOPS values is deferred to feature #6 (twin-load + post-renderer wiring).

330/330 bats green.

Stage-3 redaction: shared atlas.values.merged + twin-load in stage-3

Extracts the loader's merge logic into a single named template (atlas.values.merged in templates/_values_loader.tpl) parameterised by .redact. Two callers share the same body:

helmfile.values-loader.yaml.gotmpl
Helmfile evaluates this once per release at release-time. Calls
atlas.values.merged with redact=false and emits the result as the
real values blob for helm.

helmfile.instance.yaml.gotmpl (per release, when redactSecrets is on)
Calls atlas.values.merged twice — once with redact=false (real),
once with redact=true (SOPS values rewritten through atlas.redact.value
before merge, transitive taint via twin-load). Diffs the two trees via
atlas.diff.values to produce a replacement map, b64-encodes it into
postRendererArgs, and wires the atlas-redact post-renderer.

Redaction-mode runs walk twice per release (real + redacted) plus once at release-time (loader). Regular ArgoCD renders skip the entire stage-3 twin-load block — only the loader runs. So the redaction overhead is paid only by the CI/redacted-render workflow, never by ArgoCD.

deployment34 now also asserts:

  • template SOPS scalar redacted
  • template SOPS override redacted (list-order precedence preserved)
  • template SOPS nested map redacted at every leaf
  • plain (non-SOPS) values pass through untouched

1MB ARG_MAX safety check ported from stage-2. Sidedump path (used by the snapshot-review workflow) deferred — not needed for prototype validation.

335/335 bats green.

Stage-3: strip release.condition and release.installed

In ATLAS, deploy intent is signalled by the presence of a deployment.yaml entry. Letting helmfile-level condition/installed through would silently produce empty deployments — indistinguishable from a misconfiguration — when ArgoCD discovers the directory and finds no manifests.

Drop both fields in the per-instance release rewrite step so app templates remain vanilla-helmfile-readable but render uniformly under ATLAS.

Fixture: deployment35 / app-stage3-condinst sets condition: never.enabled (would be a parse error in vanilla helmfile) and installed: false (would be silently skipped). With the strip in place both render normally.

Stage-3: add template + instance commonLabels

Auto-inject two labels on every stage-3 release so consumers can filter by app-template family and named instance:

  • template: the app-template directory name
  • instance: the apps[].name value (or template name when name unset)

Useful for ArgoCD ApplicationSet selectors and helmfile list -l .... The remaining commonLabels (cluster, clusterName, deploymentName, variant, clusterGroup) are unchanged.

Test covers both the default-name path (deployment30 — instance defaults to template) and the explicit-name path (deployment33 — primary and secondary instances of the same template).

Stage-3: auto-munge release names with instance.name

When apps[].name is set and differs from the template name, ATLAS rewrites release names so each instance produces a distinct release without forcing the template author to embed {{ .Values.atlas.instance.name }}. Default style is <instance>-<release> (prefix); apps[].nameStyle: suffix flips to <release>-<instance> for naming conventions like vm-cust-abc.

Both helmfile.instance and the values-loader apply the same munge logic: helmfile.instance rewrites the emitted release.name; the loader applies the same transform when matching the template's author-intended name against helmfile's .Release.Name (now the munged form).

Existing fixture deployment33/app-stage3-named drops its manual {{ .Values.atlas.instance.name }} munge — the template author intent is the bare release name "stage3-named" and ATLAS produces "primary-stage3-named" / "secondary-stage3-named" automatically.

New fixture deployment36/app-stage3-namestyle covers all three flavours in one deployment: default-prefix, explicit-prefix, suffix.

Stage-3: fail render when template sets release.postRenderer

ATLAS owns the helm post-renderer slot for its redaction pipeline. Helm 4 does not support post-renderer chaining, and silently overwriting a user- set postRenderer would mask intent and produce confusing diffs. Reject the field with a clear pointer to open an issue so the use case can be discussed before any workaround.

Negative-test fixture lives under tests/fixtures-negative/ so the standard bulk render in render.bash is unaffected. The bats test invokes helmfile directly with its own --state-values-set roots.

Stage-3: default skipSchemaValidation to true (preserve user value)

helm runs schema validation only on install/upgrade/lint, not the template/build paths ATLAS exercises during ArgoCD render. The flag primarily affects consumer-side bootstrap, where missing CRDs are a common false-positive failure source. Default it on so that path stays clean. User-set values (true or false) are preserved unchanged.

Fixture deployment37 / app-stage3-schema covers both branches: a release without the field gets defaulted to true; a release that sets false explicitly keeps false.

Stage-3: re-accept release.secrets, merge last, path-resolve

Helmfile's env-level secrets contract: "non-HCL secrets are loaded first but merged last". ATLAS replicates this at the release level — every entry in release.secrets and apps[].secrets lands AFTER all values: entries. Paths resolve template-relative for release.secrets and deployment-relative for apps[].secrets. Decryption goes through the existing SOPS fetchSecretValue path (same as .sops.yaml in values:), so the helm-secrets plugin is not required.

helmfile.instance strips release.secrets from the emitted state once the loader has handled them, so helmfile does not also try to decrypt via helm-secrets (avoiding duplicate work and a hard plugin dependency).

Fixture deployment38 / app-stage3-secrets demonstrates all three layers: template values, template secrets (overrides values), instance secrets (overrides template secrets).

tests/.sops.yaml: relax creation-rule regex from values\.sops\.yaml$ to \.sops\.yaml$ so secret fixtures named for their role (template-secret.sops.yaml, instance-secret.sops.yaml) get the testing-age recipient automatically.

Stage-3: wire applyListOverride for patches/transformers

Route release.strategicMergePatches, release.jsonPatches, and release.transformers through atlas.applyListOverride during the per- instance rewrite, mirroring stage-2's behaviour. Template-relative paths become absolute so helmfile can locate the patch files regardless of consumer CWD; instance-level overrides from deployment.yaml apps[] are appended.

release.values is intentionally excluded — stage-3's loader replaces it wholesale and resolves entries itself via the values-loader pipeline.

Fixture deployment39 / app-stage3-patches uses a strategicMergePatch to add an annotation onto chart1's ConfigMap, exercising both the path resolution wiring and helmfile's actual patch application via kustomize.

Note: convertPaths joins targetPath with the original entry verbatim, so "./foo" becomes "

/./foo" rather than the cleaner "/foo". kustomize accepts either; normalization is left for a follow-up.

Stage-3: path-resolve keyring + set[].file + setString[].file

Resolve template-relative paths for the remaining helm-CLI fields that take a path:
keyring — single scalar (passed to helm as --keyring)
set[].file — nested key inside a list of maps (--set-file)
setString[].file — same shape as set[]

Mirrors the chart: handling — only ./ and ../ prefixes are rewritten; absolute paths and bare names pass through. Without resolution helm would look in the consumer's CWD instead of the template dir.

Fixture deployment40 / app-stage3-paths exercises all three; the bats test caches the build-state YAML once via setup_file (helmfile build is expensive) and asserts each path is absolute and points inside the template dir.

Stage-3: flip default engine, port missing wiring, migrate d9 + d11

helmfile.single now defaults to engine: stage3 when deployment.yaml omits the field. All 1-29 fixtures (and any consumer that doesn't opt in explicitly) flow through the per-instance pipeline.

Three pieces of stage-2 functionality the previous parallel-fixture work hadn't touched moved over so the existing suite stays green:

  • _values_loader: seed $merged with .Values.atlas so the chart sees .Values.atlas.* (deployment context, instance fields, redactSecrets) just as stage-2 did. Mirrors the dict "atlas" .Values.atlas baseline in helmfile.single.

  • helmfile.instance: merge apps[].labels into release.labels with instance taking priority. atlas-template's bootstrap selector (bootstrap: true) depends on this.

  • helmfile.instance: optional ATLAS_SIDEDUMP_MAP_DIR side-dump second argv element on the atlas-redact post-renderer. snapshot-review's scrub-baseline replays each map against an older-baseline render.

Fixture migration:

  • tests/templates/app-named: drop the manual {{ .Values.atlas.instance.name }} munge from release.name; ATLAS auto-munges now (slice 3). Release becomes "primary-app-named" / "secondary-app-named" via prefix.
  • cluster1-deployment9-{primary,secondary}.bats: update INSTANCE to the munged form.

Full suite: 369/369 with stage-3 as default.

Retire stage-2 path; thin helmfile.single to fan-out shim

helmfile.single now does only what the stage-3 default needs:

  1. Validate deployment.yaml exists.
  2. Render deployment.yaml as a Go template with the atlas object as context, parse the resulting apps[] list.
  3. Emit one helmfiles[] entry per app instance, each pointing at helmfile.instance.yaml.gotmpl with the atlas object augmented by instance.{template, name}.

Removed (no longer reachable now that engine: stage2 is gone):

  • hierarchy twin-load ($realValues / $redactedValues)
  • hierarchy-level $replMap construction
  • environments.default.values emission
  • per-app-template release rewriting (chart paths, secrets rejection, progressive twin-load merge, applyListOverride wiring, redaction post-renderer wiring with side-dump)
  • the engine if/else branch and the get "engine" "stage2" default

The deleted block weighed ~400 lines; helmfile.single is 75 lines now. All of that logic lives in either helmfile.instance (per-release rewrites, redaction post-renderer, side-dump) or _values_loader (twin- load, hierarchy walk, progressive merge).

Cleanup:

  • Drop explicit engine: stage3 from fixtures 30-40 + postrenderer-fail.
  • Update doc-comments that referenced stage-2 (helmfile.instance, _values_loader, _functions, helmfile.all, app-stage3-probe).

Full suite: 369/369.

docs: align README and CLAUDE.md with stage-3-only architecture

README.md

  • "Using the same template multiple times": drop the manual {{ .Values.atlas.instance.name }} instruction. Document auto-munge with apps[].nameStyle: prefix|suffix (prefix default).
  • "Labels": expand the commonLabels table to cover clusterName, clusterGroup, template, instance, variant — all auto-injected now.
  • "App Templates": add a "How ATLAS treats helmfile release options" table covering the per-instance rewrites (name, chart, values, secrets, condition/installed, skipSchemaValidation, postRenderer, keyring + set[].file + setString[].file, patches/transformers, labels). Lead-in clarifies the "stay close to vanilla helmfile" goal.
  • "Value Inheritance Logic": insert template-secrets and instance- secrets layers between values and hierarchy. Document the "secrets merged last among per-release lists" semantics.
  • "Full Load Order": renumber to include the new secrets layers (5-6); standalone-cluster skip range shifts to 10-12.
  • "How It Works": replace the two-step (load + render) description with the four-stage pipeline (discover, fan-out, per-instance rewrite, release-time loader) that matches the actual code now.

CLAUDE.md

  • Common Commands: switch to the subcommand-first invocation form (helmfile template -f X / helmfile build -f X) so the commands match the prefix-based allowlist patterns.
  • Core Template Pipeline: describe all four stages explicitly (helmfile.all → helmfile.single → helmfile.instance → loader) instead of conflating values resolution into helmfile.single.

Add .claude/utils/run-bats.sh + allowlist

Wraps the test ritual (rm stale tmp dirs, run bats, count results, list failures) into one allowlistable invocation. Modes:

.claude/utils/run-bats.sh full suite, summary
.claude/utils/run-bats.sh tests/bats/foo.bats single file, full output
.claude/utils/run-bats.sh -v full suite, full output

Exit code reflects test result (0 pass, 1 fail), so CI/agents can use status without parsing output.

Allowlist additions: Bash(.claude/utils/run-bats.sh) and the -form for arguments. The script's internal rm of /tmp/atlas-bats- runs without prompting because the script itself is the allowlisted unit.

Also rolls in the broader allowlist set added on main (bats *, sops --decrypt *, helm template *, helmfile template *, helmfile build *, etc.) so the worktree shares the same prompt-free surface as main.

v0.3.0 (commit fad007e on add-selectors-to-subhelmfiles, merged via #16)
added a hardcoded `selectors:` field to every emitted sub-helmfile entry
on the assumption that helmfile/helmfile#2545 was available. In stock
helmfile (1.4.4 and earlier — including the pre-1.0 versions consumers
still run), child `selectors:` REPLACE the inherited CLI selector when
filtering releases inside that sub-helmfile, so `--selector` became a
no-op and every render produced manifests for every deployment in the
repo. Verified empirically: 1.4.4 + --selector emitted 40 release dirs
instead of the targeted 1.

The same change deleted ATLAS's stage-1 atlas.filter.* env path on the
same wrong assumption, leaving consumers (especially the ArgoCD
ApplicationSet pattern) with no efficient way to render a single
deployment.

Restore both:
* Drop the hardcoded sub-helmfile `selectors:` block — CLI --selector
now reaches release-level commonLabels (cluster, clusterName,
clusterGroup, deploymentName, template, instance, variant) again.
* Re-add stage-1 atlas.filter.{cluster,deploymentName} short-circuit in
helmfile.all.yaml.gotmpl — non-matching clusters skip FS scan,
deployment collection, and sub-helmfile emission entirely. Wired from
env (ATLAS_FILTER_CLUSTER / ATLAS_FILTER_DEPLOYMENT_NAME) at the
top-level helmfile.yaml.gotmpl so consumers can pass it via `helmfile -e
...` or directly from an ArgoCD Application's container env.
* `selectorsInherited: true` removed — pointless without child selectors
to inherit through.

Tests: tests/bats/filtering.bats covers both contracts (CLI --selector
+ env-var stage-1 filter) with positive and negative assertions, so the
v0.3.0 regression cannot recur silently. helmfile_list helper drops
stderr because bats's `run` captures both streams and helmfile's "err:
no releases found..." would otherwise corrupt the JSON-parse assertion
for empty-result selectors.

Full suite: 290/290.

snapshot-review: leverage stage-1 atlas.filter.* env vars in fast path

The reusable PR-review workflow's fast path renders one (cluster,
deploymentName) pair at a time with --selector. Pre-fix, every
invocation parsed every sub-helmfile and decrypted SOPS for unrelated
deployments — the --selector only filtered AFTER the work was done.

With the stage-1 atlas.filter.* env path restored, ATLAS_FILTER_CLUSTER
and ATLAS_FILTER_DEPLOYMENT_NAME short-circuit in helmfile.all so
non-matching sub-helmfiles never get emitted in the first place.
helmfile.single only sees one entry, SOPS only loads for that one
deployment.

Both env vars AND --selector are passed:
- Current ATLAS: env vars short-circuit at stage-1, --selector is a
redundant safety net.
- Older ATLAS pinned by consumer (no stage-1 filter): env vars are
no-op, --selector handles release-level filtering.

Probe + bulk fallback path is untouched — still detects pinned ATLAS
refs that carry the broken hardcoded sub-helmfile selectors and falls
back to a single bulk render. Probe diagnostic and warning text updated
to reflect that the broken behavior is now historical, not the helmfile
binary's fault.

Two near-identical edits: the merge-result render (PR side) and the
baseline render (target branch HEAD). Both fast paths now export the env
vars per pair.

Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>

Add stage-3 release-aware values pipeline (prototype)

Introduces a third pipeline stage where release values are resolved at
helmfile's release-evaluation time rather than at stage-2. The new path
makes .Release.Name / .Release.Namespace available inside values gotmpls,
which the existing stage-2 progressive-merge cannot offer.

Pipeline shape:
  stage 1 (helmfile.all)      -> per-deployment sub-helmfiles
  stage 2 (helmfile.single)   -> per-instance sub-helmfiles (when engine: stage3)
  stage 3 (helmfile.instance) -> emits releases with values: [<loader>]
  stage 3' (values-loader)    -> resolves values per-release with .Release.* in scope

Each app instance must live in its own helmfile file because multiple
environments: blocks in one file collapse to a single merged env-values
(verified by spike). Splitting by instance keeps atlas.instance unambiguous
and removes the need for a release-name lookup table.

Deployments opt in via top-level `engine: stage3` in deployment.yaml.
Existing deployments without the flag run through the unchanged stage-2
path; all 284 prior assertions stay green.

Prototype scope (deferred to follow-ups):
  - SOPS decryption in the loader
  - Redaction twin-load + post-renderer wiring
  - applyListOverride for patches/transformers in stage-3
  - .yaml.gotmpl + group-level hierarchy variants in the loader
  - Progressive accumulator across the values: list

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Stage-3 loader: complete hierarchy walk (.sops.yaml + .yaml + .yaml.gotmpl)

Extends the values loader's hierarchy walk to mirror stage-2's full source
order at every level (global, group, cluster, deployment). Each level now
loads .sops.yaml -> .yaml -> .yaml.gotmpl with later sources overriding
earlier ones. .yaml.gotmpl files are tpl-rendered with a fresh ctx that
exposes the running merge so a .gotmpl at one level can reference any key
defined earlier — including SOPS-decrypted values.

SOPS at the hierarchy level was bundled in (rather than deferred to the
template/instance SOPS milestone) because every existing hierarchy
.yaml.gotmpl fixture references SOPS keys; .yaml.gotmpl is unusable
without it. Template- and instance-level SOPS files remain in the next
milestone.

deployment31 (group1/cluster2 + engine: stage3) covers:
  - plain .yaml at all four hierarchy levels
  - .yaml.gotmpl variants at every level
  - .yaml.gotmpl chaining through earlier .yaml and earlier .gotmpl
  - override precedence (group beats global, cluster beats group)
  - nested deep merge across the four levels

309/309 bats green.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Stage-3 loader: progressive accumulator across the values: list

Reorders the loader so hierarchy is computed first as a baseline ctx for
the template + instance walks, then overlaid at the end so it still wins
precedence (chart < template < instance < hierarchy is preserved). Each
.yaml.gotmpl entry in a release's values: list is now tpl-rendered with a
fresh ctx that exposes:

  - .Values (atlas baseline)
  - the hierarchy as already loaded
  - the running merge of all earlier entries in the same values: list
  - .Release.* (Name, Namespace, Chart, Labels)
  - .Values.foo self-ref so wrapped form ({{ .Values.foo }}) keeps working

A later entry can therefore reference any key defined by an earlier inline
map, plain .yaml, or .yaml.gotmpl — same semantics stage-2 already
provides via its progressive twin-load merge, only this version is
release-aware.

deployment32 (cluster1 + engine: stage3) covers:
  - inline + plain .yaml + .yaml.gotmpl in one values: list
  - .gotmpl reading earlier inline keys
  - .gotmpl reading earlier plain .yaml keys
  - .gotmpl mixing hierarchy + earlier-list keys
  - .gotmpl using .Release.Name alongside earlier keys
  - .gotmpl using .Values.foo (wrapped) form

317/317 bats green.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Stage-3 named instances: cover via test, no loader change needed

deployment33 instantiates the same app template twice in one deployment
via apps[].name (primary, secondary). Stage-2's per-instance fan-out
already gives each instance its own helmfile state file with a fully
populated atlas.instance.name in env-values; the loader's existing
instance-matcher already keys on (template, instance-name). The whole
flow worked on the first run with no loader edit — adding the fixture
locks in the behaviour as a regression guard.

deployment33 covers:
  - both releases render (primary, secondary)
  - each release sees its own atlas.instance.name in the values loader
  - release name correctly embeds the instance name
  - shared template name surfaces identically on both

325/325 bats green.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Stage-3 loader: SOPS in template + instance values: lists

Adds a .sops.yaml branch to the template-level and instance-level walks
in the values loader. fetchSecretValue with the ref+sops:// protocol
performs whole-document decryption — same primitive used at the hierarchy
level (commit 38500a4) and by stage-2. List-order precedence is preserved:
a .sops.yaml listed after an inline map overrides the inline keys.

deployment34 reuses the existing app-tplsops template (values.yaml.gotmpl
+ inline map + values.sops.yaml) through engine: stage3. Asserts:
  - plain template-include value reaches output
  - template-inline default reaches output
  - SOPS scalar decrypts
  - SOPS nested map decrypts (preserves nested structure)
  - SOPS value listed last overrides earlier inline of the same key

The instance-level SOPS branch is added symmetrically to the loader but
no existing fixture exercises it; covered when a test surfaces.

Redaction of these template-level SOPS values is deferred to feature #6
(twin-load + post-renderer wiring).

330/330 bats green.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Stage-3 redaction: shared atlas.values.merged + twin-load in stage-3

Extracts the loader's merge logic into a single named template
(atlas.values.merged in templates/_values_loader.tpl) parameterised by
.redact. Two callers share the same body:

  helmfile.values-loader.yaml.gotmpl
    Helmfile evaluates this once per release at release-time. Calls
    atlas.values.merged with redact=false and emits the result as the
    real values blob for helm.

  helmfile.instance.yaml.gotmpl  (per release, when redactSecrets is on)
    Calls atlas.values.merged twice — once with redact=false (real),
    once with redact=true (SOPS values rewritten through atlas.redact.value
    before merge, transitive taint via twin-load). Diffs the two trees via
    atlas.diff.values to produce a replacement map, b64-encodes it into
    postRendererArgs, and wires the atlas-redact post-renderer.

Redaction-mode runs walk twice per release (real + redacted) plus once at
release-time (loader). Regular ArgoCD renders skip the entire stage-3
twin-load block — only the loader runs. So the redaction overhead is paid
only by the CI/redacted-render workflow, never by ArgoCD.

deployment34 now also asserts:
  - template SOPS scalar redacted
  - template SOPS override redacted (list-order precedence preserved)
  - template SOPS nested map redacted at every leaf
  - plain (non-SOPS) values pass through untouched

1MB ARG_MAX safety check ported from stage-2. Sidedump path (used by the
snapshot-review workflow) deferred — not needed for prototype validation.

335/335 bats green.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Stage-3: strip release.condition and release.installed

In ATLAS, deploy intent is signalled by the presence of a deployment.yaml
entry. Letting helmfile-level condition/installed through would silently
produce empty deployments — indistinguishable from a misconfiguration —
when ArgoCD discovers the directory and finds no manifests.

Drop both fields in the per-instance release rewrite step so app templates
remain vanilla-helmfile-readable but render uniformly under ATLAS.

Fixture: deployment35 / app-stage3-condinst sets condition: never.enabled
(would be a parse error in vanilla helmfile) and installed: false (would
be silently skipped). With the strip in place both render normally.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Stage-3: add template + instance commonLabels

Auto-inject two labels on every stage-3 release so consumers can filter
by app-template family and named instance:
  - template: the app-template directory name
  - instance: the apps[].name value (or template name when name unset)

Useful for ArgoCD ApplicationSet selectors and `helmfile list -l ...`.
The remaining commonLabels (cluster, clusterName, deploymentName, variant,
clusterGroup) are unchanged.

Test covers both the default-name path (deployment30 — instance defaults
to template) and the explicit-name path (deployment33 — primary and
secondary instances of the same template).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Stage-3: auto-munge release names with instance.name

When apps[].name is set and differs from the template name, ATLAS rewrites
release names so each instance produces a distinct release without forcing
the template author to embed `{{ .Values.atlas.instance.name }}`. Default
style is `<instance>-<release>` (prefix); apps[].nameStyle: suffix flips
to `<release>-<instance>` for naming conventions like `vm-cust-abc`.

Both helmfile.instance and the values-loader apply the same munge logic:
helmfile.instance rewrites the emitted release.name; the loader applies
the same transform when matching the template's author-intended name
against helmfile's .Release.Name (now the munged form).

Existing fixture deployment33/app-stage3-named drops its manual
`{{ .Values.atlas.instance.name }}` munge — the template author intent
is the bare release name "stage3-named" and ATLAS produces
"primary-stage3-named" / "secondary-stage3-named" automatically.

New fixture deployment36/app-stage3-namestyle covers all three flavours
in one deployment: default-prefix, explicit-prefix, suffix.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Stage-3: fail render when template sets release.postRenderer

ATLAS owns the helm post-renderer slot for its redaction pipeline. Helm 4
does not support post-renderer chaining, and silently overwriting a user-
set postRenderer would mask intent and produce confusing diffs. Reject
the field with a clear pointer to open an issue so the use case can be
discussed before any workaround.

Negative-test fixture lives under tests/fixtures-negative/ so the
standard bulk render in render.bash is unaffected. The bats test invokes
helmfile directly with its own --state-values-set roots.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Stage-3: default skipSchemaValidation to true (preserve user value)

helm runs schema validation only on install/upgrade/lint, not the
template/build paths ATLAS exercises during ArgoCD render. The flag
primarily affects consumer-side bootstrap, where missing CRDs are a
common false-positive failure source. Default it on so that path stays
clean. User-set values (true or false) are preserved unchanged.

Fixture deployment37 / app-stage3-schema covers both branches: a release
without the field gets defaulted to true; a release that sets false
explicitly keeps false.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Stage-3: re-accept release.secrets, merge last, path-resolve

Helmfile's env-level secrets contract: "non-HCL secrets are loaded first
but merged last". ATLAS replicates this at the release level — every
entry in release.secrets and apps[].secrets lands AFTER all values:
entries. Paths resolve template-relative for release.secrets and
deployment-relative for apps[].secrets. Decryption goes through the
existing SOPS fetchSecretValue path (same as .sops.yaml in values:),
so the helm-secrets plugin is not required.

helmfile.instance strips release.secrets from the emitted state once the
loader has handled them, so helmfile does not also try to decrypt via
helm-secrets (avoiding duplicate work and a hard plugin dependency).

Fixture deployment38 / app-stage3-secrets demonstrates all three layers:
template values, template secrets (overrides values), instance secrets
(overrides template secrets).

tests/.sops.yaml: relax creation-rule regex from `values\.sops\.yaml$`
to `\.sops\.yaml$` so secret fixtures named for their role
(template-secret.sops.yaml, instance-secret.sops.yaml) get the
testing-age recipient automatically.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Stage-3: wire applyListOverride for patches/transformers

Route release.strategicMergePatches, release.jsonPatches, and
release.transformers through atlas.applyListOverride during the per-
instance rewrite, mirroring stage-2's behaviour. Template-relative paths
become absolute so helmfile can locate the patch files regardless of
consumer CWD; instance-level overrides from deployment.yaml apps[] are
appended.

release.values is intentionally excluded — stage-3's loader replaces it
wholesale and resolves entries itself via the values-loader pipeline.

Fixture deployment39 / app-stage3-patches uses a strategicMergePatch to
add an annotation onto chart1's ConfigMap, exercising both the path
resolution wiring and helmfile's actual patch application via kustomize.

Note: convertPaths joins targetPath with the original entry verbatim, so
"./foo" becomes "<dir>/./foo" rather than the cleaner "<dir>/foo".
kustomize accepts either; normalization is left for a follow-up.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Stage-3: path-resolve keyring + set[].file + setString[].file

Resolve template-relative paths for the remaining helm-CLI fields that
take a path:
  keyring          — single scalar (passed to helm as --keyring)
  set[].file       — nested key inside a list of maps (--set-file)
  setString[].file — same shape as set[]

Mirrors the chart: handling — only ./ and ../ prefixes are rewritten;
absolute paths and bare names pass through. Without resolution helm
would look in the consumer's CWD instead of the template dir.

Fixture deployment40 / app-stage3-paths exercises all three; the bats
test caches the build-state YAML once via setup_file (helmfile build is
expensive) and asserts each path is absolute and points inside the
template dir.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Stage-3: flip default engine, port missing wiring, migrate d9 + d11

helmfile.single now defaults to engine: stage3 when deployment.yaml omits
the field. All 1-29 fixtures (and any consumer that doesn't opt in
explicitly) flow through the per-instance pipeline.

Three pieces of stage-2 functionality the previous parallel-fixture work
hadn't touched moved over so the existing suite stays green:

  * _values_loader: seed $merged with .Values.atlas so the chart sees
    .Values.atlas.* (deployment context, instance fields, redactSecrets)
    just as stage-2 did. Mirrors the dict "atlas" .Values.atlas baseline
    in helmfile.single.

  * helmfile.instance: merge apps[].labels into release.labels with
    instance taking priority. atlas-template's bootstrap selector
    (bootstrap: true) depends on this.

  * helmfile.instance: optional ATLAS_SIDEDUMP_MAP_DIR side-dump second
    argv element on the atlas-redact post-renderer. snapshot-review's
    scrub-baseline replays each map against an older-baseline render.

Fixture migration:
  * tests/templates/app-named: drop the manual
    `{{ .Values.atlas.instance.name }}` munge from release.name; ATLAS
    auto-munges now (slice 3). Release becomes "primary-app-named" /
    "secondary-app-named" via prefix.
  * cluster1-deployment9-{primary,secondary}.bats: update INSTANCE to
    the munged form.

Full suite: 369/369 with stage-3 as default.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Retire stage-2 path; thin helmfile.single to fan-out shim

helmfile.single now does only what the stage-3 default needs:
  1. Validate deployment.yaml exists.
  2. Render deployment.yaml as a Go template with the atlas object as
     context, parse the resulting apps[] list.
  3. Emit one helmfiles[] entry per app instance, each pointing at
     helmfile.instance.yaml.gotmpl with the atlas object augmented by
     instance.{template, name}.

Removed (no longer reachable now that engine: stage2 is gone):
  - hierarchy twin-load ($realValues / $redactedValues)
  - hierarchy-level $replMap construction
  - environments.default.values emission
  - per-app-template release rewriting (chart paths, secrets rejection,
    progressive twin-load merge, applyListOverride wiring, redaction
    post-renderer wiring with side-dump)
  - the engine if/else branch and the get "engine" "stage2" default

The deleted block weighed ~400 lines; helmfile.single is 75 lines now.
All of that logic lives in either helmfile.instance (per-release
rewrites, redaction post-renderer, side-dump) or _values_loader (twin-
load, hierarchy walk, progressive merge).

Cleanup:
  - Drop explicit `engine: stage3` from fixtures 30-40 + postrenderer-fail.
  - Update doc-comments that referenced stage-2 (helmfile.instance,
    _values_loader, _functions, helmfile.all, app-stage3-probe).

Full suite: 369/369.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

docs: align README and CLAUDE.md with stage-3-only architecture

README.md
  * "Using the same template multiple times": drop the manual
    `{{ .Values.atlas.instance.name }}` instruction. Document auto-munge
    with apps[].nameStyle: prefix|suffix (prefix default).
  * "Labels": expand the commonLabels table to cover clusterName,
    clusterGroup, template, instance, variant — all auto-injected now.
  * "App Templates": add a "How ATLAS treats helmfile release options"
    table covering the per-instance rewrites (name, chart, values,
    secrets, condition/installed, skipSchemaValidation, postRenderer,
    keyring + set[].file + setString[].file, patches/transformers,
    labels). Lead-in clarifies the "stay close to vanilla helmfile"
    goal.
  * "Value Inheritance Logic": insert template-secrets and instance-
    secrets layers between values and hierarchy. Document the
    "secrets merged last among per-release lists" semantics.
  * "Full Load Order": renumber to include the new secrets layers
    (5-6); standalone-cluster skip range shifts to 10-12.
  * "How It Works": replace the two-step (load + render) description
    with the four-stage pipeline (discover, fan-out, per-instance
    rewrite, release-time loader) that matches the actual code now.

CLAUDE.md
  * Common Commands: switch to the subcommand-first invocation form
    (`helmfile template -f X` / `helmfile build -f X`) so the commands
    match the prefix-based allowlist patterns.
  * Core Template Pipeline: describe all four stages explicitly
    (helmfile.all → helmfile.single → helmfile.instance → loader)
    instead of conflating values resolution into helmfile.single.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Add .claude/utils/run-bats.sh + allowlist

Wraps the test ritual (rm stale tmp dirs, run bats, count results,
list failures) into one allowlistable invocation. Modes:

  .claude/utils/run-bats.sh                       full suite, summary
  .claude/utils/run-bats.sh tests/bats/foo.bats   single file, full output
  .claude/utils/run-bats.sh -v                    full suite, full output

Exit code reflects test result (0 pass, 1 fail), so CI/agents can use
status without parsing output.

Allowlist additions: Bash(.claude/utils/run-bats.sh) and the *-form for
arguments. The script's internal rm of /tmp/atlas-bats-* runs without
prompting because the script itself is the allowlisted unit.

Also rolls in the broader allowlist set added on main (bats *, sops
--decrypt *, helm template *, helmfile template *, helmfile build *,
etc.) so the worktree shares the same prompt-free surface as main.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
@max06 max06 force-pushed the worktree-stage3-values-prototype branch from 9d94296 to 4501353 Compare May 1, 2026 15:19
@max06 max06 merged commit ea3911d into main May 1, 2026
1 check passed
@max06 max06 deleted the worktree-stage3-values-prototype branch May 1, 2026 17:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant