healthCheckExprs is a no-op for resources with a stale status.observedGeneration, which includes failing Flux CRs #1737

.spec.healthCheckExprs can't be used to say "this kind's readiness is not a deployment signal" for any resource that reports a stale status.observedGeneration while it is failing. The built-in generation check in StatusEvaluator.Evaluate returns InProgress before any expression is evaluated, so even current: "true" is a no-op — and it is a no-op precisely for the objects the escape hatch is wanted for.

Flux's own CRs are in that category, which is how I ran into it.

Reproducer

An ImagePolicy that can never resolve:

apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
  name: probe
  namespace: hce-probe
spec:
  imageRepositoryRef:
    name: does-not-exist
  policy:
    semver:
      range: '>=0.0.0'

applied by a Kustomization that tries to exempt it:

spec:
  wait: true
  timeout: 1m
  healthCheckExprs:
    - apiVersion: image.toolkit.fluxcd.io/v1
      kind: ImagePolicy
      current: "true"

Expected: the ImagePolicy is treated as Current and the Kustomization goes Ready.

Actual:

health check failed after 1m0.2s: timeout waiting for:
  [ImagePolicy/hce-probe/probe status: 'InProgress']

The object sits at:

gen=1  obsGen=-1  Ready=False/DependencyNotReady

stable across every reconcile — it is not a transient window.

The control, because my first guess was wrong

I initially assumed the custom status reader simply wasn't matching. It is. Six Kustomizations, same source, timeout: 1m, against a broken ImagePolicy and a working one:

Row 4 is the one that matters: current: "false" does hold a healthy object down, so the reader is engaged and the expressions are being used. Row 2 fails anyway. The expressions are simply never reached.

Why

Evaluate early-returns ahead of the expression loop:

observedGeneration, ok, err := unstructured.NestedInt64(unsObj, "status", "observedGeneration")
if err == nil && ok {
	generation, ok, err := unstructured.NestedInt64(unsObj, "metadata", "generation")
	if err == nil && ok && observedGeneration != generation {
		return &status.Result{Status: status.InProgressStatus}, nil
	}
}

and AddPatchOptions only attaches WithStatusObservedGeneration{} when the object is Ready or Stalled:

if conditions.IsStalled(obj) || conditions.IsReady(obj) {
	opts = append(opts, patch.WithStatusObservedGeneration{})
}

A Flux CR failing on a retriable error is neither, so its observedGeneration never catches up and the pre-check can never be satisfied.

Is the pre-check meant to be unconditional?

There is a good argument for it: don't evaluate expressions against a status the controller has not observed yet. That is right for the transient case. The difficulty is that it is indistinguishable from the permanent case, and for a resource whose failure mode is "never observes the current generation", the escape hatch becomes unreachable exactly when it is needed.

The use case, concretely: wait: true health-gates every applied object, so an ImageUpdateAutomation that cannot push (protected branch, bad credentials) stalls the whole path it sits in, including the application it has nothing to do with. Its Ready condition reports on the supply chain, not on whether the app deployed, and healthCheckExprs looked like the right way to say so per-kind rather than enumerating every workload in healthChecks. I ended up moving those objects into a Kustomization of their own instead, which works, but it is a layout constraint standing in for what looks like a config knob.

If the current behaviour is intended, documenting it on the healthCheckExprs reference would save the next person the experiment — the interaction is not obvious from either the field docs or the CEL cheatsheet, and current: "true" failing to make something current is a surprising place to land.

Versions

Measured on Flux 2.7.5 / kustomize-controller v1.7.3. Both code paths above are unchanged on main today (status_evaluator.go last touched in 3619c73, AddPatchOptions in runtime/reconcile/result.go unchanged), so I would expect this to reproduce on 2.9.x — but I have not run it there, so treat that as inference rather than a measurement.

Unrelated to the above and purely a version note, in case someone else lands here from the same search: on v1.7.3 an entry with kind: "" is accepted by the CRD and then silently never matches, because pkg/runtime v0.91.0 predates 6b6497f ("runtime/cel: allow empty kind in status reader"). Current releases vendor v0.110.3 and handle it correctly. Worth knowing only because reading status_reader.go on main suggests the group-wide form works on older controllers, and the failure is quiet — the entry does nothing and the Kustomization reports healthy for the wrong reason.