Summary

Run the existing operator-agent chainsaw test suite under k8s-tests/operator-agent/ against the new agent-go container image from #220, in CI, alongside the existing Python runs. This is the parity safety net for the cutover: every behavior the operator e2e tests pin down — log format, flag file shape, history file format, interrupt idempotence, log retention, the dont_write_logs mode — must work identically against the Go agent before #222 deletes the Python source.

Depends on #220 (the agent-go image must exist and be pullable).

Motivation

Unit tests in #213–#219 cover each module in isolation, but they can't catch:

• A log line format change that breaks an operator-side log scraper.
• A flag file path drift that causes the operator to think no step has run.
• A history file shape change that confuses an upgrade.
• A subtle env var leak between the chrooted step and the host.
• An exit code that's -15 in Python but 255 in Go and breaks the NodeRestart path.

The five existing chainsaw scenarios (simple, interrupt, reap_old_logs, dont_write_logs, check_node) collectively exercise these surfaces against a real kind cluster with a real operator. If the Go image passes them, we have high confidence the cutover is safe.

Feature description

A new CI job in .github/workflows/agent-go-ci.yaml that:

1. Spins up a kind cluster with the operator (matching the existing operator-agent-tests job in .github/workflows/agent-ci.yaml lines 266–335).
2. Sets AGENT_IMAGE to the agent-go image tag from [FEA]: agent-go Dockerfile and container CI #220.
3. Runs make setup-kind-cluster operator-agent-tests from operator/.
4. Fails the PR if any chainsaw scenario fails.

Proposed direction

1. CI job

Mirror the existing operator-agent-tests job almost line-for-line. The only differences:

• needs: now includes the create-manifest job for the Go image (from [FEA]: agent-go Dockerfile and container CI #220), not the Python one.
• AGENT_IMAGE env points at the Go image tag.

Keep the existing Python operator-agent-tests job in place — both run, both must pass. Until cutover, the operator is built and tested against both images on every relevant PR.

2. Expect divergences; fix forward in this PR

Run the suite locally first against the Go image and expect failures. Each failure is a parity bug — fix it in this PR (not by amending #213–#219), so the diff that closes the gap is visible to the reviewer alongside the test that caught it.

Common categories to anticipate:

• Log line format: exact [out] / [err] prefix, timestamp precision, line-by-line newline handling. [FEA]: tee streaming and run_step #217 territory.
• Flag path bytes: MakeFlagPath marker reproduction. [FEA]: Filesystem helpers (paths, flags, log paths) + history file #215 territory.
• History JSON shape: key order, time format, list orientation. [FEA]: Filesystem helpers (paths, flags, log paths) + history file #215 territory.
• SUCEEDED: typo: presence required (e2e greps for it). [FEA]: tee streaming and run_step #217 territory.
• NodeRestart -15 exit code: [FEA]: tee streaming and run_step #217 + [FEA]: do_interrupt with NodeRestart -15 special case #218 territory.
• dont_write_logs scenario: SKYHOOK_AGENT_WRITE_LOGS=false must produce no log files but still stream to stdout. [FEA]: tee streaming and run_step #217 territory.
• reap_old_logs scenario: log retention keeps current + 4 previous. [FEA]: Filesystem helpers (paths, flags, log paths) + history file #215 territory.
• interrupt scenario: idempotence under repeated reconciles. [FEA]: do_interrupt with NodeRestart -15 special case #218 territory.
• check_node scenario: check_results file format and _ALL_CHECKED flag. [FEA]: Controller main / agent_main / SIGTERM + CLI parsing and entrypoint banner #219 territory.

If a fix is large enough that it changes the shape of one of the earlier PRs' modules, prefer landing it as a small follow-up (not amending the merged PR). Note this in the PR description.

3. Log capture for diagnostics

Configure the chainsaw run to dump:

• All agent pod logs (both Python and Go) on failure.
• The contents of /etc/skyhook/flags/, /etc/skyhook/history/, /var/log/skyhook/ from a target node.

This is the difference between "this scenario failed" being actionable vs. unhelpful. Add a dump-on-fail step to the job.

4. Don't gate the cutover prematurely

This PR does not flip the operator default to the Go image. Operator-side make targets and the Python image continue to be the production path until #222 explicitly cuts over. This PR only proves we could cut over without regression.

5. Tests

The PR is itself a test job. Specific assertions:

• All five chainsaw scenarios under k8s-tests/operator-agent/ pass against agent-go.
• All five also still pass against the Python agent image (regression guard — make sure splitting the workflow didn't break anything).

Scope boundaries

In scope:

• New CI job running existing chainsaw suite against the Go image.
• Parity bug-fixes that the suite uncovers (small, surgical).
• Log dump on failure.

Out of scope:

• Editing the chainsaw scenarios themselves — they are the contract. If a scenario is wrong (asserts on something that shouldn't be a contract), open a separate issue.
• Cutting over to the Go image as the default ([FEA]: Cutover: flip default to Go, delete Python, flatten dir, update docs #222).

Acceptance criteria

• New operator-agent-go-tests job exists and passes.
• All five chainsaw scenarios pass against the Go image.
• All five chainsaw scenarios still pass against the Python image (no regression).
• Failure dumps are captured for both jobs.
• No changes to operator code or Helm chart.
• No changes to chainsaw scenarios themselves (or, if any, justified in the PR with a separate small issue link).

Open questions

• Is the kind cluster shared across the two operator-agent-tests jobs (faster) or per-job (cleaner isolation)? Per-job is safer; faster is fine if both pass reliably. Start per-job.
• Should this PR also run the suite against an upgrade scenario — Python agent ran first, Go agent takes over mid-rollout? That's the most realistic cutover risk. Recommend yes — add a sixth scenario agent-upgrade that uses the Python image for apply and the Go image for apply-check/subsequent stages, and asserts no flag/history/log drift.
• If the operator-agent suite reveals a parity bug that isn't fixable without rework, what's the escalation? Recommend opening a small follow-up issue per bug rather than blocking the whole rewrite on a single PR.

References (codebase)

• k8s-tests/operator-agent/ — the five chainsaw scenarios.
• .github/workflows/agent-ci.yaml lines 266–335 — the existing job to mirror.
• operator/Makefile — setup-kind-cluster, operator-agent-tests targets.

Alternatives considered

• Skip this PR and rely on [FEA]: Cutover: flip default to Go, delete Python, flatten dir, update docs #222 to catch parity issues during cutover. Rejected — discovering a regression after deleting the Python source means a confusing revert. Prove parity first, delete second.
• Write a brand-new Go-only e2e suite. Rejected — the existing chainsaw suite is the operator-side contract; rewriting it loses the value.

Code of Conduct

• I agree to follow Skyhook's Code of Conduct.