Summary

Wire the per-mode helpers from #215–#218 into the top-level controller, and add the CLI arg parser, env handling, and dashed startup banner so the binary is functionally a drop-in replacement for the Python entrypoint. Combines what would otherwise be two PRs since the CLI is just ~150 LOC of mechanical wiring on top of the controller and adds a serial round-trip if split.

After this PR the agent works end-to-end against a real config — only the Docker image (#220) and the operator-agent parity tests (#221) remain before the cutover (#222).

Depends on #213–#218.

Motivation

This is the orchestration layer plus its public entry point. The pieces all exist after #213–#218; this PR is the glue that:

• Decides which helper to call for a given Mode.
• Manages SIGTERM-driven graceful shutdown so in-flight steps can stop cleanly when the agent pod is terminated.
• Parses os.Args in three legacy + one new positional form (different operator versions in production may use different forms — backward compatibility per the AGENTS.md "CLI must remain backward-compatible" rule).
• Prints the dashed banner that operators and e2e tests grep for.

Splitting controller and CLI would mean #219 lands code that has no entry point and #220 lands an entry point with no functionality to gate. Better to land them together as "the binary now works".

Feature description

A new internal/controller package with the orchestration logic, plus a complete cmd/agent/main.go that parses args, sets env, prints the banner, and dispatches to controller.Run.

Proposed direction

1. controller.Run

Run(ctx, mode step.Mode, rootMount, copyDir, interruptData string, alwaysRunStep bool) (failed bool, err error):

1. Validate mode against the known set; if unknown, log warn and return failed=false (matches Python).
2. If the data dir hasn't been copied to {rootMount}{copyDir}, recursively copy SKYHOOK_DATA_DIR (default /skyhook-package). Also copy /etc/nvidia-bootstrap/node-files/ if present (legacy mode).
3. If mode == step.Interrupt, delegate to interrupts.Run(...) ([FEA]: do_interrupt with NodeRestart -15 special case #218) and return.
4. Otherwise read {rootMount}/{copyDir}/config.json and call config.Load(...) ([FEA]: Config loader with embedded JSON schemas + Steps.validate #214).
5. If mode is not Uninstall/UninstallCheck:
   • If {rootMount}/{copyDir}/root_dir exists, recursively copy onto rootMount (lets packages drop files into the host filesystem).
   • Validate every cfg.ExpectedConfigFiles entry is present under {rootMount}/{copyDir}/configmaps/. Missing → SkyhookValidationError.
6. Call agentRun(ctx, mode, rootMount, copyDir, cfg, interruptData, alwaysRunStep).
7. If agentRun returns an error AND ctx.Err() != nil (SIGTERM), log graceful-shutdown and return failed=true instead of the error.

2. agentRun per-step loop

Lines ~590–636 of agent/skyhook-agent/src/skyhook_agent/controller.py:

1. Write {flagDir}/START flag ([FEA]: Filesystem helpers (paths, flags, log paths) + history file #215's SetFlag).
2. If cfg.Modes[mode] is empty, log There are no {mode} steps defined. This will be ran as a no-op. and skip the loop.
3. For each step:
   • Bail out cleanly if ctx.Err() != nil.
   • Compute flag path via flags.MakeFlagPath ([FEA]: Filesystem helpers (paths, flags, log paths) + history file #215).
   • For Mode.Upgrade / Mode.UpgradeCheck, call history.Read(...) ([FEA]: Filesystem helpers (paths, flags, log paths) + history file #215) so CURRENT_VERSION/PREVIOUS_VERSION env (and UpgradeStep args) are populated.
   • For non-check modes:
     • If flags.CheckFlagFile(...) returns true, continue (already done).
     • Print {mode} {step.Path} {step.Arguments} {step.Returncodes} {step.Idempotence} {step.OnHost} exactly (operator e2e log grep target).
     • Call runner.RunStep(...) ([FEA]: tee streaming and run_step #217). If failed, return failed=true.
     • Write the flag with last_run: {ISO time}\nstep_always_runs: {bool} body.
   • For check modes:
     • Print the same step header.
     • Append runner.RunStep(...) result to a results []bool slice. (Don't return early on check failure — let SummarizeCheckResults decide.)
4. After the loop, if mode is a check mode and there are steps, call SummarizeCheckResults(...). Returns true → failed=true.
5. If mode is ApplyCheck, UpgradeCheck, or UninstallCheck and we haven't returned failed=true, call history.Write(...) ([FEA]: Filesystem helpers (paths, flags, log paths) + history file #215).
6. If mode == UninstallCheck, also call RemoveFlags(...).

3. SummarizeCheckResults

Port lines 440–463 verbatim:

• If len(results) != len(steps) → print "It does not look like you have successfully run all check steps yet." and return true.
• Otherwise write {flagDir}/check_results containing one line per step in the form {step.Path} {True|False} (Python literal, capitalized).
• If any result is true (failure), return true.
• Otherwise write {flagDir}/{mode}_ALL_CHECKED empty flag and return false.

4. RemoveFlags

Port lines 537–541. Walk every step in every mode, build the flag path, os.Remove if it exists. Ignore not-exist errors.

5. SIGTERM handling

Replace Python's global received_sigterm flag with a context:

ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM)
defer stop()

Plumb ctx through every step iteration, every runner.RunStep call, and every interrupts.Run call. The runner from #217 already accepts ctx. When ctx.Err() != nil mid-loop, return failed=true with no error and let Run log the graceful shutdown.

Match Python's log lines: Received SIGTERM signal - initiating graceful shutdown and Gracefully shutting down due to SIGTERM.

6. CLI arg parsing

Match Python's cli() (lines 653–718) verbatim. Four supported forms:

• Legacy 2-arg: agent {mode} {copy_dir} with root_mount = "/root".
• Legacy 3-arg interrupt: agent interrupt {copy_dir} {interrupt_data} with root_mount = "/root".
• New 3-arg: agent {mode} {root_mount} {copy_dir}.
• New 4-arg: agent {mode} {root_mount} {copy_dir} {interrupt_data}.

args := os.Args[1:]
var mode, rootMount, copyDir, interruptData string
switch len(args) {
case 2:
    mode, copyDir = args[0], args[1]
    rootMount = "/root"
case 3:
    if args[0] == string(step.Interrupt) {
        mode, copyDir, interruptData = args[0], args[1], args[2]
        rootMount = "/root"
    } else {
        mode, rootMount, copyDir = args[0], args[1], args[2]
    }
case 4:
    mode, rootMount, copyDir, interruptData = args[0], args[1], args[2], args[3]
default:
    // print usage to stderr, exit 2
}

Add // why: legacy operator versions invoke the 2- and 3-arg forms; we must accept all four for backward compatibility (see AGENTS.md CLI compat rule).

7. Env handling

• COPY_RESOLV (default "true"): if truthy, copy /etc/resolv.conf into {root_mount}/etc/resolv.conf. Match Python's shutil.copyfile semantics.
• OVERLAY_ALWAYS_RUN_STEP (default "false"): pass to controller.Run's alwaysRunStep arg.
• Read SKYHOOK_RESOURCE_ID, SKYHOOK_DATA_DIR, SKYHOOK_ROOT_DIR, SKYHOOK_LOG_DIR, SKYHOOK_AGENT_BUFFER_LIMIT, SKYHOOK_AGENT_WRITE_LOGS for the banner.

8. Startup banner

Print exactly the layout from Python print lines 687–711, with the same field order, same dashes, same colon spacing, same str.center(s, 20, "-") quirk on section headers (which produces e.g. --CLI CONFIGURATION- because the string is 17 chars and center adds the extra dash on the right). The directory section uses MakeConfigDataFromResourceID() (the helper #218 added).

The banner is a public contract — comment it as such.

9. Subcommand routing

Detect os.Args[1] == "chroot-exec" and route to the #216 subcommand handler before the positional-arg parser.

10. Exit code

controller.Run returns (failed bool, err error). CLI rules:

• err != nil → print error to stderr, exit 1.
• failed → exit 1.
• Else → exit 0.

11. Tests

Port the orchestration-level cases from agent/skyhook-agent/tests/test_controller.py:

• Unknown mode → warn, return failed=false.
• Interrupt mode → delegated to interrupts.Run.
• Apply mode end-to-end → START flag written, step's flag written, success returned.
• Apply-check end-to-end with mixed results → check_results file written, _ALL_CHECKED flag conditional.
• Uninstall-check → flags removed afterward.
• Upgrade mode → history read populates env.
• Apply-check success → history written.
• Missing expected_config_files → SkyhookValidationError.
• SIGTERM during step loop → returns failed=true, doesn't run subsequent steps.

CLI tests:

• Each of the four arg forms parses to the expected (mode, rootMount, copyDir, interruptData) tuple.
• COPY_RESOLV=false skips the resolv.conf copy.
• OVERLAY_ALWAYS_RUN_STEP=true is forwarded.
• Banner output matches a golden text fixture (compare line-by-line).
• Exit code 1 on failed=true.
• Exit code 0 on failed=false.
• chroot-exec subcommand still routes correctly.

For step execution use a mock runner.RunStep injected through an interface seam. Use a runMain(args []string, env map[string]string, stdout, stderr io.Writer) int seam to make the CLI testable without spawning a subprocess.

Scope boundaries

In scope:

• Top-level Run, agentRun, mode dispatch, SIGTERM, summarize/remove helpers.
• All four CLI arg forms, env handling, resolv.conf copy, banner, exit code.
• Subcommand routing for chroot-exec.

Out of scope:

• Docker image ([FEA]: agent-go Dockerfile and container CI #220).
• Operator-side changes — the operator already invokes the agent in one of these forms; no contract change.

Acceptance criteria

• All ported orchestration and CLI tests pass.
• SIGTERM stops mid-loop within one step iteration (test-asserted).
• START flag is always written before any step runs.
• check_results file format matches Python (path True|False with capitalized booleans).
• _ALL_CHECKED flag is conditional on no failures.
• History is written exactly when Python writes it (only on the three *-check modes, only on success).
• Uninstall-check removes flags exactly once.
• All four arg forms route correctly with table-driven tests.
• Banner golden test matches Python output line-for-line.
• COPY_RESOLV and OVERLAY_ALWAYS_RUN_STEP defaults match Python (true and false).
• chroot-exec subcommand still works.
• Exit code is 1 on either err != nil or failed=true.

Open questions

• Drain on SIGTERM: the Python implementation just sets a flag and lets the in-flight step finish naturally (subprocess gets the signal too via process group). We should do the same — passing ctx to exec.CommandContext already gives this behavior. Confirm in a test that the in-flight subprocess gets SIGTERM and we wait for it.
• Should we add a --help flag? Python doesn't. Recommend yes (hidden / non-breaking) — useful for operators debugging on a node, no contract impact.
• Reproduce the dashed banner verbatim or normalize? Recommend verbatim — operator e2e logs grep for the dashed strings, and "fixing" them is a behavior change.
• Should Run take a *log/slog.Logger or use the package-level logger? Use whatever [FEA]: Bootstrap agent/go/ module + port pure-data types #213 standardized on.

References (codebase)

• agent/skyhook-agent/src/skyhook_agent/controller.py lines 44–53, 440–463, 537–718.
• agent/skyhook-agent/tests/test_controller.py.
• agent/README.md (env var docs).
• .claude/CLAUDE.md "CLI must remain backward-compatible" rule.

Alternatives considered

• Use controller-runtime patterns (Reconcile, RequeueAfter). Rejected — the agent is not a controller, it's a one-shot CLI invoked per pod. Borrowing the vocabulary would mislead readers.
• Use a flag-based CLI (--mode, --root-mount, --copy-dir). Rejected — would break the operator's existing invocation. Could be added later as an alternative form.
• Keep controller and CLI as separate PRs. Rejected — CLI is small wiring on top, splitting just adds a serial round-trip.

Code of Conduct

• I agree to follow Skyhook's Code of Conduct.