fix: run the nok8s smoketest without a cluster - #1708

Description

Fixes #1698.

llmdbenchmark ... smoketest on a no-Kubernetes scenario died before step 00 with
"Failed to load Kubernetes configuration", contradicting docs/nok8s.md, which says
no cluster access is required. Two layers were broken.

_do_smoketest was the only phase builder that never computed container_only, so
resolve_cluster() took the Kubernetes branch and the phase exited 1 before any step
ran. It also enforced a namespace unconditionally, which standup, teardown and run all
relax for nok8s. Fixing the phase builder rather than the subcommand dispatcher also
repairs experiment, which calls _do_smoketest with no nok8s guard.

Even past that, the validators were cluster-shaped by construction: require_cmd()
plus require_namespace(), Service IP discovery, pod readiness waits, pod IPs and an
oc get route lookup, none of which exist in a nok8s deployment.
BaseSmoketest.run_health_checks and run_inference_test now branch on
container_only to two HTTP probes that read the same rendered
34_nok8s-containers.yaml launch spec step 06 deployed from, so the endpoint probed
and the model asserted match what was actually launched. No validator subclass
overrides either method, so the single base branch covers every stack. Step 01's
kubectl-based ephemeral pod cleanup is skipped for the same reason.

The probes fail loudly on an unreachable endpoint, a non-200, a non-JSON body, a
malformed body shape, a model the server does not list, and an empty completion, and
issue no request at all under --dry-run.

Three deliberate calls:

• Separate timeouts. GET /v1/models gets 15s, POST /v1/completions gets 120s, matching
  BaseSmoketest._curl_post's existing default. A 16-token generation on a cold or
  accelerator: cpu stack is not reliably sub-15s, and a single 15s budget would have
  turned that into a false failure.
• A retry budget. Both probes go through a requests.Session with a urllib3 Retry
  adapter (total=3, backoff_factor=2, 502/503/504, raise_on_status=False).
  base.py already records that the old no-retry cluster behaviour "made every transient
  warmup race a hard smoketest failure", and the first draft of these probes had
  reintroduced exactly that. Reusing base.py's _is_retryable/_compute_backoff was
  not possible, since base.py imports nok8s.py at module level and importing back would
  be circular. raise_on_status=False preserves the existing
  "returned HTTP 503: " diagnostic instead of surfacing a RetryError.
• The standup phase still does not chain the smoketest for nok8s. Changing which
  phases auto-chain is a separate behavioural decision for maintainers, and I cannot
  validate it on a host with no GPU. The skip is now documented in docs/nok8s.md and
  explained honestly in the code comment rather than by a stale rationale.

Not done here: a /v1/chat/completions fallback. The nok8s stack launches
vllm serve <model> from the same rendered spec the probe reads, and vLLM's OpenAI
server always registers /v1/completions. The cluster path's fallback exists because
it probes arbitrary user-supplied routes; nok8s cannot produce that configuration.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• This change requires a documentation update (included)

How Has This Been Tested?

Adds tests/test_nok8s_smoketest.py (23 tests) driving the real code against a stdlib
stub server on an ephemeral loopback port.

$ python3 -m pytest tests/ -x -q -n2
762 passed, 31 skipped in 27.56s

Baseline on main is 739; the branch adds 23.

Revert check, with the three production files reset to main and the tests kept:

FAILED tests/test_nok8s_smoketest.py::TestBaseRoutesContainerOnly::test_health_checks_do_not_touch_the_cluster
FAILED tests/test_nok8s_smoketest.py::TestBaseRoutesContainerOnly::test_inference_does_not_touch_the_cluster
FAILED tests/test_nok8s_smoketest.py::test_do_smoketest_sets_container_only
3 failed, 20 passed in 1.82s

Two mutation checks on the new logic: emptying the Retry status_forcelist fails
test_transient_503_is_retried, and deleting the container_only short-circuit in
ExecutionContext.resolve_cluster fails test_do_smoketest_sets_container_only.

Exercised through the real CLI with KUBECONFIG=/nonexistent, no --dry-run and no
mocking: healthy gives exit 0; wrong model, null body, array body, empty choices,
string choices, null completion body, persistent 503 and no server at all each give
exit 1 with the correct diagnostic and no traceback; 503-once-then-200 gives exit 0,
which is the retry budget working. --dry-run issues zero I/O.

pre-commit run --files on the 5 changed files: py-compile, pytest, render-changed-
scenarios, detect-secrets, ruff-check and ruff-format all Passed. The SBOM hook
self-skipped on its file filter.

Test Configuration

• Kubernetes version: none, which is the point of this change. A live nok8s standup
  was not verifiable here, since vLLM needs --gpus all and this host has no GPU. The
  probes are exercised against a stub server speaking the same OpenAI-shaped contract,
  not against a real vLLM.

Checklist

• My changes follows the style guidelines of this project
• I have performed a self-review of my own changes
• I confirm that a full ./setup/standup.sh -> run.sh -> ./setup/teardown.sh sequence completed successfully
• I confirm that pre-commit run was run and all checks passed
• I have updated the documentation accordingly (docs/nok8s.md,
  llmdbenchmark/smoketests/README.md)

No cluster and no GPU here, so the standup/run/teardown sequence was not run.