fix: carry the skipped PVC bind budget into the data-access pod wait - #1696

Description

Fixes #1694.

CommandExecutor.wait_for_pvc correctly refuses to block on a StorageClass with
volumeBindingMode: WaitForFirstConsumer, because binding needs a consumer pod
that has not been applied yet. But it returned a plain success result,
indistinguishable from "the PVC actually bound", so the caller had no way to know
its 240s pvc_bind_timeout budget went unspent.

In step 05 that meant the 120s harness_data_access_timeout wait had to cover
two serialised phases, dynamic volume provisioning (which only starts once the
pod schedules) plus container start, on half the budget of the easier job. And
--pvc-bind-timeout, whose help text is where the project already wrote down
that slow shared-storage backends (weka, ceph, gpfs) need minutes, never applied
on those backends, because they are all WaitForFirstConsumer in the common
case. First standup on a cold volume failed deterministically and the immediate
retry passed because the volume was by then provisioned, so it presented as
flakiness.

The change reports the skipped wait on CommandResult (wait_skipped) and adds
the unspent bind budget to the data-access pod wait when it is set. On failure
the error now names the skipped bind and the two knobs to raise. The run phase
inherits step 05, so both pipelines pick this up.

It also registers --data-access-timeout on the standup subparser and passes it
into the standup ExecutionContext. That knob was only wired for run and
experiment, so standup, which is the phase this failure is reported from and
the phase the issue's LLMDBENCH_DATA_ACCESS_TIMEOUT=900 workaround targets,
rejected the flag outright and silently ignored the env var while cli.py's
override table still advertised it as active. Without this, the new failure hint
would name a knob that phase does not have.

Two deliberate calls, both also recorded in the commit message:

• wait_skipped means the bind wait was skipped, not that the volume was
  actually unprovisioned. On an idempotent re-standup the PVC may already be
  Bound. The widened value is only a ceiling, so it costs nothing on that warm
  path, and the log and error text say "skipped" rather than claiming a deferral
  that may not have happened. Probing .status.phase to tell the two apart would
  cost an extra kubectl call on every standup and every run, and would add a new
  question (what does the flag mean when the probe itself errors or returns
  empty) to make a diagnostic string marginally more specific.
• The other two wait_for_pvc callers, in step_04_model_namespace.py, need no
  adjustment: their downstream waits are the model download job (2400s) and the
  serving deploy (1500s/900s), both already larger than pvc_bind_timeout. The
  docstring says so, so it does not read as a rule that is violated in-tree.

Not done here: the issue's suggestion (2), distinguishing "pod Pending because
its PVC is still provisioning" from "pod failed to start". That needs the pod
wait to inspect the PVC's own phase and is a separate change to
wait_for_pods. Suggestion (3) is covered by the new hint text.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• This change requires a documentation update (included)

How Has This Been Tested?

Adds tests/test_harness_pvc_bind_deferral.py (5 tests), which drives the real
HarnessNamespaceStep against a stub executor: the deferred-bind path extends
the wait to 120 + 240, the non-deferred path does not, the failure message
names the cause and the knobs, and the real standup subparser accepts
--data-access-timeout 900.

$ python3 -m pytest tests/test_harness_pvc_bind_deferral.py -q
5 passed in 3.60s

$ python3 -m pytest tests/ -q --ignore=tests/test_direct_service_mode.py
733 passed, 31 skipped in 33.65s

$ uvx [email protected] check --output-format=full .
All checks passed!
$ uvx [email protected] format --check .
255 files already formatted

The --ignore is a pre-existing gap in my environment, not something this
change causes: tests/test_direct_service_mode.py needs the external
llm-d-planner dependency, and it fails the same way on main here.

Confirmed the new tests fail without the change, reverting one file at a time in
a throwaway worktree:

• Revert step_05_harness_namespace.py only: 2 failed, 3 passed.
  test_deferred_bind_extends_data_access_wait fails with
  assert 120 == (120 + 240), which is issue WaitForFirstConsumer PVCs hand the provisioning wait to a 120s timer, so the first standup on slow RWX storage always fails #1694 reproduced numerically, and
  test_deferred_bind_failure_names_the_cause fails with
  assert 'WaitForFirstConsumer' in 'Data access pod not ready: Timed out after 120s waiting for pods',
  which is main's exact unhelpful message.
• Additionally revert interface/standup.py and cli.py: 3 failed, 2 passed,
  the new flag test failing with
  error: unrecognized arguments: --data-access-timeout 900.
• Additionally revert executor/command.py: 5 failed, all
  TypeError: CommandResult.__init__() got an unexpected keyword argument 'wait_skipped'.

Test Configuration

• Kubernetes version: none. I do not have a cluster available, so the full
  standup/run/teardown sequence was not run, and the WaitForFirstConsumer
  behaviour itself is exercised through a stub executor rather than against real
  storage. The arithmetic and the message are covered; a real cold-volume
  standup on a slow RWX backend is not something I can verify here, and I would
  rather flag that than imply otherwise.

Checklist

• My changes follows the style guidelines of this project
• I have performed a self-review of my own changes
• I confirm that a full ./setup/standup.sh -> run.sh -> ./setup/teardown.sh sequence completed successfully
• I confirm that pre-commit run was run and all checks passed
• I have updated the documentation accordingly (llmdbenchmark/executor/README.md,
  llmdbenchmark/interface/README.md)

On the two unchecked boxes: no cluster here, and pre-commit's
render-validation hook needs a populated .venv plus helm, which this
environment does not have. I ran ruff check, ruff format --check and the
full pytest suite directly instead. This diff touches no rendering code.