fix: guidellm harness reports the real cause for an unsupported profile - #1683

Description

Fixes #1510, specifically the misleading-error half of it (see the note at the end about the image pin).

profile: replay, and any other profile the installed guidellm doesn't know, fails inside guidellm's own argument parsing. The real error gets swallowed and resurfaces as something unrelated. Operators see:

Error: Missing field '--target': Field required

when the actual cause is:

Invalid value for --profile: Input should be 'synchronous', 'concurrent', 'throughput', 'constant' or 'poisson'

so they go and debug --target, which is fine.

This adds a pre-flight check to guidellm-llm-d-benchmark.sh that reads the workload's profile field and asks the installed guidellm which profiles it actually supports (a small python3 -c introspection of ProfileType | StrategyType) before guidellm is invoked. An unsupported profile now fails fast naming the real cause and the installed guidellm version. If the introspection fails for any reason it fails open, so the guard never blocks a run it couldn't validate, and a workload with no profile field is unaffected. The error goes to stderr, matching the precedent in lm-eval-llm-d-benchmark.sh.

On why the guard doesn't just pass --profile and let guidellm validate it: guidellm's --profile does validate against click.Choice(STRATEGY_PROFILE_CHOICES), but this harness hands the whole workload file over via --scenario and never passes --profile, so that built-in validation never runs. Reproducing the check in shell keeps the change inside one harness script and avoids depending on --scenario/--profile precedence that isn't documented.

Out of scope: bumping the guidellm pin so profile: replay actually works. replay currently only exists on guidellm's main branch (e.g. 73d91f311bd9). build/Dockerfile does install from a git commit rather than a PyPI release, so pinning a newer commit is technically possible today, but whether to ship an unreleased guidellm is a maintainer call rather than something this fix should decide.

Type of change

• Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Added tests/test_guidellm_profile_guard.py, which extracts the guard block from the shipped script itself (rather than a copy, so the test can't drift from the script) and runs it against fake yq, python3, and guidellm stubs. Four cases: unsupported profile fails with the real cause, supported profile passes through, a missing profile field doesn't block, and introspection failure fails open.

$ python3 -m pytest tests/test_guidellm_profile_guard.py -q
4 passed

Confirmed the tests actually fail without the change by reverting workload/harnesses/guidellm-llm-d-benchmark.sh to main and re-running: all four fail. Also ran bash -n on the script (syntax OK) and ruff check on the new test (clean).

Test Configuration

• Kubernetes version: none. This change is confined to a harness shell script and a unit test that stubs out yq/python3/guidellm, so no cluster is involved in exercising it. I don't have a cluster available, so the full standup/run/teardown sequence was not run. Flagging that rather than claiming it.

Checklist

• My changes follows the style guidelines of this project
• I have performed a self-review of my own changes
• I confirm that a full ./setup/standup.sh -> run.sh -> ./setup/teardown.sh sequence completed successfully
• I confirm that pre-commit run was run and all checks passed
• I have updated the documentation accordingly

On the two unchecked boxes: no cluster available here for the standup/run/teardown sequence, and pre-commit isn't installed in this environment, so I ran ruff check and pytest directly instead. Happy to have CI confirm both. No documentation change seemed warranted for an error-message fix, but say the word if you'd like the profile constraint written down somewhere.