Skip to content

docs(discovery): make the file discovery example run as written - #3681

Merged
ahg-g merged 1 commit into
llm-d:mainfrom
fjvbn2003:docs/file-discovery-example
Oct 30, 2026
Merged

ahg-g merged 1 commit into
llm-d:mainfrom
fjvbn2003:docs/file-discovery-example

Conversation

@fjvbn2003

Copy link
Copy Markdown

What type of PR is this?

/kind documentation

What this PR does / why we need it:

The walkthrough in docs/discovery.md, "Running EPP with file discovery (no Kubernetes)", does not work as written with the v1.0.0 epp. Following it with the v1.0.0 epp, Envoy 1.33.2 and three llm-d-inference-sim v1.0.0 backends on 127.0.0.1:8001-8003 (all in one Linux container), it fails in three places:

  1. The config has no apiVersion/kind header, so the EPP exits:
    failed to parse config - failed to decode configuration JSON/YAML: Object 'Kind' is missing in 'plugins: ...
    
  2. With the header added, the scheduler config build fails because the profile lists the profile handler:
    scheduler config build failed: failed to add plugin 'single-profile' to profile 'default': plugin 'single-profile/single-profile-handler' must implement Filter, Scorer, or Picker
    
    single-profile-handler also takes no parameters (its factory ignores them), so the scorer/picker parameters in the example have no effect.
  3. With a valid config and the documented command, every request through Envoy returns 500. The EPP serves ext_proc over TLS by default (server TLS log line with tls: true), and the epp cluster in the Envoy config connects in plaintext. For 12 requests, Envoy reported ext_proc.streams_failed: 12 and downstream_rq_5xx: 12.

Changes, all in that section:

  • Add apiVersion: llm-d.ai/v1 and kind: EndpointPickerConfig to the config.
  • Reference the scorer and the picker from the scheduling profile and drop the single-profile-handler entry. With one profile the EPP adds that handler itself; it shows up in the Effective raw configuration log line.
  • Add --secure-serving=false to the epp command, with one paragraph on the TLS default and a link to tls.md.
  • The text said the EPP logs a line for each endpoint loaded from the file. At the default verbosity it does not; it logs watching endpoints file for changes when watchFile is true. The sentence now names that line.

Tests:

  • The config and the two commands copied from the updated section (endpoint addresses changed to 127.0.0.1:8001-8003) start the EPP and Envoy. 30 POST /v1/completions requests through :8080 all return 200, spread over the three backends (10/11/9 by the simulator's x-inference-pod header). Envoy reports streams_failed: 0.
  • The config from the current section fails with error 1 above, and with the header added it fails with error 2.

The runs were made on 2026-10-26 (UTC) with the v1.0.0 release epp (commit b9393d1). docs/discovery.md and the config loader, scheduler profile and --secure-serving code are unchanged on main at 6e5f3f0 (2026-10-27), and the patch applies to it as is.

make presubmit was not run because this change only touches Markdown. The TLS path (Envoy with an UpstreamTlsContext to the EPP) was not tested.

Which issue(s) this PR fixes:

None. This is a documentation correction. It is unrelated to #3615 / #3630 (file-discovery reload after atomic replace on Docker Desktop).

Release note (write NONE if no user-facing change):

NONE

AI disclosure: this change was prepared with the help of an AI coding assistant (Claude Code). Claude ran the reproduction and the checks above in a container in my home lab and drafted the diff and this description.

🤖 Generated with Claude Code

The example in "Running EPP with file discovery (no Kubernetes)" did
not start with the v1.0.0 epp: the config had no apiVersion/kind
header, and it listed single-profile-handler inside the scheduling
profile, which only accepts filters, scorers and pickers. With a valid
config, Envoy's plaintext epp cluster failed every request because the
EPP serves ext_proc over TLS by default.

Add the header, reference the scorer and picker from the profile, and
start the EPP with --secure-serving=false. Also describe the startup
log lines the EPP emits in this mode.

Signed-off-by: YoungjuKim <[email protected]>
@fjvbn2003
fjvbn2003 requested a review from a team as a code owner October 27, 2026 09:47
@github-actions github-actions Bot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. kind/documentation Categorizes issue or PR as related to documentation. area/docs labels Oct 27, 2026
@elevran elevran self-assigned this Oct 29, 2026
@ahg-g
ahg-g added this pull request to the merge queue Oct 30, 2026
Merged via the queue into llm-d:main with commit cc346a6 Oct 30, 2026
46 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/docs kind/documentation Categorizes issue or PR as related to documentation. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants