Repository navigation
demos/parking: size each worker for one actor and fail load.sh on req… - #3391
Merged
Dmitry Berkovich (dberkov) merged 1 commit intoNov 3, 2026
Merged
Conversation
Bo-Cheng Chu (bochengchu)
force-pushed
the
parking-demo-fix
branch
from
October 28, 2026 17:55
61a0141 to
cfa0421
Compare
Bo-Cheng Chu (bochengchu)
marked this pull request as ready for review
October 28, 2026 20:15
Bo-Cheng Chu (bochengchu)
requested review from
Benjamin Elder (BenTheElder) and
Lior Lieberman (LiorLieberman)
October 28, 2026 20:15
Bo-Cheng Chu (bochengchu)
force-pushed
the
parking-demo-fix
branch
2 times, most recently
from
October 28, 2026 23:20
696f9cd to
c91c92c
Compare
Bo-Cheng Chu (bochengchu)
force-pushed
the
parking-demo-fix
branch
from
October 29, 2026 17:21
c91c92c to
db9e56f
Compare
Bowei Du (bowei)
previously approved these changes
Oct 29, 2026
| printf " other : %s\n" "${other}" | ||
| printf " 200 latency : avg %ss, slowest %ss <- parked requests sit here\n" "${avg_ok}" "${slowest_ok:-n/a}" | ||
| echo | ||
| if [[ "${other}" -gt 0 ]]; then |
Collaborator
There was a problem hiding this comment.
🤖 question 🟢 – This makes any single non-200/503 fatal, including a 504 from a resume that outlives the router timeout or a 000 from one dropped proxy connection mid-run. Is one transient error meant to fail the whole demo run, or should the hard failure be reserved for ok == 0?
Collaborator
Author
There was a problem hiding this comment.
Yes, any non-200/503 is intended to fail the script: the router returns 503 when a parked request's budget expires or the lot is full, whereas 000/4xx/502/504 indicate a broken proxy, auth/routing misconfiguration, or backend error rather than normal parking/shedding behavior.
WorkerPool/parking omitted container resource limits, so the Kubernetes Downward API reported the node's allocatable capacity in /run/worker-capacity. Two workers held all four 1000m actors at once without triggering request parking. Set per-worker container limits to 1 CPU and 1Gi memory so each worker hosts at most one actor. Add a wakeupProbe on /readyz port 80 to parking-template.yaml.tmpl so golden memory snapshots are captured after the counter server binds port 80. In load.sh, wait for each proxy to start listening before generating load, aggregate only actor request logs (excluding .proxy.log files), and exit non-zero when requests fail with a non-200/503 status or when zero requests return 200 OK. Signed-off-by: Bo-Cheng Chu <[email protected]>
Bo-Cheng Chu (bochengchu)
force-pushed
the
parking-demo-fix
branch
from
November 3, 2026 17:55
db9e56f to
b974eb6
Compare
Dmitry Berkovich (dberkov)
approved these changes
Nov 3, 2026
Eitan Yarmush (EItanya)
pushed a commit
to kagent-dev/substrate
that referenced
this pull request
Nov 3, 2026
agent-substrate#3391) Fixes agent-substrate#3287 `WorkerPool/parking` in `demos/parking/parking.yaml.tmpl` did not set container resource limits, so the Kubernetes Downward API exposed the node's allocatable capacity in `/run/worker-capacity`. As a result, 2 workers had room for all 4 `1000m` actors at once and request parking was never triggered. Set per-worker container limits to `1` CPU and `1Gi` memory so each worker hosts at most one actor. Also add a `wakeupProbe` on `/readyz` port `80` to `demos/parking/parking-template.yaml.tmpl` (matching `demos/counter/counter-template.yaml.tmpl`) so golden memory snapshots are captured after the counter HTTP server binds port 80. In `demos/parking/load.sh`: - Write `kubectl ate proxy actor` output to `.proxy` files instead of `.proxy.log` so `cat "${TMP}"/*.log` does not count proxy startup lines as failed requests. - Wait for each local proxy to start listening before generating load, and fail fast with the proxy output if a proxy exits early. - Exit `1` when any request fails with a non-200/503 status (`other > 0`) or when no requests succeed (`ok == 0`). Signed-off-by: Bo-Cheng Chu <[email protected]> (cherry picked from commit a9fe809) Signed-off-by: Eitan Yarmush <[email protected]>
1 of 2 tasks
Dmitry Berkovich (dberkov)
pushed a commit
that referenced
this pull request
Nov 3, 2026
#3391) Fixes #3287 `WorkerPool/parking` in `demos/parking/parking.yaml.tmpl` did not set container resource limits, so the Kubernetes Downward API exposed the node's allocatable capacity in `/run/worker-capacity`. As a result, 2 workers had room for all 4 `1000m` actors at once and request parking was never triggered. Set per-worker container limits to `1` CPU and `1Gi` memory so each worker hosts at most one actor. Also add a `wakeupProbe` on `/readyz` port `80` to `demos/parking/parking-template.yaml.tmpl` (matching `demos/counter/counter-template.yaml.tmpl`) so golden memory snapshots are captured after the counter HTTP server binds port 80. In `demos/parking/load.sh`: - Write `kubectl ate proxy actor` output to `.proxy` files instead of `.proxy.log` so `cat "${TMP}"/*.log` does not count proxy startup lines as failed requests. - Wait for each local proxy to start listening before generating load, and fail fast with the proxy output if a proxy exits early. - Exit `1` when any request fails with a non-200/503 status (`other > 0`) or when no requests succeed (`ok == 0`). Signed-off-by: Bo-Cheng Chu <[email protected]> (cherry picked from commit a9fe809) Signed-off-by: Eitan Yarmush <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3287
WorkerPool/parkingindemos/parking/parking.yaml.tmpldid not set container resource limits, so the Kubernetes Downward API exposed the node's allocatable capacity in/run/worker-capacity. As a result, 2 workers had room for all 41000mactors at once and request parking was never triggered. Set per-worker container limits to1CPU and1Gimemory so each worker hosts at most one actor.Also add a
wakeupProbeon/readyzport80todemos/parking/parking-template.yaml.tmpl(matchingdemos/counter/counter-template.yaml.tmpl) so golden memory snapshots are captured after the counter HTTP server binds port 80.In
demos/parking/load.sh:kubectl ate proxy actoroutput to.proxyfiles instead of.proxy.logsocat "${TMP}"/*.logdoes not count proxy startup lines as failed requests.1when any request fails with a non-200/503 status (other > 0) or when no requests succeed (ok == 0).