Repository navigation
Closed
Labels
Description
Summary & Root Cause
- Worker capacity comes from the WorkerPool's container limits. With no limits, each
parkingworker advertises the whole node's allocatable capacity (3920mCPU on the test nodes). A 1000m actor then fits three times per worker, so 2 workers hold 6 actors, and the demo's 4 actors never contend. load.shtargets a plaintext router port that no longer exists and sends no client cert, so every curl records000. The verdict counts only503as failure, so 0 successes with 0503s prints=> 0 failures under saturationand exits 0. Thetotal == 0guard does not fire because000lines are still recorded.
Affected Code Locations
demos/parking/parking.yaml.tmpl:42-48(WorkerPool/parking:replicas: 2with the comment "Intentionally small: 2 workers, so 3+ concurrently-active actors saturate the pool", and notemplate.resources; the file ends at line 48)demos/counter/counter.yaml.tmpl:46-63(comparison:template.resources.limits: {cpu: "1", memory: 1Gi}; the comment notes that capacity comes from the limits)demos/parking/parking-template.yaml.tmpl:29-34(per-actorcpuMillicores: 1000,memoryBytes: 536870912; the comment says "The pool sets no worker limits, so the worker offers the node's allocatable")demos/parking/load.sh:40, 45, 61(ROUTER="http://localhost:8000", prerequisiteport-forward ... svc/atenet-router 8000:80)demos/parking/load.sh:108-110(plaincurlwith no TLS or client cert)demos/parking/load.sh:133-146(busycounts only503;if [[ "${busy}" -eq 0 ]]prints the success line; there is no non-zero exit forok == 0orother > 0)
Detailed Reproduction Guide
-
Deploy and inspect the pool:
go run ./cmd/ate-setup deploy demo parking kubectl get workerpool -n ate-demo-parking parking -o jsonpath='{.spec.replicas} resources={.spec.template.resources}' -
Run the load script as documented:
kubectl port-forward -n ate-system svc/atenet-router 8000:80 & # fails: no port 80 ./demos/parking/load.sh -d 5; echo "exit=$?"
-
Wake
p1..p4concurrently over mTLS (from a client pod as in S-2) and check placement:for a in p1 p2 p3 p4; do kubectl exec -n test-client test-client -- curl -sS -o /dev/null -w "$a: %{http_code} %{time_total}s\n" \ --cacert /run/servicedns-ca/trust-bundle.pem --cert /run/podidentity/credential-bundle.pem --key /run/podidentity/credential-bundle.pem \ -H "ate-target-actor: ate-demo-parking/$a" https://atenet-router.ate-system.svc/ & done; wait kubectl ate get workers -n ate-demo-parking
Expected vs. Actual Behavior
- Expected: With 2 workers sized for one 1000m actor each, waking 4 actors runs 2 and parks 2.
load.shexits non-zero when requests fail to connect or none succeed. - Actual: All 4 actors run at once (2 per worker) with no parking.
load.shreports 0 OK, 0503, all "other", prints the success line, and exits 0.
Observed Output & Logs
$ kubectl get workerpool -n ate-demo-parking parking -o jsonpath='{.spec.replicas} resources={.spec.resources}'
2 resources=
$ ./demos/parking/load.sh -d 5 # exit 0
total requests : 28
200 OK : 0
503 unavailable: 0
other : 28
=> 0 failures under saturation: parking absorbed the contention.
p1: 200 0.438695s | p3: 200 0.501199s | p4: 200 0.497656s | p2: 200 0.542469s
parking WORKER_STATE_ACTIVE 2/1000 2000/3920 ... parking-65bf544cf-6wqrn
parking WORKER_STATE_ACTIVE 2/1000 2000/3920 ... parking-65bf544cf-bfr9g parking 3/1000 3000/3920 (worker A)
parking 1/1000 1000/3920 (worker B)
Activity
[-]demos/parking WorkerPool omits resources.limits (failing to trigger parking with 4 actors) and load.sh exits 0 reporting "0 failures" when 100% of requests fail[/-][+]load.sh exits 0 reporting "0 failures" when 100% of requests fail AND demos/parking WorkerPool omits resources.limits (failing to trigger parking with 4 actors).[/+]Delete hack/install-ate.sh and hack/install-ate-kind.sh (agent-substr…
demos/parking: size each worker for one actor and fail load.sh on req… (
demos/parking: size each worker for one actor and fail load.sh on req… (
demos/parking: size each worker for one actor and fail load.sh on req… (