Repository navigation
research(interference): deconfound ANE cost at fixed GPU load - #51
Merged
Merged
Conversation
#46 measured releasing the GIL around Core ML predict costing the ANE 13.5% of its throughput, and process placement 9.0%. The GPU client there was closed loop, so each fix also sent the GPU up to 51% more work, and the ANE cost could not be attributed to the binding or placement. Here the GPU runs open loop with the same scheduled arrivals and request seeds in every configuration; a SHA-256 over them is recorded and matches across all runs. At a GPU load of 46.65 req/s, against the product thread path, ANE throughput is +0.5% for process placement and +0.1% for the GIL-releasing thread path, with no tail regression. Both give the same GPU completion path (return P50 0.035 / 0.037 ms against 7.68 ms). So the earlier ANE regression was GPU load, and the no-GIL path meets the criteria in #45 at this load. Process placement is not clearly better.
This was referenced Sep 24, 2026
tc3oliver
added a commit
that referenced
this pull request
Sep 24, 2026
Before switching ANE placement to a worker process, its resource and lifecycle cost had to be measured against the thread placement, for all three models. The criteria were fixed before the data. Result: FAIL. Process placement removes the GPU completion wait for all three models. Its memory (+182 to +236 MB footprint), start-up (+0.9 s), IPC (0.15-0.21 ms per request), shutdown and orphan cleanup pass. It fails on CPU per request (x2.7 and x2.9 for laya and laya-typed-decisions). That comparison is confounded: the closed-loop GPU runs more requests once unblocked. At equal GPU load (#51) the CPU per forward is the same in both placements. laya-multilingual's parent RSS steps up once by 6.7% over 10 create/close cycles, then stays flat. No default, placement data or runtime code changes.
tc3oliver
added a commit
that referenced
this pull request
Sep 24, 2026
) ## Summary This PR adds the acceptance gate for #52: is ANE process placement's resource and lifecycle cost acceptable? Nothing in the runtime, its defaults or `placement.json` changes. Details are in `benchmarks/ane-placement-gate/README.md`. **Result: FAIL**, under criteria fixed before the data. Gate: FAIL. The pre-registered CPU criterion failed under closed-loop load. The result may be confounded because process isolation lets the GPU complete substantially more work. A fixed-offered-load follow-up is required before interpreting this as intrinsic process CPU overhead. | model | correctness | isolation | memory | start-up | CPU | IPC | shutdown | lifecycle | orphan | |---|---|---|---|---|---|---|---|---|---| | laya | pass | pass | pass | pass | **FAIL** | pass | pass | pass | pass | | laya-typed-decisions | pass | pass | pass | pass | **FAIL** | pass | pass | pass | pass | | laya-multilingual | pass | pass | pass | pass | pass | pass | pass | **FAIL** | pass | **CPU (failed):** - Tree CPU per request is ×2.7 / ×2.9 the thread placement's; the limit was ×1.5. The extra CPU is in the GPU worker. - The smoke is closed loop, so the unblocked GPU runs 16–19% more requests. - At equal GPU load (#51), CPU per forward is the same in both placements. - In the unblocked regime (#46), the no-GIL path pays more than process placement: GPU forward CPU 4.54 vs 3.69 ms. **Lifecycle, laya-multilingual (failed):** parent RSS rises 706 → 753 MB over 10 create/close cycles. It is a single step at the 3rd iteration, then flat. This model already ships with process placement. **Isolation (passed):** GPU return P50 with the ANE busy, thread → process: - laya 3.87 → 0.13 ms - laya-typed-decisions 3.77 → 0.13 ms - laya-multilingual 3.15 → 0.07 ms **Other costs (passed):** - Memory: tree `phys_footprint` +182 to +236 MB. - Start-up: +0.85 to 0.88 s. - IPC: 0.15–0.21 ms per request. - `close()`: 0.23 s. - Workers: none left alive or zombie after any close, after an exception inside `with`, or after a parent SIGKILL (they exit within 0.22 s). ## Validation - `ruff check .`, `ruff format --check`, fast suite (266 passed). - `scripts/ane_placement_gate_report.py --check`. - Integration: `test_workers.py`, `test_no_silent_fallback.py` and `test_request_trace_workers.py`, 44 passed. - 0 answer mismatches. **Changes of method, disclosed in the README:** - The first smoke run was discarded. macOS denies USS for child processes, so the memory criterion now uses `phys_footprint`. - Lifecycle and shutdown are judged on the process placement only. Refs #52
tc3oliver
added a commit
that referenced
this pull request
Sep 24, 2026
…oad (#56) ## Summary This PR adjudicates the one CPU criterion of the #54 gate under the condition it was meant to measure: equal GPU offered load. Nothing in the runtime, its defaults or `placement.json` changes. Details are in `benchmarks/ane-equal-load-cpu/README.md`. **Question.** At an identical GPU offered load, does process placement intrinsically cost > 1.5× CPU? **Answer: no.** | model | thread tree CPU ms / request | process | ratio (limit ×1.5) | |---|---|---|---| | laya | 1.82 | 1.80 | ×0.99 | | laya-typed-decisions | 1.80 | 1.83 | ×1.02 | The #54 gate formally remains **FAIL**. This follow-up shows that its failing CPU criterion measured a closed-loop load difference, not intrinsic process overhead. It does not rewrite the gate. **Workload** - #51's fixed-load harness, unchanged: GPU L128 open loop at a 30 req/s target, ANE L128 closed loop, 3 cycles × 20 s. - Thread vs process, both with coremltools; no PyObjC. - Identical in all 4 runs: the arrival SHA-256 `e537412cf4e7026a…` in every window, 693 scheduled arrivals per window, and 31.55 req/s offered and completed. - 3/3 cycles stable for every run; 0 errors, 0 mismatches. **Where the CPU goes** - GPU worker per GPU request: 2.49 → 2.28 ms (laya), 2.51 → 2.29 (typed). #54's closed-loop run measured 6.46 / 6.44. - Parent + ANE worker per ANE request: +4% / +8%. - Tree CPU per second: +2.6% / +4.5%, while the process runs also complete 4.4% / 3.6% more ANE requests. **Sanity:** GPU return P50 is 7.46 → 0.033 ms (laya) and 7.50 → 0.034 (typed). ## Validation - `ruff check .`, `ruff format --check`, fast suite (270 passed; 4 new unit tests for the CPU interpolation). - `benchmarks/ane-equal-load-cpu/analyze.py --check`. Refs #52
This was referenced Sep 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This is round 2 of #45 and is research only: nothing under
laya_apple/changes. Details are inresearch/coreml-placement-deconfounding/README.md.#46 measured the ANE losing 13.5% of its throughput with the GIL released around Core ML predict, and 9.0% with process placement. The GPU client there was closed loop, so unblocking the GPU also sent it 44–51% more work. This round holds the GPU offered load fixed and compares:
Workload:
Workload equality
b98fee08…) matches in all 18 windows.Results (GPU and ANE concurrent)
Validation
ruff check .andruff format --check: pass.analyze.py --check: pass.Refs #45