Skip to content

research(interference): deconfound ANE cost at fixed GPU load - #51

Merged
tc3oliver merged 1 commit into
mainfrom
research/coreml-placement-deconfounding
Sep 24, 2026
Merged

tc3oliver merged 1 commit into
mainfrom
research/coreml-placement-deconfounding

Conversation

@tc3oliver

Copy link
Copy Markdown
Owner

Summary

This is round 2 of #45 and is research only: nothing under laya_apple/ changes. Details are in research/coreml-placement-deconfounding/README.md.

#46 measured the ANE losing 13.5% of its throughput with the GIL released around Core ML predict, and 9.0% with process placement. The GPU client there was closed loop, so unblocking the GPU also sent it 44–51% more work. This round holds the GPU offered load fixed and compares:

  • A: thread + coremltools (the product path);
  • B: process + coremltools;
  • C: thread + PyObjC, GIL released.

Workload:

  • GPU L128: open loop, Poisson, target 45 req/s (46.65 req/s realized in-window).
  • ANE L128: closed loop.
  • 3 cycles × 20 s. E5 cache not cleared; oMLX stopped.

Workload equality

  • Every window of A, B and C schedules the same 1,020 GPU arrivals, with the same offsets and request seeds. The SHA-256 over them (b98fee08…) matches in all 18 windows.
  • 0 errors and 0 answer mismatches.
  • The queue-stability rule, fixed before the runs, passes in all 9 cycles. A is near saturation: its queue P99 is 420 ms.

Results (GPU and ANE concurrent)

GPU return P50 / P99 GPU e2e P99 ANE req/s ANE predict P99 ANE e2e P99 ANE CPU/request
A 7.677 / 8.961 440.9 97.5 10.20 10.79 0.65
B 0.035 / 0.125 74.9 98.0 (+0.5%) 10.12 10.59 0.57
C 0.037 / 0.203 75.1 97.6 (+0.1%) 10.22 10.63 0.65

Validation

  • ruff check . and ruff format --check: pass.
  • Fast suite: 266 passed, including 6 new tests of the stability rule.
  • analyze.py --check: pass.
  • An independent check recomputed the arrival hashes from the raw offsets, and every README number from the data.

Refs #45

#46 measured releasing the GIL around Core ML predict costing the ANE
13.5% of its throughput, and process placement 9.0%. The GPU client there
was closed loop, so each fix also sent the GPU up to 51% more work, and
the ANE cost could not be attributed to the binding or placement.

Here the GPU runs open loop with the same scheduled arrivals and request
seeds in every configuration; a SHA-256 over them is recorded and matches
across all runs. At a GPU load of 46.65 req/s, against the product thread
path, ANE throughput is +0.5% for process placement and +0.1% for the
GIL-releasing thread path, with no tail regression. Both give the same
GPU completion path (return P50 0.035 / 0.037 ms against 7.68 ms).

So the earlier ANE regression was GPU load, and the no-GIL path meets the
criteria in #45 at this load. Process placement is not clearly better.
@tc3oliver
tc3oliver merged commit dab238f into main Sep 24, 2026
1 check passed
@tc3oliver
tc3oliver deleted the research/coreml-placement-deconfounding branch September 24, 2026 06:55
tc3oliver added a commit that referenced this pull request Sep 24, 2026
Before switching ANE placement to a worker process, its resource and
lifecycle cost had to be measured against the thread placement, for all
three models. The criteria were fixed before the data.

Result: FAIL. Process placement removes the GPU completion wait for all
three models. Its memory (+182 to +236 MB footprint), start-up (+0.9 s),
IPC (0.15-0.21 ms per request), shutdown and orphan cleanup pass.

It fails on CPU per request (x2.7 and x2.9 for laya and
laya-typed-decisions). That comparison is confounded: the closed-loop
GPU runs more requests once unblocked. At equal GPU load (#51) the CPU
per forward is the same in both placements. laya-multilingual's parent
RSS steps up once by 6.7% over 10 create/close cycles, then stays flat.

No default, placement data or runtime code changes.
tc3oliver added a commit that referenced this pull request Sep 24, 2026
)

## Summary

This PR adds the acceptance gate for #52: is ANE process placement's
resource and lifecycle cost acceptable? Nothing in the runtime, its
defaults or `placement.json` changes. Details are in
`benchmarks/ane-placement-gate/README.md`.

**Result: FAIL**, under criteria fixed before the data.

Gate: FAIL. The pre-registered CPU criterion failed under closed-loop
load. The result may be confounded because process isolation lets the
GPU complete substantially more work. A fixed-offered-load follow-up is
required before interpreting this as intrinsic process CPU overhead.

| model | correctness | isolation | memory | start-up | CPU | IPC |
shutdown | lifecycle | orphan |
|---|---|---|---|---|---|---|---|---|---|
| laya | pass | pass | pass | pass | **FAIL** | pass | pass | pass |
pass |
| laya-typed-decisions | pass | pass | pass | pass | **FAIL** | pass |
pass | pass | pass |
| laya-multilingual | pass | pass | pass | pass | pass | pass | pass |
**FAIL** | pass |

**CPU (failed):**
- Tree CPU per request is ×2.7 / ×2.9 the thread placement's; the limit
was ×1.5. The extra CPU is in the GPU worker.
- The smoke is closed loop, so the unblocked GPU runs 16–19% more
requests.
- At equal GPU load (#51), CPU per forward is the same in both
placements.
- In the unblocked regime (#46), the no-GIL path pays more than process
placement: GPU forward CPU 4.54 vs 3.69 ms.

**Lifecycle, laya-multilingual (failed):** parent RSS rises 706 → 753 MB
over 10 create/close cycles. It is a single step at the 3rd iteration,
then flat. This model already ships with process placement.

**Isolation (passed):** GPU return P50 with the ANE busy, thread →
process:
- laya 3.87 → 0.13 ms
- laya-typed-decisions 3.77 → 0.13 ms
- laya-multilingual 3.15 → 0.07 ms

**Other costs (passed):**
- Memory: tree `phys_footprint` +182 to +236 MB.
- Start-up: +0.85 to 0.88 s.
- IPC: 0.15–0.21 ms per request.
- `close()`: 0.23 s.
- Workers: none left alive or zombie after any close, after an exception
inside `with`, or after a parent SIGKILL (they exit within 0.22 s).

## Validation

- `ruff check .`, `ruff format --check`, fast suite (266 passed).
- `scripts/ane_placement_gate_report.py --check`.
- Integration: `test_workers.py`, `test_no_silent_fallback.py` and
`test_request_trace_workers.py`, 44 passed.
- 0 answer mismatches.

**Changes of method, disclosed in the README:**
- The first smoke run was discarded. macOS denies USS for child
processes, so the memory criterion now uses `phys_footprint`.
- Lifecycle and shutdown are judged on the process placement only.

Refs #52
tc3oliver added a commit that referenced this pull request Sep 24, 2026
…oad (#56)

## Summary

This PR adjudicates the one CPU criterion of the #54 gate under the
condition it was meant to measure: equal GPU offered load. Nothing in
the runtime, its defaults or `placement.json` changes. Details are in
`benchmarks/ane-equal-load-cpu/README.md`.

**Question.** At an identical GPU offered load, does process placement
intrinsically cost > 1.5× CPU?

**Answer: no.**

| model | thread tree CPU ms / request | process | ratio (limit ×1.5) |
|---|---|---|---|
| laya | 1.82 | 1.80 | ×0.99 |
| laya-typed-decisions | 1.80 | 1.83 | ×1.02 |

The #54 gate formally remains **FAIL**. This follow-up shows that its
failing CPU criterion measured a closed-loop load difference, not
intrinsic process overhead. It does not rewrite the gate.

**Workload**
- #51's fixed-load harness, unchanged: GPU L128 open loop at a 30 req/s
target, ANE L128 closed loop, 3 cycles × 20 s.
- Thread vs process, both with coremltools; no PyObjC.
- Identical in all 4 runs: the arrival SHA-256 `e537412cf4e7026a…` in
every window, 693 scheduled arrivals per window, and 31.55 req/s offered
and completed.
- 3/3 cycles stable for every run; 0 errors, 0 mismatches.

**Where the CPU goes**
- GPU worker per GPU request: 2.49 → 2.28 ms (laya), 2.51 → 2.29
(typed). #54's closed-loop run measured 6.46 / 6.44.
- Parent + ANE worker per ANE request: +4% / +8%.
- Tree CPU per second: +2.6% / +4.5%, while the process runs also
complete 4.4% / 3.6% more ANE requests.

**Sanity:** GPU return P50 is 7.46 → 0.033 ms (laya) and 7.50 → 0.034
(typed).

## Validation

- `ruff check .`, `ruff format --check`, fast suite (270 passed; 4 new
unit tests for the CPU interpolation).
- `benchmarks/ane-equal-load-cpu/analyze.py --check`.

Refs #52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant