Skip to content

feat(runtime): adaptive ANE execution with a slow-state breaker (1.5.0) - #105

Merged
tc3oliver merged 9 commits into
mainfrom
release/1.5-adaptive-execution
Sep 26, 2026
Merged

tc3oliver merged 9 commits into
mainfrom
release/1.5-adaptive-execution

Conversation

@tc3oliver

@tc3oliver tc3oliver commented Sep 26, 2026 •

Copy link
Copy Markdown
Owner

Closes #104.

laya-apple 1.5.0: GPU + Neural Engine serving adapts to system load. laya and laya-typed-decisions (workers, auto) run the in-process ANE on the prebound async Core ML path while it is healthy and fall back to the 1.4 path when a breaker sees a slow state.

  • Phase 0 (replay.md): C3 on RequestTrace prepare_ms caught 16/16 recorded slow states, worst 126 ms.
  • Phase 1 (phase1_tables.md): falling back to A recovered A-like latency within 164-414 ms in 12/12 slow episodes.
  • Runtime: two independent reviews; fast suite, integration/parity/ANE (incl. async path vs goldens) on Apple silicon.
  • Release validation (val_tables.md): 154 production episodes (laya, typed-decisions incl. 66-episode soak, product-mix soak with bursts), all on the fast path, 0 trips, GPU return 0.035-0.043 ms against 4.28-8.60 ms, throughput 1.038-1.042 x the 1.4 path, 0 correctness failures; multilingual unchanged.
  • Release gate: benchmarks/release-gate-1.5.0.md PASS.
  • Docs: README (en/zh-TW/zh-CN), guide, api, no-silent-fallback, serve, support matrix, compatibility, reproducibility, CHANGELOG, release notes.

Six candidate detectors on RequestTrace prepare_ms, replayed over the 67 recorded
PB-ASYNC / H / P hetero episodes of #92, #96, #99, #102 and #103, with a latency
ground truth that never reads prepare. All six catch 16/16 sustained episodes;
C3 (3 consecutive host-slow ANE requests) is recommended.
Phase 1 asks whether routing every new ANE request to production A after the frozen
C3 detector trips brings user-visible behaviour back to A-like within 1 s. Cells A,
B and R (PB-ASYNC plus the breaker) on #102's harness, 12 runs in a fixed order.
C3 counts from 1.0 s after the episode start: every hetero start has a prepare
transient, and C3 from t0 trips in 40 of 40 production-A windows.
12 runs in the preregistered order (ad82202). B reproduced the slow state in 6 of 6
episodes; R tripped in 12 of 12, all confirmed slow at the trip, and every one
returned to sustained A-like behaviour within 164-414 ms of the trip. After the
fallback R ran at 0.999 x A's throughput with A's P99. No mismatch, routing
failure, request loss, crash or breaker inconsistency.
laya and laya-typed-decisions now run the in-process ANE adaptively by default:
64 synchronous forwards per hetero episode, then the prebound async Core ML path
(PB-ASYNC), which keeps GPU completions off the GIL. A breaker watches every
completed async request's prepare time; 3 consecutive above 0.3 ms (C3) send the
rest of the episode to the 1.4 path. Research: research/coreml-adaptive-breaker
(Phase 0: 16/16 slow states caught; Phase 1: 12/12 recovered within 414 ms).

ane_handoff=None (the default) uses it where eligible and falls back to the 1.4
path with one warning when pyobjc or the async models are unavailable; False
turns it off; True requires it. laya-multilingual keeps process placement. An
async predict failure re-runs that forward on coremltools and disables the path
instead of failing the request. The completion callback is now attached before a
job is queued, so the breaker observes a request before the next forward.
Evidence reuse plus targeted production runs of the combined runtime at 9028137:
laya product mix, typed-decisions product mix and soak, multilingual smoke, a
bursty product mix and a 66-episode soak, with the release gate as numbers.
…e product-mix soak

Phase 2 (laya product mix): 12/12 episodes async, 96.8% residency, 0 trips, GPU
return 0.037 ms against A's 4.29, 1.042 x A throughput. Phase 3 (typed-decisions,
78 episodes incl. a 66-episode soak): 0 trips, 93.4% residency, GPU return
0.043 ms against 8.60, 1.038 x A. Addendum 1 merges phases 5 and 6 into one
64-episode productsoak run with a bursty A reference; no gate changes.
Phase 4: laya-multilingual keeps process placement, no adaptive execution,
ane_handoff=True rejected. Phase 5 (64-episode product-mix soak): 64/64 async,
91.5% residency, 0 trips, GPU return 0.035 ms against 4.28, 1.042 x A, P99
0.18 ms lower. Across phases 2-5: 154 production episodes, 0 correctness
failures, 0 trips, 0 crashes.
Version 1.5.0 in pyproject.toml, laya_apple.__version__ and CHANGELOG. README
(English, zh-TW, zh-CN) gains the adaptive-execution section and its limitation;
guide, API, no-silent-fallback and reproducibility docs describe ane_handoff and
trace every number to research/coreml-adaptive-breaker; release notes in
docs/releases/v1.5.0.md.
The release gate passed at 50707a7 (full mode: pytest 405 s, clean install
matrix, artifacts, docs versions). It reports dirty only because of eight
untracked serve logs that predate this branch; every tracked file is 50707a7.

Serve uses adaptive execution by default but was not measured with it: stated
in the README limitations (en, zh-TW, zh-CN), serve.md, support-matrix and
compatibility. CHANGELOG lists the default change and its one-time warning.
Reproducibility rows moved out of the serve block, with the right data paths.
The release notes drop an OS version the data does not record. The async
parity test skips only on unavailable artifacts and closes its model.
@tc3oliver
tc3oliver marked this pull request as ready for review September 26, 2026 11:41
@tc3oliver tc3oliver changed the title research(execution): adaptive PB-ASYNC execution with a slow-state breaker feat(runtime): adaptive ANE execution with a slow-state breaker (1.5.0) Sep 26, 2026
@tc3oliver
tc3oliver merged commit 5e0edcf into main Sep 26, 2026
1 check passed
@tc3oliver
tc3oliver deleted the release/1.5-adaptive-execution branch September 26, 2026 11:46
tc3oliver added a commit that referenced this pull request Sep 26, 2026
Adds research/README.md, an evidence index of the GPU + ANE research line from #16 to #105, links it from the three READMEs, and cites the open upstream pull request apple/coremltools#2876 with its API boundary. No runtime, benchmark or research data changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1.5 adaptive execution: PB-ASYNC circuit breaker

1 participant