Repository navigation
feat(runtime): adaptive ANE execution with a slow-state breaker (1.5.0) - #105
Merged
Merged
Conversation
Phase 1 asks whether routing every new ANE request to production A after the frozen C3 detector trips brings user-visible behaviour back to A-like within 1 s. Cells A, B and R (PB-ASYNC plus the breaker) on #102's harness, 12 runs in a fixed order. C3 counts from 1.0 s after the episode start: every hetero start has a prepare transient, and C3 from t0 trips in 40 of 40 production-A windows.
12 runs in the preregistered order (ad82202). B reproduced the slow state in 6 of 6 episodes; R tripped in 12 of 12, all confirmed slow at the trip, and every one returned to sustained A-like behaviour within 164-414 ms of the trip. After the fallback R ran at 0.999 x A's throughput with A's P99. No mismatch, routing failure, request loss, crash or breaker inconsistency.
laya and laya-typed-decisions now run the in-process ANE adaptively by default: 64 synchronous forwards per hetero episode, then the prebound async Core ML path (PB-ASYNC), which keeps GPU completions off the GIL. A breaker watches every completed async request's prepare time; 3 consecutive above 0.3 ms (C3) send the rest of the episode to the 1.4 path. Research: research/coreml-adaptive-breaker (Phase 0: 16/16 slow states caught; Phase 1: 12/12 recovered within 414 ms). ane_handoff=None (the default) uses it where eligible and falls back to the 1.4 path with one warning when pyobjc or the async models are unavailable; False turns it off; True requires it. laya-multilingual keeps process placement. An async predict failure re-runs that forward on coremltools and disables the path instead of failing the request. The completion callback is now attached before a job is queued, so the breaker observes a request before the next forward.
Evidence reuse plus targeted production runs of the combined runtime at 9028137: laya product mix, typed-decisions product mix and soak, multilingual smoke, a bursty product mix and a 66-episode soak, with the release gate as numbers.
…e product-mix soak Phase 2 (laya product mix): 12/12 episodes async, 96.8% residency, 0 trips, GPU return 0.037 ms against A's 4.29, 1.042 x A throughput. Phase 3 (typed-decisions, 78 episodes incl. a 66-episode soak): 0 trips, 93.4% residency, GPU return 0.043 ms against 8.60, 1.038 x A. Addendum 1 merges phases 5 and 6 into one 64-episode productsoak run with a bursty A reference; no gate changes.
Phase 4: laya-multilingual keeps process placement, no adaptive execution, ane_handoff=True rejected. Phase 5 (64-episode product-mix soak): 64/64 async, 91.5% residency, 0 trips, GPU return 0.035 ms against 4.28, 1.042 x A, P99 0.18 ms lower. Across phases 2-5: 154 production episodes, 0 correctness failures, 0 trips, 0 crashes.
Version 1.5.0 in pyproject.toml, laya_apple.__version__ and CHANGELOG. README (English, zh-TW, zh-CN) gains the adaptive-execution section and its limitation; guide, API, no-silent-fallback and reproducibility docs describe ane_handoff and trace every number to research/coreml-adaptive-breaker; release notes in docs/releases/v1.5.0.md.
The release gate passed at 50707a7 (full mode: pytest 405 s, clean install matrix, artifacts, docs versions). It reports dirty only because of eight untracked serve logs that predate this branch; every tracked file is 50707a7. Serve uses adaptive execution by default but was not measured with it: stated in the README limitations (en, zh-TW, zh-CN), serve.md, support-matrix and compatibility. CHANGELOG lists the default change and its one-time warning. Reproducibility rows moved out of the serve block, with the right data paths. The release notes drop an OS version the data does not record. The async parity test skips only on unavailable artifacts and closes its model.
tc3oliver
marked this pull request as ready for review
September 26, 2026 11:41
This was referenced Sep 26, 2026
tc3oliver
added a commit
that referenced
this pull request
Sep 26, 2026
Adds research/README.md, an evidence index of the GPU + ANE research line from #16 to #105, links it from the three READMEs, and cites the open upstream pull request apple/coremltools#2876 with its API boundary. No runtime, benchmark or research data changes.
This was referenced Oct 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #104.
laya-apple 1.5.0: GPU + Neural Engine serving adapts to system load. laya and laya-typed-decisions (
workers,auto) run the in-process ANE on the prebound async Core ML path while it is healthy and fall back to the 1.4 path when a breaker sees a slow state.replay.md): C3 on RequestTrace prepare_ms caught 16/16 recorded slow states, worst 126 ms.phase1_tables.md): falling back to A recovered A-like latency within 164-414 ms in 12/12 slow episodes.val_tables.md): 154 production episodes (laya, typed-decisions incl. 66-episode soak, product-mix soak with bursts), all on the fast path, 0 trips, GPU return 0.035-0.043 ms against 4.28-8.60 ms, throughput 1.038-1.042 x the 1.4 path, 0 correctness failures; multilingual unchanged.benchmarks/release-gate-1.5.0.mdPASS.