Repository navigation
Commit a45caba
[fix] Run the sampler weight broadcast off the event loop (#2524)
# What does this PR do?
Moves the sampler weight transfer in
`WorkerDispatch.save_weights_for_sampler` off the trainer's event loop.
TLDR: `_broadcast_and_finish()` is a blocking `ray.get` called inline
from an `async def`. Under the fully-async trainer, that freezes every
generator coroutine on the loop for the whole transfer. It now runs
through `asyncio.to_thread`, and a cancellation waits for it, so
generation cannot resume mid-transfer. The offload-KV path also wakes
the KV cache before resuming after a failed or cancelled sync.
## Why
1. All six call sites in `save_weights_for_sampler` call
`_broadcast_and_finish()` synchronously.
`_broadcast_to_inference_engines` is `deadline.ray_get(...)` over the
policy workers, so the event loop is blocked for the whole broadcast.
2. In the fully-async trainer, the `_run_generate_for_a_group_loop`
workers, and whatever HTTP clients and environments they drive, run on
that same loop. The engines are paused during the sync, but non-engine
work (environment steps, tool calls, reward computation) stalls with
them, so the overlap the fully-async trainer is meant to provide is lost
for the length of the transfer. For a large MoE policy we measured
`timing/sync_weights_only_transfer` at 38–43 s on every step.
3. A timeout armed by a coroutine just before the freeze fires on the
first tick after it, even though the awaited operation was healthy.
In-tree, that includes the 10 s bound on `finish_session`. In our case
it was a custom generator's 10 s HTTP read. The resulting `TimeoutError`
became the generator's `BackgroundFailure`, was re-raised at the next
`_drain_next_group`, and killed the run. That failure is
timing-dependent, so it can strike on any step once the transfer is
longer than the shortest in-flight timeout.
<details><summary>Second-order effect seen with uvloop 0.22.1</summary>
The cancelled `create_connection(sock=...)` behind that timeout hit
[MagicStack/uvloop#738](MagicStack/uvloop#738),
a double close of a recycled fd. The next few generator groups then
failed with `RuntimeError('File descriptor N is used by transport
...')`. That bug is fixed in uvloop 0.23.0. This PR removes the freeze
that triggered it.
</details>
## Solution
`_broadcast_and_finish` is now `async`. It runs the broadcast plus
`_finish_weight_sync` via `asyncio.to_thread` and is awaited at every
call site. The pause, sleep, wake and resume brackets around it, their
ordering, and the trailing `increment_weight_version()` are unchanged.
- **Engine safety is unchanged.** Requests are kept off half-updated
weights by the engine-side bracket (`pause_generation` in KEEP mode, or
sleep/wake), not by the frozen driver loop. Generators that run outside
the driver process already send requests during a sync. Only the trainer
coroutine issues engine control calls, and it still awaits the sync
before moving on.
- **Cancellation.** A cancellation that arrives mid-transfer, for
example on shutdown, waits for the transfer to finish (`asyncio.wait` on
the thread's future) and then propagates, chained to the transfer's
error if it failed. Without that wait,
`cleanup_preserving_primary(resume_generation)` would resume generation
on the NCCL and offload-KV paths while the policy workers were still
broadcasting. The blocking version could not be cancelled mid-transfer
at all, and this keeps that guarantee.
- **Deadlines.** `step_deadline` state is ContextVar-based and
`to_thread` copies the context, so the threaded `deadline.ray_get` still
gets `remaining()` as its timeout. On expiry, `asyncio.timeout` cancels
the awaiting task and the thread's `ray.get` times out at the same
instant, so the wait adds nothing. The await is wrapped in
`deadline.operation("broadcast_to_inference_engines")`, so a
`StepTimeoutError` names the same operation whichever side fires first.
- **Thread safety.** The thread makes only Ray calls (actor-method
dispatch plus `ray.get`, both thread-safe). `last_weight_sync_seconds`
is set after the await. The colocated `_finish_weight_sync` → `_offload`
path is likewise `ray.get` only.
- **KV cache on an interrupted sync (pre-existing on `main`).** On the
fully-async offload-KV path, a failed or cancelled broadcast skipped
`wake_for_weight_sync(kv_cache)`, and the resume cleanup then scheduled
the frozen requests onto a sleeping KV cache. The KV wake is now its own
`cleanup_preserving_primary` bracket inside the resume bracket, as the
synchronous-trainer path already does for `wake_up(kv_cache)`.
- **Not done: per-caller fixes,** such as a longer timeout in each HTTP
client. Any coroutine on the trainer loop can be hit, and the freeze
grows with model size.
## Test plan
New CPU tests in `TestSaveWeights`, with a fake `time.sleep` broadcast:
- `test_broadcast_does_not_block_event_loop` uses a 0.5 s broadcast. It
arms a 0.2 s `asyncio.timeout` around a 0.05 s sleep while the sync
runs, then asserts that the timeout did not fire, that the sync was
still in progress, and that `last_weight_sync_seconds >= 0.5`. On `main`
the timeout fires, which is the production failure mode.
- `test_cancellation_waits_for_the_transfer_before_resuming` cancels the
sync during a 0.3 s broadcast and asserts the order `broadcast_done` →
`resumed`. With `to_thread` but no wait it fails with `['resumed']`. On
`main` it fails with `DID NOT RAISE`, because the blocking sync
completes before the cancellation can land.
- `test_cancellation_survives_a_failed_transfer` cancels, then makes the
broadcast fail. It asserts that `CancelledError` propagates with the
`RuntimeError` as its `__cause__`.
- `test_offload_kv_wakes_kv_cache_before_resuming[failure|cancellation]`
runs the fully-async offload-KV path and asserts the order
`wake_weights` → `broadcast_done` → `wake_kv_cache` → `resumed`. On
`main` the failure case resumes without the KV wake.
All three fail on this PR's first commit.
The existing `TestSaveWeights` cases run unchanged. They cover bracket
ordering, resume after a broadcast failure, and the delta, LoRA and
colocated paths.
| command (CPU, `RAY_ADDRESS=local`) | `main` b304652 | this PR |
|---|---|---|
| `uv run --isolated --extra skyrl-train --extra dev pytest
tests/backends/skyrl_train/distributed/test_worker_dispatch.py` | 2
failed / 22 passed | 27 passed |
| `... pytest tests/train/test_fully_async_trainer.py
tests/train/test_async_utils.py tests/train/test_trainer.py -m "not
vllm"` | — | 32 passed |
| `pre-commit run --files <changed>` | — | ruff, black, secrets pass |
No GPU CI was run. The broadcast itself is unchanged; only the thread it
runs on moved.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Dominic Yurk <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>1 parent 49949ad commit a45caba
2 files changed
Lines changed: 134 additions & 11 deletions
File tree
- skyrl/backends/skyrl_train/workers
- tests/backends/skyrl_train/distributed
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| 11 | + | |
11 | 12 | | |
12 | 13 | | |
13 | 14 | | |
| |||
798 | 799 | | |
799 | 800 | | |
800 | 801 | | |
801 | | - | |
| 802 | + | |
| 803 | + | |
| 804 | + | |
| 805 | + | |
| 806 | + | |
802 | 807 | | |
803 | 808 | | |
| 809 | + | |
| 810 | + | |
| 811 | + | |
| 812 | + | |
| 813 | + | |
| 814 | + | |
| 815 | + | |
804 | 816 | | |
805 | 817 | | |
806 | 818 | | |
807 | 819 | | |
808 | 820 | | |
809 | 821 | | |
810 | | - | |
811 | | - | |
| 822 | + | |
| 823 | + | |
| 824 | + | |
| 825 | + | |
| 826 | + | |
| 827 | + | |
| 828 | + | |
| 829 | + | |
| 830 | + | |
| 831 | + | |
| 832 | + | |
812 | 833 | | |
813 | 834 | | |
814 | 835 | | |
| |||
818 | 839 | | |
819 | 840 | | |
820 | 841 | | |
821 | | - | |
| 842 | + | |
822 | 843 | | |
823 | 844 | | |
824 | 845 | | |
| |||
827 | 848 | | |
828 | 849 | | |
829 | 850 | | |
830 | | - | |
| 851 | + | |
831 | 852 | | |
832 | 853 | | |
833 | 854 | | |
| |||
843 | 864 | | |
844 | 865 | | |
845 | 866 | | |
846 | | - | |
847 | | - | |
848 | | - | |
| 867 | + | |
| 868 | + | |
| 869 | + | |
| 870 | + | |
| 871 | + | |
| 872 | + | |
| 873 | + | |
| 874 | + | |
849 | 875 | | |
850 | 876 | | |
851 | 877 | | |
| |||
855 | 881 | | |
856 | 882 | | |
857 | 883 | | |
858 | | - | |
| 884 | + | |
859 | 885 | | |
860 | 886 | | |
861 | 887 | | |
862 | 888 | | |
863 | 889 | | |
864 | 890 | | |
865 | | - | |
| 891 | + | |
866 | 892 | | |
867 | 893 | | |
868 | 894 | | |
869 | 895 | | |
870 | 896 | | |
871 | | - | |
| 897 | + | |
872 | 898 | | |
873 | 899 | | |
874 | 900 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
| 9 | + | |
9 | 10 | | |
10 | 11 | | |
11 | 12 | | |
| |||
259 | 260 | | |
260 | 261 | | |
261 | 262 | | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
262 | 359 | | |
263 | 360 | | |
264 | 361 | | |
| |||
0 commit comments