Skip to content

Commit a45caba

Browse files
dyurk-lilaclaude
andauthored
[fix] Run the sampler weight broadcast off the event loop (#2524)
# What does this PR do? Moves the sampler weight transfer in `WorkerDispatch.save_weights_for_sampler` off the trainer's event loop. TLDR: `_broadcast_and_finish()` is a blocking `ray.get` called inline from an `async def`. Under the fully-async trainer, that freezes every generator coroutine on the loop for the whole transfer. It now runs through `asyncio.to_thread`, and a cancellation waits for it, so generation cannot resume mid-transfer. The offload-KV path also wakes the KV cache before resuming after a failed or cancelled sync. ## Why 1. All six call sites in `save_weights_for_sampler` call `_broadcast_and_finish()` synchronously. `_broadcast_to_inference_engines` is `deadline.ray_get(...)` over the policy workers, so the event loop is blocked for the whole broadcast. 2. In the fully-async trainer, the `_run_generate_for_a_group_loop` workers, and whatever HTTP clients and environments they drive, run on that same loop. The engines are paused during the sync, but non-engine work (environment steps, tool calls, reward computation) stalls with them, so the overlap the fully-async trainer is meant to provide is lost for the length of the transfer. For a large MoE policy we measured `timing/sync_weights_only_transfer` at 38–43 s on every step. 3. A timeout armed by a coroutine just before the freeze fires on the first tick after it, even though the awaited operation was healthy. In-tree, that includes the 10 s bound on `finish_session`. In our case it was a custom generator's 10 s HTTP read. The resulting `TimeoutError` became the generator's `BackgroundFailure`, was re-raised at the next `_drain_next_group`, and killed the run. That failure is timing-dependent, so it can strike on any step once the transfer is longer than the shortest in-flight timeout. <details><summary>Second-order effect seen with uvloop 0.22.1</summary> The cancelled `create_connection(sock=...)` behind that timeout hit [MagicStack/uvloop#738](MagicStack/uvloop#738), a double close of a recycled fd. The next few generator groups then failed with `RuntimeError('File descriptor N is used by transport ...')`. That bug is fixed in uvloop 0.23.0. This PR removes the freeze that triggered it. </details> ## Solution `_broadcast_and_finish` is now `async`. It runs the broadcast plus `_finish_weight_sync` via `asyncio.to_thread` and is awaited at every call site. The pause, sleep, wake and resume brackets around it, their ordering, and the trailing `increment_weight_version()` are unchanged. - **Engine safety is unchanged.** Requests are kept off half-updated weights by the engine-side bracket (`pause_generation` in KEEP mode, or sleep/wake), not by the frozen driver loop. Generators that run outside the driver process already send requests during a sync. Only the trainer coroutine issues engine control calls, and it still awaits the sync before moving on. - **Cancellation.** A cancellation that arrives mid-transfer, for example on shutdown, waits for the transfer to finish (`asyncio.wait` on the thread's future) and then propagates, chained to the transfer's error if it failed. Without that wait, `cleanup_preserving_primary(resume_generation)` would resume generation on the NCCL and offload-KV paths while the policy workers were still broadcasting. The blocking version could not be cancelled mid-transfer at all, and this keeps that guarantee. - **Deadlines.** `step_deadline` state is ContextVar-based and `to_thread` copies the context, so the threaded `deadline.ray_get` still gets `remaining()` as its timeout. On expiry, `asyncio.timeout` cancels the awaiting task and the thread's `ray.get` times out at the same instant, so the wait adds nothing. The await is wrapped in `deadline.operation("broadcast_to_inference_engines")`, so a `StepTimeoutError` names the same operation whichever side fires first. - **Thread safety.** The thread makes only Ray calls (actor-method dispatch plus `ray.get`, both thread-safe). `last_weight_sync_seconds` is set after the await. The colocated `_finish_weight_sync` → `_offload` path is likewise `ray.get` only. - **KV cache on an interrupted sync (pre-existing on `main`).** On the fully-async offload-KV path, a failed or cancelled broadcast skipped `wake_for_weight_sync(kv_cache)`, and the resume cleanup then scheduled the frozen requests onto a sleeping KV cache. The KV wake is now its own `cleanup_preserving_primary` bracket inside the resume bracket, as the synchronous-trainer path already does for `wake_up(kv_cache)`. - **Not done: per-caller fixes,** such as a longer timeout in each HTTP client. Any coroutine on the trainer loop can be hit, and the freeze grows with model size. ## Test plan New CPU tests in `TestSaveWeights`, with a fake `time.sleep` broadcast: - `test_broadcast_does_not_block_event_loop` uses a 0.5 s broadcast. It arms a 0.2 s `asyncio.timeout` around a 0.05 s sleep while the sync runs, then asserts that the timeout did not fire, that the sync was still in progress, and that `last_weight_sync_seconds >= 0.5`. On `main` the timeout fires, which is the production failure mode. - `test_cancellation_waits_for_the_transfer_before_resuming` cancels the sync during a 0.3 s broadcast and asserts the order `broadcast_done` → `resumed`. With `to_thread` but no wait it fails with `['resumed']`. On `main` it fails with `DID NOT RAISE`, because the blocking sync completes before the cancellation can land. - `test_cancellation_survives_a_failed_transfer` cancels, then makes the broadcast fail. It asserts that `CancelledError` propagates with the `RuntimeError` as its `__cause__`. - `test_offload_kv_wakes_kv_cache_before_resuming[failure|cancellation]` runs the fully-async offload-KV path and asserts the order `wake_weights` → `broadcast_done` → `wake_kv_cache` → `resumed`. On `main` the failure case resumes without the KV wake. All three fail on this PR's first commit. The existing `TestSaveWeights` cases run unchanged. They cover bracket ordering, resume after a broadcast failure, and the delta, LoRA and colocated paths. | command (CPU, `RAY_ADDRESS=local`) | `main` b304652 | this PR | |---|---|---| | `uv run --isolated --extra skyrl-train --extra dev pytest tests/backends/skyrl_train/distributed/test_worker_dispatch.py` | 2 failed / 22 passed | 27 passed | | `... pytest tests/train/test_fully_async_trainer.py tests/train/test_async_utils.py tests/train/test_trainer.py -m "not vllm"` | — | 32 passed | | `pre-commit run --files <changed>` | — | ruff, black, secrets pass | No GPU CI was run. The broadcast itself is unchanged; only the thread it runs on moved. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Dominic Yurk <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent 49949ad commit a45caba

2 files changed

Lines changed: 134 additions & 11 deletions

File tree

‎skyrl/backends/skyrl_train/workers/worker_dispatch.py‎

Lines changed: 37 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88
The trainer interacts with the worker dispatch if all models are always on GPU.
99
"""
1010

11+
import asyncio
1112
import time
1213
from dataclasses import dataclass
1314
from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple
@@ -798,17 +799,37 @@ async def save_weights_for_sampler(self, model_id: Optional[str] = None) -> None
798799

799800
adapter_only_sync = self.colocate_all and self._is_lora_no_merge()
800801

801-
def _broadcast_and_finish() -> None:
802+
def _transfer() -> None:
803+
self._broadcast_to_inference_engines(self._inference_engine_client, model_id=model_id)
804+
self._finish_weight_sync(adapter_only_sync=adapter_only_sync)
805+
806+
async def _broadcast_and_finish() -> None:
802807
"""The weight transfer proper, timed on its own.
803808
809+
The transfer blocks on ``ray.get`` for the whole broadcast, so it runs in a
810+
worker thread: on the event loop it would stall every coroutine sharing the
811+
loop (the fully-async generators) and expire their timeouts. A cancellation
812+
waits for the transfer to finish before propagating (chained to the transfer's
813+
error, if any), so the enclosing resume bracket never resumes generation onto
814+
partially transferred weights.
815+
804816
``last_weight_sync_seconds`` is surfaced as
805817
``timing/sync_weights_only_transfer`` via :meth:`get_timing_metrics`,
806818
alongside the trainer's own ``sync_weights`` timer, which wraps the
807819
enclosing pause/resume bracket too.
808820
"""
809821
start = time.perf_counter()
810-
self._broadcast_to_inference_engines(self._inference_engine_client, model_id=model_id)
811-
self._finish_weight_sync(adapter_only_sync=adapter_only_sync)
822+
with deadline.operation("broadcast_to_inference_engines"):
823+
transfer = asyncio.ensure_future(asyncio.to_thread(_transfer))
824+
cancelled = False
825+
while not transfer.done():
826+
try:
827+
await asyncio.wait({transfer})
828+
except asyncio.CancelledError:
829+
cancelled = True
830+
if cancelled:
831+
raise asyncio.CancelledError() from transfer.exception()
832+
transfer.result()
812833
self.last_weight_sync_seconds = time.perf_counter() - start
813834

814835
# Sync weights to inference engine
@@ -818,7 +839,7 @@ def _broadcast_and_finish() -> None:
818839
self.ensure_active_adapter("policy", model_id, require_model_resident=not adapter_only_sync)
819840
if self.colocate_all:
820841
await self._inference_engine_client.wake_up(tags=["weights"])
821-
_broadcast_and_finish()
842+
await _broadcast_and_finish()
822843
await self._inference_engine_client.wake_up(tags=["kv_cache"])
823844
else:
824845
strategy = self.cfg.trainer.strategy
@@ -827,7 +848,7 @@ def _broadcast_and_finish() -> None:
827848
strategy == "megatron" and self.cfg.trainer.policy.megatron_config.lora_config.merge_lora
828849
):
829850
# in-place lora case (mostly for multi-tenant training) - no need to pause - can just rely on load_lora_adapter to swap adapter in place
830-
_broadcast_and_finish()
851+
await _broadcast_and_finish()
831852
elif self.cfg.generator.inference_engine.offload_kv_for_weight_sync:
832853
# Sleep the engine to free GPU memory during the sync (wake weights,
833854
# broadcast, wake KV cache) so gpu_memory_utilization can run higher.
@@ -843,9 +864,14 @@ def _broadcast_and_finish() -> None:
843864
self._inference_engine_client.resume_generation, "resume_generation"
844865
):
845866
await self._inference_engine_client.sleep_for_weight_sync(offload_kv=offload_kv)
846-
await self._inference_engine_client.wake_for_weight_sync(tags=["weights"])
847-
_broadcast_and_finish()
848-
await self._inference_engine_client.wake_for_weight_sync(tags=["kv_cache"])
867+
# Wake the KV cache even if the sync fails, so the resume above never
868+
# schedules the frozen requests onto a sleeping KV cache.
869+
async with cleanup_preserving_primary(
870+
lambda: self._inference_engine_client.wake_for_weight_sync(tags=["kv_cache"]),
871+
"wake_for_weight_sync(kv_cache)",
872+
):
873+
await self._inference_engine_client.wake_for_weight_sync(tags=["weights"])
874+
await _broadcast_and_finish()
849875
else:
850876
# Synchronous trainer: generation is complete at sync time, so there
851877
# are no in-flight requests to preserve. A plain sleep (discards KV,
@@ -855,20 +881,20 @@ def _broadcast_and_finish() -> None:
855881
lambda: self._inference_engine_client.wake_up(tags=["kv_cache"]), "wake_up(kv_cache)"
856882
):
857883
await self._inference_engine_client.wake_up(tags=["weights"])
858-
_broadcast_and_finish()
884+
await _broadcast_and_finish()
859885
else:
860886
# Non-colocated single tenant: pause generation to prevent in-flight requests from
861887
# reading partially-updated weights during the NCCL broadcast.
862888
if self.cfg.generator.inference_engine.weight_sync_backend == "delta":
863889
# Delta disk sync performs publish/fetch before pausing and
864890
# pauses internally only around the final reload.
865-
_broadcast_and_finish()
891+
await _broadcast_and_finish()
866892
else:
867893
await self._inference_engine_client.pause_generation()
868894
async with cleanup_preserving_primary(
869895
self._inference_engine_client.resume_generation, "resume_generation"
870896
):
871-
_broadcast_and_finish()
897+
await _broadcast_and_finish()
872898

873899
# Advance the policy version so prefix-cache salting isolates blocks from the previous weights
874900
# (see `GeneratorConfig.use_cache_salt`).

‎tests/backends/skyrl_train/distributed/test_worker_dispatch.py‎

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66
dispatch is built via ``__new__`` with mocked actor groups and client.
77
"""
88

9+
import asyncio
910
import time
1011
from types import SimpleNamespace
1112
from unittest.mock import AsyncMock, MagicMock
@@ -259,6 +260,102 @@ async def test_non_colocated_megatron_merge_lora_still_pauses(self):
259260
dispatch._inference_engine_client.pause_generation.assert_awaited_once()
260261
dispatch._inference_engine_client.resume_generation.assert_awaited_once()
261262

263+
@staticmethod
264+
def _nccl_dispatch(broadcast):
265+
from skyrl.backends.skyrl_train.workers.worker_dispatch import WorkerDispatch
266+
267+
dispatch = WorkerDispatch.__new__(WorkerDispatch)
268+
dispatch.colocate_all = False
269+
dispatch.cfg = _fft_dispatch_cfg()
270+
dispatch._inference_engine_client = AsyncMock()
271+
dispatch._inference_engine_client.increment_weight_version = MagicMock()
272+
dispatch._broadcast_to_inference_engines = MagicMock(side_effect=broadcast)
273+
dispatch._prepare_for_weight_sync = AsyncMock()
274+
dispatch._finish_weight_sync = MagicMock()
275+
dispatch.ensure_active_adapter = MagicMock()
276+
return dispatch
277+
278+
@pytest.mark.asyncio
279+
async def test_broadcast_does_not_block_event_loop(self):
280+
"""The broadcast blocks on ``ray.get``. A timeout armed by a coroutine sharing the
281+
loop (a fully-async generator's HTTP request) must not expire because of it."""
282+
dispatch = self._nccl_dispatch(lambda *args, **kwargs: time.sleep(0.5))
283+
284+
sync = asyncio.create_task(dispatch.save_weights_for_sampler())
285+
async with asyncio.timeout(0.2):
286+
await asyncio.sleep(0.05)
287+
assert not sync.done()
288+
await sync
289+
290+
dispatch._broadcast_to_inference_engines.assert_called_once()
291+
assert dispatch.last_weight_sync_seconds >= 0.5
292+
293+
@pytest.mark.asyncio
294+
async def test_cancellation_waits_for_the_transfer_before_resuming(self):
295+
"""A cancellation mid-broadcast must not resume generation onto partially transferred weights."""
296+
events = []
297+
298+
def broadcast(*args, **kwargs):
299+
time.sleep(0.3)
300+
events.append("broadcast_done")
301+
302+
dispatch = self._nccl_dispatch(broadcast)
303+
dispatch._inference_engine_client.resume_generation.side_effect = lambda: events.append("resumed")
304+
305+
sync = asyncio.create_task(dispatch.save_weights_for_sampler())
306+
await asyncio.sleep(0.05)
307+
sync.cancel()
308+
with pytest.raises(asyncio.CancelledError):
309+
await sync
310+
311+
assert events == ["broadcast_done", "resumed"]
312+
313+
@pytest.mark.asyncio
314+
async def test_cancellation_survives_a_failed_transfer(self):
315+
"""A transfer that fails after the cancellation arrived must not replace the cancellation."""
316+
317+
def broadcast(*args, **kwargs):
318+
time.sleep(0.3)
319+
raise RuntimeError("broadcast failed")
320+
321+
dispatch = self._nccl_dispatch(broadcast)
322+
323+
sync = asyncio.create_task(dispatch.save_weights_for_sampler())
324+
await asyncio.sleep(0.05)
325+
sync.cancel()
326+
with pytest.raises(asyncio.CancelledError) as raised:
327+
await sync
328+
329+
assert isinstance(raised.value.__cause__, RuntimeError)
330+
331+
@pytest.mark.asyncio
332+
@pytest.mark.parametrize("cancel, expected", [(False, RuntimeError), (True, asyncio.CancelledError)])
333+
async def test_offload_kv_wakes_kv_cache_before_resuming(self, cancel, expected):
334+
"""However the sync is interrupted, generation must not resume onto a sleeping KV cache."""
335+
events = []
336+
337+
def broadcast(*args, **kwargs):
338+
time.sleep(0.2)
339+
events.append("broadcast_done")
340+
if not cancel:
341+
raise RuntimeError("broadcast failed")
342+
343+
dispatch = self._nccl_dispatch(broadcast)
344+
dispatch.cfg.generator.inference_engine.offload_kv_for_weight_sync = True
345+
dispatch.cfg.trainer.fully_async = SimpleNamespace(enabled=True, clear_kv_cache_on_weight_sync=False)
346+
client = dispatch._inference_engine_client
347+
client.wake_for_weight_sync.side_effect = lambda tags: events.append(f"wake_{tags[0]}")
348+
client.resume_generation.side_effect = lambda: events.append("resumed")
349+
350+
sync = asyncio.create_task(dispatch.save_weights_for_sampler())
351+
await asyncio.sleep(0.05)
352+
if cancel:
353+
sync.cancel()
354+
with pytest.raises(expected):
355+
await sync
356+
357+
assert events == ["wake_weights", "broadcast_done", "wake_kv_cache", "resumed"]
358+
262359

263360
def _adapter_sync_dispatch(*, model_on_gpu: bool, optimizer_on_gpu: bool = False):
264361
"""Dispatch wired for the colocated adapter-only sync path

0 commit comments

Comments
 (0)