Repository navigation
Commit ab16cf4
[skyrl-train][bugfix] Wake the KV cache before resuming after an interrupted sync
On the fully-async offload-KV path, a broadcast that failed or was cancelled
skipped wake_for_weight_sync(kv_cache), and the resume cleanup then scheduled
the frozen requests onto a sleeping KV cache. Bracket the KV wake with
cleanup_preserving_primary inside the resume bracket, as the synchronous-trainer
path already does for its wake_up(kv_cache).
Also wait on the transfer with asyncio.wait rather than shield: a transfer that
failed after a cancellation arrived escaped the wait loop and replaced the
cancellation. The cancellation now propagates, chained to the transfer's error.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Dominic Yurk <[email protected]>1 parent eb3d520 commit ab16cf4
2 files changed
Lines changed: 59 additions & 9 deletions
File tree
- skyrl/backends/skyrl_train/workers
- tests/backends/skyrl_train/distributed
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
809 | 809 | | |
810 | 810 | | |
811 | 811 | | |
812 | | - | |
813 | | - | |
| 812 | + | |
| 813 | + | |
| 814 | + | |
814 | 815 | | |
815 | 816 | | |
816 | 817 | | |
| |||
823 | 824 | | |
824 | 825 | | |
825 | 826 | | |
826 | | - | |
| 827 | + | |
827 | 828 | | |
828 | 829 | | |
829 | 830 | | |
830 | | - | |
831 | | - | |
832 | | - | |
| 831 | + | |
833 | 832 | | |
834 | 833 | | |
835 | 834 | | |
| |||
865 | 864 | | |
866 | 865 | | |
867 | 866 | | |
868 | | - | |
869 | | - | |
870 | | - | |
| 867 | + | |
| 868 | + | |
| 869 | + | |
| 870 | + | |
| 871 | + | |
| 872 | + | |
| 873 | + | |
| 874 | + | |
871 | 875 | | |
872 | 876 | | |
873 | 877 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
310 | 310 | | |
311 | 311 | | |
312 | 312 | | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
313 | 359 | | |
314 | 360 | | |
315 | 361 | | |
| |||
0 commit comments