Repository navigation
[BugFix] Bind RayExecutorV2 TCPStore before publishing its port - #50969
AndreasKaratzas merged 9 commits into
Conversation
…ds it The driver probed the TCPStore port on its own node, but the store is bound by rank 0 on rank 0's node; on multi-node placements the probe is invalid and concurrent cold starts fail intermittently with EADDRINUSE. Probe on rank 0's actor right before worker init instead, riding the existing deferred initialization; the per-rank port window machinery becomes dead code and is removed. Co-authored-by: Claude <[email protected]> Signed-off-by: aoshen02 <[email protected]>
Co-authored-by: Claude <[email protected]> Signed-off-by: aoshen02 <[email protected]>
|
/ci run |
|
✅ CI is already running for this commit: https://buildkite.com/vllm/ci/builds/82860 |
Co-authored-by: Codex <[email protected]> Signed-off-by: aoshen02 <[email protected]> Signed-off-by: aoshen02 <[email protected]>
|
/ci run |
|
✅ Triggered Buildkite CI #82950 for commit |
Keep PR vllm-project#50969 scoped to RayExecutorV2, whose rank zero may be remote from the driver. Co-authored-by: Codex <[email protected]> Signed-off-by: aoshen02 <[email protected]> Signed-off-by: aoshen02 <[email protected]>
Create and retain the TCPStore server on rank zero before publishing its kernel-assigned port. Co-authored-by: OpenAI Codex <[email protected]> Signed-off-by: aoshen02 <[email protected]>
|
/ci run |
|
✅ Triggered Buildkite CI #83002 for commit |
vllm/v1/executor/ray_executor_v2.py: keep binder-side TCPStore port selection on the rank-0 actor; drop main's driver-side _select_tcpstore_port call (vllm-project#53666), which the bind-and-hold approach subsumes. main's sharded_rdt actor options (vllm-project#43375) auto-merged. tests/distributed/test_ray_v2_executor.py: keep test_dist_init_store_owns_port; drop main's _select_tcpstore_port tests including test_colocated_tcpstore_ports_differ, since that helper no longer exists. Co-authored-by: Claude Fable 5 <[email protected]> Signed-off-by: aoshen02 <[email protected]>
|
/ci run |
|
✅ Triggered Buildkite CI #85917 for commit |
|
/ci run |
|
✅ Triggered Buildkite CI #85994 for commit |
|
@aoshen02 I m going to merge this PR to resolve a ci failure. However can you open a follow up one? The old port-selection-specific tests can reasonably disappear, but test_colocated_tcpstore_ports_differ captures an externally meaningful invariant rather than an implementation detail. Could we retain that coverage using the new create_dist_init_method() path (e.g. create two rank-0 workers/stores and assert their held ephemeral ports differ)? The new test verifies ownership/reuse for a single store, but no longer directly protects the co-located-engine collision scenario that motivated the earlier regression test. |
Thanks, will do. |
…-project#50969) Signed-off-by: aoshen02 <[email protected]> Signed-off-by: aoshen02 <[email protected]> Co-authored-by: aoshen02 <[email protected]> Co-authored-by: Claude <[email protected]> Co-authored-by: OpenAI Codex <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]>
…-project#50969) Signed-off-by: aoshen02 <[email protected]> Signed-off-by: aoshen02 <[email protected]> Co-authored-by: aoshen02 <[email protected]> Co-authored-by: Claude <[email protected]> Co-authored-by: OpenAI Codex <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]>
…-project#50969) Signed-off-by: aoshen02 <[email protected]> Signed-off-by: aoshen02 <[email protected]> Co-authored-by: aoshen02 <[email protected]> Co-authored-by: Claude <[email protected]> Co-authored-by: OpenAI Codex <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> (cherry picked from commit 67e86d1)
…-project#50969) Signed-off-by: aoshen02 <[email protected]> Signed-off-by: aoshen02 <[email protected]> Co-authored-by: aoshen02 <[email protected]> Co-authored-by: Claude <[email protected]> Co-authored-by: OpenAI Codex <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]>
…969 is in 0.33.0 (#2729) # What does this PR do? Deletes the `VLLM_DP_MASTER_PORT` workaround that gave every vLLM engine its own TCPStore probe window. vLLM 0.33.0, which SkyRL pins since #2604, includes vllm-project/vllm#50969, so the race it worked around no longer exists. TLDR: SkyRL stops setting `VLLM_DP_MASTER_PORT`. The helpers, constants and tests that existed only for it are removed, following the removal checklist in `inference_servers/common.py`. No behavior change on vLLM 0.33.0. ## Background Added with the vLLM 0.26.0 bump (#1854). Until vllm#50969, `RayExecutorV2._select_tcpstore_port` chose the workers' TCPStore port by probing `[VLLM_DP_MASTER_PORT + 100, +32)`, closing the probe socket, and binding the TCPStore afterwards. With DP disabled `VLLM_DP_MASTER_PORT` defaults to 0, so every engine on a node probed ports 100-131. In containers with `net.ipv4.ip_unprivileged_port_start=0`, two co-located engines both chose port 100 and the second died with `EADDRINUSE`. SkyRL avoided this by setting `VLLM_DP_MASTER_PORT` per server actor so that each engine probed a disjoint window inside its own `SERVER_PORT_STRIDE` block. In vLLM 0.33.0, `_select_tcpstore_port` is gone. `RayExecutorV2.create_dist_init_method` creates the TCPStore with `port=0` and keeps the bound store, so there is no probe/bind gap and `VLLM_DP_MASTER_PORT` no longer affects the port. What still reads `VLLM_DP_MASTER_PORT` in 0.33.0, and why none of it matters to SkyRL: - `ParallelConfig`'s env fallback copies it into `data_parallel_master_port` when DP is off; that field is only consumed on DP > 1 paths. - With DP > 1, SkyRL passes `data_parallel_size` / `data_parallel_rank` / `data_parallel_address` / `data_parallel_rpc_port` as CLI args, and vLLM takes the DP master port from `get_open_ports_list()` without reading the env var. - `get_open_port()` skips `[VLLM_DP_MASTER_PORT, +10)` while the variable is set; the kernel's random ports come from the ephemeral range, far from SkyRL's 8000-range blocks. ## Solution Exactly the removal checklist at `common.py:174-181`: - `common.py`: delete the `DP_TCPSTORE_*` constants and their assert, the checklist, `compute_dp_master_port` and `dp_tcpstore_probe_window`. - `vllm_server_actor.py`: delete the `VLLM_DP_MASTER_PORT` assignment in `VLLMServerActor.__init__`, and `_seed_dp_master_port` with its call in `_build_and_serve_vllm_server` (the standalone `python -m` path). - `test_common.py`: delete `TestDPMasterPort` and the imports only it used. - `setup.py`, `opd/teacher_launch.py`: drop "plus the DP TCPStore probe range" from the port-window docstrings. `SERVER_PORT_STRIDE` stays at 100: the HTTP, NIXL and Mooncake ports and the OPD teacher offset all stride by it, and changing it would move every port SkyRL assigns. Alternatives considered: - Keep `VLLM_DP_MASTER_PORT` set as a safety net. Rejected: nothing on 0.33.0 uses it to pick the TCPStore port, and keeping it means keeping docs and tests for a mechanism that no longer exists. - Replace `TestDPMasterPort` with a test that the variable is not set. Rejected: it would assert the absence of a line. The behavior that matters, co-located engines starting, is covered by `gpu_ci/inference_servers/test_inference_server_group.py`. Both training backends share this inference-server code, so Megatron and FSDP behave the same. ## Test plan - `tests/backends/skyrl_train/inference_servers/test_common.py` keeps covering `find_and_reserve_port` and the rest of `common.py`. - `gpu_ci/inference_servers/test_inference_server_group.py` starts two TP=2 servers on one node through `RayExecutorV2`, the co-located case the workaround existed for. ## Validation | Check | Hardware | Command | Result | |---|---|---|---| | CPU unit tests | macOS | `pytest --noconftest tests/backends/skyrl_train/inference_servers/test_common.py` | 18 passed (23 before, minus the 5 deleted) | | Co-located engines | 4×L4 (`l4_ci`, Anyscale) | `pytest tests/backends/skyrl_train/gpu/gpu_ci/inference_servers/test_inference_server_group.py -v` | 4 passed in 101.6 s, no `EADDRINUSE` | | Lint | | `pre-commit run --all-files` | passed | The L4 node had `ip_unprivileged_port_start=1024`, so it could not reproduce the original container condition; the run confirms co-located engines start without `VLLM_DP_MASTER_PORT`, and the 0.33.0 source shows the probe is gone. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Low Risk** > Cleanup tied to a pinned vLLM release; port assignment behavior is unchanged aside from dropping obsolete env seeding. > > **Overview** > SkyRL **stops setting `VLLM_DP_MASTER_PORT`** now that vLLM 0.33.0 includes the upstream fix (vllm#50969) for TCPStore port probing on co-located Ray engines. > > The workaround is fully removed: `DP_TCPSTORE_*` constants, `compute_dp_master_port`, `dp_tcpstore_probe_window`, the env assignment in `VLLMServerActor.__init__`, and `_seed_dp_master_port` on the standalone `python -m` path. **`TestDPMasterPort`** and related imports are deleted from `test_common.py`. Docstrings in `setup.py` and `teacher_launch.py` no longer mention a DP TCPStore range inside each **`SERVER_PORT_STRIDE`** window—those windows still only reserve HTTP (and related) ports via **`find_and_reserve_port`**. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit da6ab9d. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> --------- Signed-off-by: Artur Niederfahrenhorst <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]>
Purpose
RayExecutorV2initialized its torch.distributed world group in two separate steps:This has two independent correctness problems:
get_open_port()closes its socket before returning. Another process can bind the port beforeinit_process_group()creates TCPStore. That is still a TOCTOU race.The second window includes a Ray RPC back to the driver and the dispatch of every worker initialization RPC. Concurrent engine cold starts make the collision much easier to hit.
Fix
Rank 0 now creates the actual server before publishing its address:
port=0delegates selection and binding to the kernel in one operation. The actor retains the Store, so the listening socket is never released between selection and publication.PyTorch documents
multi_tenant=Trueas sharing the underlyingTCPServerbetween TCPStore instances in the same process with the same host/port. Therefore the existingtcp://rendezvous remains unchanged: rank 0's later master TCPStore reuses the already-bound server, while nonzero ranks create clients.There is no fallback and no application-managed port window. The change is limited to
RayExecutorV2; legacy Ray V1 is intentionally unchanged.Why this is not a duplicate
get_open_port(); this PR removesget_open_port()from this call site.maincontaining that fix is merged into this branch.initialize_ray_cluster(..., require_gpu_on_driver=False), so it requires a network-reachable TCPStore._select_tcpstore_portso co-located non-DP replicas pick different ports, withtest_colocated_tcpstore_ports_differ. This PR removes_select_tcpstore_portentirely: the port is bound withport=0on the rank-0 host and held in_dist_init_store, so distinct ports per co-located replica are guaranteed by the kernel. The merge ofmaintherefore drops [Bugfix] Avoid TCPStore port collision for co-located non-DP Ray engines #53666's patch and its test deliberately — it is subsumed, not regressed.51275 in:bodyandRayExecutorV2 TCPStore portfound no other open PR covering this Ray V2 call site.Tests
Repository checks
Results:
1 passed, 21 deselected.use_libuvpairs (True/True,True/False,False/True, andFalse/False); all reused the held server successfully on Linux.The unit test asserts three observable invariants:
EADDRINUSEwhile the actor holds the Store;multi_tenant=Truecan reuse the same server and exchange a key.Deterministic Linux A/B
This script forces the old race, validates the final handoff with a real two-process Gloo all-reduce, and stress-checks 128 concurrent ephemeral servers:
Observed:
Real two-node Ray V2 integration
Tested with one H200 exposed on each of
h200-0andh200-1, forcing a TP=2 world group to span nodes. Ray reported two active nodes and two total GPUs. Both nodes used their routable interface:Driver command:
Observed on the two workers:
The model loaded on both nodes, NCCL/TP groups initialized, and generation completed. Model evaluation is not applicable because the change only affects startup rendezvous and does not alter model execution or outputs.
AI disclosure
AI assistance (Claude Code and OpenAI Codex) was used for investigation, implementation, testing, and drafting. The human submitter must review every changed line and can explain the change and tests end-to-end before merge.