Skip to content

[BugFix] Bind RayExecutorV2 TCPStore before publishing its port - #50969

Merged
AndreasKaratzas merged 9 commits into
vllm-project:mainfrom
aoshen02:fix/ray-tcpstore-binder-side-port
Aug 28, 2026
Merged

AndreasKaratzas merged 9 commits into
vllm-project:mainfrom
aoshen02:fix/ray-tcpstore-binder-side-port

Conversation

@aoshen02

@aoshen02 aoshen02 commented Aug 4, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

RayExecutorV2 initialized its torch.distributed world group in two separate steps:

driver (possibly another node)                 rank-0 Ray actor
-----------------------------                  ----------------
choose/probe a free integer port
close the probing socket
publish tcp://<rank-0-ip>:<port>  -----------> init_process_group(...)
                                                TCPStore binds <port>

This has two independent correctness problems:

  1. On a multi-node placement, the driver can probe a port on the wrong host. Port occupancy is host-local state.
  2. Even if the probe is moved to the rank-0 actor, get_open_port() closes its socket before returning. Another process can bind the port before init_process_group() creates TCPStore. That is still a TOCTOU race.

The second window includes a Ray RPC back to the driver and the dispatch of every worker initialization RPC. Concurrent engine cold starts make the collision much easier to hit.

Fix

Rank 0 now creates the actual server before publishing its address:

rank-0 Ray actor                               driver / other ranks
----------------                              --------------------
TCPStore(host, port=0, is_master=True,
         wait_for_workers=False,
         multi_tenant=True)
  kernel atomically selects + binds port
  actor retains Store ownership
return tcp://<actor-ip>:<store.port>  --------> initialize workers
                                                rank 0 reuses server
                                                remote ranks connect as clients

port=0 delegates selection and binding to the kernel in one operation. The actor retains the Store, so the listening socket is never released between selection and publication.

PyTorch documents multi_tenant=True as sharing the underlying TCPServer between TCPStore instances in the same process with the same host/port. Therefore the existing tcp:// rendezvous remains unchanged: rank 0's later master TCPStore reuses the already-bound server, while nonzero ranks create clients.

There is no fallback and no application-managed port window. The change is limited to RayExecutorV2; legacy Ray V1 is intentionally unchanged.

Why this is not a duplicate

Tests

Repository checks

.venv/bin/python -m pytest \
  tests/distributed/test_ray_v2_executor.py \
  -k dist_init_store_owns_port -q

pre-commit run --files \
  vllm/v1/executor/ray_executor_v2.py \
  tests/distributed/test_ray_v2_executor.py

pre-commit run mypy-3.12 \
  --files vllm/v1/executor/ray_executor_v2.py \
  --hook-stage manual

Results:

  • Targeted pytest in a CUDA vLLM image: 1 passed, 21 deselected.
  • All changed-file pre-commit hooks passed.
  • Python 3.10 and 3.12 mypy passed.
  • Same-process server reuse was also checked for all use_libuv pairs (True/True, True/False, False/True, and False/False); all reused the held server successfully on Linux.

The unit test asserts three observable invariants:

  1. the published port is nonzero;
  2. a competing socket gets Linux EADDRINUSE while the actor holds the Store;
  3. a second master TCPStore with multi_tenant=True can reuse the same server and exchange a key.

Deterministic Linux A/B

This script forces the old race, validates the final handoff with a real two-process Gloo all-reduce, and stress-checks 128 concurrent ephemeral servers:

.venv/bin/python - <<'PY'
import multiprocessing as mp
import socket
from concurrent.futures import ThreadPoolExecutor
from datetime import timedelta

import torch
import torch.distributed as dist

# Old path: probe -> close -> later bind. A contender deterministically wins.
probe = socket.socket()
probe.bind(("127.0.0.1", 0))
old_port = probe.getsockname()[1]
probe.close()

contender = socket.socket()
contender.bind(("127.0.0.1", old_port))
contender.listen()
try:
    dist.TCPStore(
        "127.0.0.1", old_port, 1, True,
        wait_for_workers=False, multi_tenant=True,
    )
except Exception as exc:
    assert "EADDRINUSE" in str(exc) or "address already in use" in str(exc).lower()
    print("old: EADDRINUSE (expected)")
else:
    raise AssertionError("old path unexpectedly rebound a stolen port")
finally:
    contender.close()

# Final path: rank 0 binds port=0 and holds the Store before publication.
ctx = mp.get_context("fork")
port_q, result_q, start = ctx.Queue(), ctx.Queue(), ctx.Event()

def rank0():
    held = dist.TCPStore(
        "127.0.0.1", 0, 2, True, timedelta(seconds=15),
        wait_for_workers=False, multi_tenant=True,
    )
    port_q.put(held.port)
    start.wait()
    dist.init_process_group(
        "gloo", init_method=f"tcp://127.0.0.1:{held.port}",
        rank=0, world_size=2, timeout=timedelta(seconds=15),
    )
    value = torch.tensor([1])
    dist.all_reduce(value)
    result_q.put((0, value.item()))
    dist.destroy_process_group()

def rank1(port):
    start.wait()
    dist.init_process_group(
        "gloo", init_method=f"tcp://127.0.0.1:{port}",
        rank=1, world_size=2, timeout=timedelta(seconds=15),
    )
    value = torch.tensor([1])
    dist.all_reduce(value)
    result_q.put((1, value.item()))
    dist.destroy_process_group()

p0 = ctx.Process(target=rank0)
p0.start()
new_port = port_q.get(timeout=10)
with socket.socket() as sock:
    try:
        sock.bind(("127.0.0.1", new_port))
    except OSError as exc:
        assert exc.errno == 98
    else:
        raise AssertionError("published port was not held")

p1 = ctx.Process(target=rank1, args=(new_port,))
p1.start()
start.set()
p0.join(20)
p1.join(20)
assert p0.exitcode == p1.exitcode == 0
results = sorted([result_q.get(timeout=2), result_q.get(timeout=2)])
assert results == [(0, 2), (1, 2)]
print("new:", results)

with ThreadPoolExecutor(max_workers=32) as pool:
    stores = list(pool.map(
        lambda _: dist.TCPStore(
            "127.0.0.1", 0, 1, True,
            wait_for_workers=False, multi_tenant=True,
        ),
        range(128),
    ))
ports = [store.port for store in stores]
assert len(set(ports)) == 128
reused = [
    dist.TCPStore(
        "127.0.0.1", port, 1, True,
        wait_for_workers=False, multi_tenant=True,
    )
    for port in ports
]
assert [store.port for store in reused] == ports
print("stress: 128 unique ports, 128 reused servers")
PY

Observed:

old: EADDRINUSE (expected)
new: [(0, 2), (1, 2)]
stress: 128 unique ports, 128 reused servers

Real two-node Ray V2 integration

Tested with one H200 exposed on each of h200-0 and h200-1, forcing a TP=2 world group to span nodes. Ray reported two active nodes and two total GPUs. Both nodes used their routable interface:

export GLOO_SOCKET_IFNAME=ens7
export NCCL_SOCKET_IFNAME=ens7

# h200-0
ray start --head --node-ip-address=172.27.48.114 --port=26379 \
  --disable-usage-stats --num-gpus=1 --block

# h200-1
ray start --address=172.27.48.114:26379 \
  --node-ip-address=172.27.55.233 \
  --disable-usage-stats --num-gpus=1 --block

Driver command:

VLLM_USE_RAY_V2_EXECUTOR_BACKEND=1 \
RAY_ADDRESS=172.27.48.114:26379 \
VLLM_ALLOW_INSECURE_SERIALIZATION=1 \
.venv/bin/python - <<'PY'
from vllm import LLM, SamplingParams

llm = LLM(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    tensor_parallel_size=2,
    distributed_executor_backend="ray",
    enforce_eager=True,
    max_model_len=256,
    gpu_memory_utilization=0.05,
)
output = llm.generate(["Hello"], SamplingParams(max_tokens=1))
print(repr(output[0].outputs[0].text))
PY

Observed on the two workers:

rank 0: world_size=2 ... distributed_init_method=tcp://172.27.48.114:37991
rank 1 (172.27.55.233): world_size=2 ... distributed_init_method=tcp://172.27.48.114:37991
RESULT ','

The model loaded on both nodes, NCCL/TP groups initialized, and generation completed. Model evaluation is not applicable because the change only affects startup rendezvous and does not alter model execution or outputs.

AI disclosure

AI assistance (Claude Code and OpenAI Codex) was used for investigation, implementation, testing, and drafting. The human submitter must review every changed line and can explain the change and tests end-to-end before merge.

…ds it

The driver probed the TCPStore port on its own node, but the store is
bound by rank 0 on rank 0's node; on multi-node placements the probe is
invalid and concurrent cold starts fail intermittently with EADDRINUSE.
Probe on rank 0's actor right before worker init instead, riding the
existing deferred initialization; the per-rank port window machinery
becomes dead code and is removed.

Co-authored-by: Claude <[email protected]>
Signed-off-by: aoshen02 <[email protected]>
@mergify mergify Bot added the bug Something isn't working label Aug 4, 2026

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@aoshen02 aoshen02 added the ready ONLY add when PR is ready to merge/full CI is needed label Aug 7, 2026
@aoshen02

aoshen02 commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown

Co-authored-by: Codex <[email protected]>

Signed-off-by: aoshen02 <[email protected]>

Signed-off-by: aoshen02 <[email protected]>
@aoshen02 aoshen02 changed the title [BugFix] RayExecutorV2: choose the TCPStore port on the node that binds it [BugFix] Choose Ray TCPStore endpoint on rank zero (v1/v2) Aug 7, 2026
@aoshen02

aoshen02 commented Aug 8, 2026

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown

✅ Triggered Buildkite CI #82950 for commit 94028f4cc6f2.

Keep PR vllm-project#50969 scoped to RayExecutorV2, whose rank zero may be remote from the driver.

Co-authored-by: Codex <[email protected]>

Signed-off-by: aoshen02 <[email protected]>

Signed-off-by: aoshen02 <[email protected]>
@aoshen02 aoshen02 changed the title [BugFix] Choose Ray TCPStore endpoint on rank zero (v1/v2) [BugFix] Choose RayExecutorV2 TCPStore port on rank zero Aug 8, 2026
Create and retain the TCPStore server on rank zero before publishing its kernel-assigned port.

Co-authored-by: OpenAI Codex <[email protected]>
Signed-off-by: aoshen02 <[email protected]>
@aoshen02 aoshen02 changed the title [BugFix] Choose RayExecutorV2 TCPStore port on rank zero [BugFix] Bind RayExecutorV2 TCPStore before publishing its port Aug 8, 2026
@aoshen02

aoshen02 commented Aug 8, 2026

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown

✅ Triggered Buildkite CI #83002 for commit fc888a0442e3.

vllm/v1/executor/ray_executor_v2.py: keep binder-side TCPStore port selection on the rank-0 actor; drop main's driver-side _select_tcpstore_port call (vllm-project#53666), which the bind-and-hold approach subsumes. main's sharded_rdt actor options (vllm-project#43375) auto-merged.
tests/distributed/test_ray_v2_executor.py: keep test_dist_init_store_owns_port; drop main's _select_tcpstore_port tests including test_colocated_tcpstore_ports_differ, since that helper no longer exists.

Co-authored-by: Claude Fable 5 <[email protected]>
Signed-off-by: aoshen02 <[email protected]>
@aoshen02

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #85917 for commit e7da3bed0f5f.

@AndreasKaratzas

Copy link
Copy Markdown
Member

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #85994 for commit a092ffd95ce0.

@AndreasKaratzas

Copy link
Copy Markdown
Member

@aoshen02 I m going to merge this PR to resolve a ci failure. However can you open a follow up one? The old port-selection-specific tests can reasonably disappear, but test_colocated_tcpstore_ports_differ captures an externally meaningful invariant rather than an implementation detail. Could we retain that coverage using the new create_dist_init_method() path (e.g. create two rank-0 workers/stores and assert their held ephemeral ports differ)? The new test verifies ownership/reuse for a single store, but no longer directly protects the co-located-engine collision scenario that motivated the earlier regression test.

@AndreasKaratzas
AndreasKaratzas merged commit 67e86d1 into vllm-project:main Aug 28, 2026
89 checks passed
@github-project-automation github-project-automation Bot moved this from Backlog to Done in Ray Aug 28, 2026
@aoshen02
aoshen02 deleted the fix/ray-tcpstore-binder-side-port branch August 29, 2026 03:47
@aoshen02

Copy link
Copy Markdown
Contributor Author

@aoshen02 I m going to merge this PR to resolve a ci failure. However can you open a follow up one? The old port-selection-specific tests can reasonably disappear, but test_colocated_tcpstore_ports_differ captures an externally meaningful invariant rather than an implementation detail. Could we retain that coverage using the new create_dist_init_method() path (e.g. create two rank-0 workers/stores and assert their held ephemeral ports differ)? The new test verifies ownership/reuse for a single store, but no longer directly protects the co-located-engine collision scenario that motivated the earlier regression test.

Thanks, will do.

am-cohere pushed a commit to am-cohere/vllm that referenced this pull request Sep 1, 2026
…-project#50969)

Signed-off-by: aoshen02 <[email protected]>
Signed-off-by: aoshen02 <[email protected]>
Co-authored-by: aoshen02 <[email protected]>
Co-authored-by: Claude <[email protected]>
Co-authored-by: OpenAI Codex <[email protected]>
Co-authored-by: Andreas Karatzas <[email protected]>
mylibrar pushed a commit to tanyuqian/vllm that referenced this pull request Sep 3, 2026
…-project#50969)

Signed-off-by: aoshen02 <[email protected]>
Signed-off-by: aoshen02 <[email protected]>
Co-authored-by: aoshen02 <[email protected]>
Co-authored-by: Claude <[email protected]>
Co-authored-by: OpenAI Codex <[email protected]>
Co-authored-by: Andreas Karatzas <[email protected]>
D-G-Dimitrov pushed a commit to D-G-Dimitrov/vllm that referenced this pull request Sep 4, 2026
…-project#50969)

Signed-off-by: aoshen02 <[email protected]>
Signed-off-by: aoshen02 <[email protected]>
Co-authored-by: aoshen02 <[email protected]>
Co-authored-by: Claude <[email protected]>
Co-authored-by: OpenAI Codex <[email protected]>
Co-authored-by: Andreas Karatzas <[email protected]>
(cherry picked from commit 67e86d1)
michael-gpu pushed a commit to practical-inference/vllm-ampere-extended that referenced this pull request Oct 20, 2026
…-project#50969)

Signed-off-by: aoshen02 <[email protected]>
Signed-off-by: aoshen02 <[email protected]>
Co-authored-by: aoshen02 <[email protected]>
Co-authored-by: Claude <[email protected]>
Co-authored-by: OpenAI Codex <[email protected]>
Co-authored-by: Andreas Karatzas <[email protected]>
erictang000 pushed a commit to NovaSky-AI/SkyRL that referenced this pull request Nov 12, 2026
…969 is in 0.33.0 (#2729)

# What does this PR do?

Deletes the `VLLM_DP_MASTER_PORT` workaround that gave every vLLM engine
its own TCPStore probe window. vLLM 0.33.0, which SkyRL pins since
#2604, includes vllm-project/vllm#50969, so the race it worked around no
longer exists.

TLDR: SkyRL stops setting `VLLM_DP_MASTER_PORT`. The helpers, constants
and tests that existed only for it are removed, following the removal
checklist in `inference_servers/common.py`. No behavior change on vLLM
0.33.0.

## Background

Added with the vLLM 0.26.0 bump (#1854). Until vllm#50969,
`RayExecutorV2._select_tcpstore_port` chose the workers' TCPStore port
by probing `[VLLM_DP_MASTER_PORT + 100, +32)`, closing the probe socket,
and binding the TCPStore afterwards. With DP disabled
`VLLM_DP_MASTER_PORT` defaults to 0, so every engine on a node probed
ports 100-131. In containers with
`net.ipv4.ip_unprivileged_port_start=0`, two co-located engines both
chose port 100 and the second died with `EADDRINUSE`. SkyRL avoided this
by setting `VLLM_DP_MASTER_PORT` per server actor so that each engine
probed a disjoint window inside its own `SERVER_PORT_STRIDE` block.

In vLLM 0.33.0, `_select_tcpstore_port` is gone.
`RayExecutorV2.create_dist_init_method` creates the TCPStore with
`port=0` and keeps the bound store, so there is no probe/bind gap and
`VLLM_DP_MASTER_PORT` no longer affects the port.

What still reads `VLLM_DP_MASTER_PORT` in 0.33.0, and why none of it
matters to SkyRL:

- `ParallelConfig`'s env fallback copies it into
`data_parallel_master_port` when DP is off; that field is only consumed
on DP > 1 paths.
- With DP > 1, SkyRL passes `data_parallel_size` / `data_parallel_rank`
/ `data_parallel_address` / `data_parallel_rpc_port` as CLI args, and
vLLM takes the DP master port from `get_open_ports_list()` without
reading the env var.
- `get_open_port()` skips `[VLLM_DP_MASTER_PORT, +10)` while the
variable is set; the kernel's random ports come from the ephemeral
range, far from SkyRL's 8000-range blocks.

## Solution

Exactly the removal checklist at `common.py:174-181`:

- `common.py`: delete the `DP_TCPSTORE_*` constants and their assert,
the checklist, `compute_dp_master_port` and `dp_tcpstore_probe_window`.
- `vllm_server_actor.py`: delete the `VLLM_DP_MASTER_PORT` assignment in
`VLLMServerActor.__init__`, and `_seed_dp_master_port` with its call in
`_build_and_serve_vllm_server` (the standalone `python -m` path).
- `test_common.py`: delete `TestDPMasterPort` and the imports only it
used.
- `setup.py`, `opd/teacher_launch.py`: drop "plus the DP TCPStore probe
range" from the port-window docstrings.

`SERVER_PORT_STRIDE` stays at 100: the HTTP, NIXL and Mooncake ports and
the OPD teacher offset all stride by it, and changing it would move
every port SkyRL assigns.

Alternatives considered:

- Keep `VLLM_DP_MASTER_PORT` set as a safety net. Rejected: nothing on
0.33.0 uses it to pick the TCPStore port, and keeping it means keeping
docs and tests for a mechanism that no longer exists.
- Replace `TestDPMasterPort` with a test that the variable is not set.
Rejected: it would assert the absence of a line. The behavior that
matters, co-located engines starting, is covered by
`gpu_ci/inference_servers/test_inference_server_group.py`.

Both training backends share this inference-server code, so Megatron and
FSDP behave the same.

## Test plan

- `tests/backends/skyrl_train/inference_servers/test_common.py` keeps
covering `find_and_reserve_port` and the rest of `common.py`.
- `gpu_ci/inference_servers/test_inference_server_group.py` starts two
TP=2 servers on one node through `RayExecutorV2`, the co-located case
the workaround existed for.

## Validation

| Check | Hardware | Command | Result |
|---|---|---|---|
| CPU unit tests | macOS | `pytest --noconftest
tests/backends/skyrl_train/inference_servers/test_common.py` | 18 passed
(23 before, minus the 5 deleted) |
| Co-located engines | 4×L4 (`l4_ci`, Anyscale) | `pytest
tests/backends/skyrl_train/gpu/gpu_ci/inference_servers/test_inference_server_group.py
-v` | 4 passed in 101.6 s, no `EADDRINUSE` |
| Lint | | `pre-commit run --all-files` | passed |

The L4 node had `ip_unprivileged_port_start=1024`, so it could not
reproduce the original container condition; the run confirms co-located
engines start without `VLLM_DP_MASTER_PORT`, and the 0.33.0 source shows
the probe is gone.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Low Risk**
> Cleanup tied to a pinned vLLM release; port assignment behavior is
unchanged aside from dropping obsolete env seeding.
> 
> **Overview**
> SkyRL **stops setting `VLLM_DP_MASTER_PORT`** now that vLLM 0.33.0
includes the upstream fix (vllm#50969) for TCPStore port probing on
co-located Ray engines.
> 
> The workaround is fully removed: `DP_TCPSTORE_*` constants,
`compute_dp_master_port`, `dp_tcpstore_probe_window`, the env assignment
in `VLLMServerActor.__init__`, and `_seed_dp_master_port` on the
standalone `python -m` path. **`TestDPMasterPort`** and related imports
are deleted from `test_common.py`. Docstrings in `setup.py` and
`teacher_launch.py` no longer mention a DP TCPStore range inside each
**`SERVER_PORT_STRIDE`** window—those windows still only reserve HTTP
(and related) ports via **`find_and_reserve_port`**.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
da6ab9d. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

---------

Signed-off-by: Artur Niederfahrenhorst <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working ray anything related with ray ready ONLY add when PR is ready to merge/full CI is needed

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants