Report when finalize waits on live CUDA graphs - #2570

Description

ncclCommFinalize waits in commDestroySync() until every CUDA graph that captured an operation on the communicator is destroyed. If such a graph stays alive, finalize blocks and prints nothing. #2568 hit this through PyTorch's destroy_process_group, with graphs kept by torch.compile(mode="reduce-overhead").

With this change, each waiting rank prints one WARN after 5 s:

NCCL WARN comm 0x... rank 0: finalize cannot complete until the CUDA graphs that captured operations on this communicator are destroyed (captured operations still referenced: 1)

The wait itself does not change: finalize still returns once the graphs are destroyed.

Related Issues

Implements the logging part of #2568. It does not add periodic messages, or the optional timeout or env var.

Changes & Impact

src/init.cc, commDestroySync():

• For the first 5 s the loop polls the callback queue without blocking and yields. Graph destructors run asynchronously, so a short wait is normal and prints nothing.
• If captured operations are still referenced after 5 s, it prints the WARN once and returns to the existing blocking wait.
• The count is comm->localPersistentRefs, the number of captured plans, not the number of graphs.
• When no captured operation is referenced, the loop body never runs.
• The finalizing thread spins for up to 5 s in that wait, where it used to sleep. This only happens while a graph still references the communicator.
• The message is a WARN rather than an ATTN because NCCL_DEBUG=WARN does not print ATTN.
• An application that calls ncclCommDestroy without ncclCommFinalize reaches this wait under NOWARN(..., NCCL_DESTROY), so there the message is logged as INFO under the DESTROY subsystem.
• Based on dev. master has the same loop, but this diff does not apply there directly because the surrounding lines differ.

Validation

dev b00866b vs this change. One aarch64 host with a GH200 GPU, two processes on that GPU (NCCL_MULTI_RANK_GPU_ENABLE=1), bootstrap through NCCL_COMM_ID, NCCL_DEBUG=WARN. The probe captures an ncclAllReduce in a CUDA graph, replays it, then calls ncclCommFinalize. One run per row.

The "never captured" row for this PR ran on a revision with the same logic and different message text. The other two rows for this PR ran on the final commit.

Not exercised: a graph destroyed during the first 5 s of the wait, the ncclCommDestroy-only path, and nonblocking communicators.

Performance Impact

None on collectives. Only the finalize path changes.