[RDMA] Initialize flush requests before posting or returning - #1106

Description

Fixes #1096.

pluginIflush returns a pooled request even when there is nothing to flush. Previously, uccl_flush left its context unset for all flushes and returned before initializing its type and completion state for no-op flushes. Polling could therefore dereference an invalid flow or interpret stale request state. A reused completion flag could also report a newly posted flush complete before its RDMA READ finished.

Initialize the request as a completed ReqFlush with the current flow before the early return, then clear the completion flag before posting a CUDA flush. HIP retains its immediate-completion behavior.

Add flush_test, automatically picked up by the existing Makefiles, and document its commands. The test uses a real local RDMA flush QP without a remote host or data engines. It checks empty/all-zero flushes, fresh and reused requests, and alternating posted/no-op flushes against registered host or GPU memory.

Type of Change

• Bug fix
• New feature
• Documentation update

How Has This Been Tested?

Validated on Linux x86-64 with one NVIDIA B200, an active mlx5 InfiniBand port, GCC 13.3, and CUDA 12.8. Baseline is upstream 54e216ee; the same regression test was compiled against baseline and the fix.

Each posted-flush test checks that the request is initially incomplete, polls its real CQ to completion with a deadline, then reuses it for a no-op. The no-op-only run covers n=0 and zero-sized buffers with null data/MR arrays.

Repeated all three fixed tests with AddressSanitizer and UndefinedBehaviorSanitizer: all passed without sanitizer diagnostics. Built all transport objects and the test with -fsanitize=address,undefined -fno-omit-frame-pointer; used ASAN_OPTIONS=detect_leaks=0:protect_shadow_gap=0 and UBSAN_OPTIONS=halt_on_error=1. Disabling shadow-gap protection was required for CUDA initialization on this host; leak checking was not performed.

Built flush_test and libnccl-net-uccl.so for both baseline and fix with the repository's pinned NCCL headers (2ea4ee94). Confirmed the fixed plugin exports ncclNetPlugin_v8. Used the existing native Makefile targets with CUDA_HOME=/usr/local/cuda-12.8; this host lacks the unused gflags/gtest libraries, so the test used LIBS="-lz -lelf -libverbs -L/usr/local/cuda-12.8/lib64 -lcudart -lpthread" and the plugin used LIBS_SHARED="-lz -lelf -libverbs".

export UCCL_IB_HCA=mlx5_0:1 UCCL_NUM_ENGINES=1
./collective/rdma/flush_test --noop
./collective/rdma/flush_test
./collective/rdma/flush_test --gpu

Changed C++ files pass clang-format 14.0.6 --dry-run --Werror; git diff --check passes.

Validation is limited to the local flush request lifecycle on NVIDIA/InfiniBand. ROCm, RoCE, cross-node NCCL integration, and throughput benchmarks were not run; this PR makes no performance claim.

• Unit tests
• Integration tests
• Manual testing

Checklist

• I have run format.sh to follow the style guidelines. (Ran clang-format 14 directly on the changed C++ files.)
• I have run build.sh to verify compilation. (Used native Makefile builds described above.)
• I have removed redundant variables and comments.
• I have updated the documentation.
• I have added tests.