[Proposal] libfabric-CXI backend for UCCL-EP on HPE Slingshot

[Proposal]: libfabric-CXI backend for UCCL-EP on HPE Slingshot

Hi UCCL team,

We're from the Apertus / Swiss AI training effort, working on large-scale open MoE training on Alps (using HPE's Slingshot-11 Cassini NIC). For Apertus 2 we're evaluating replacements for the default NCCL all-gather-style MoE dispatcher, and UCCL-EP looks like a strong fit because Slingshot does not provide IBGDA and we can only use some CPU-proxy design for all2all communication.

On Alps the network interface uses libfabric's cxi provider, not typical ibverbs. Before we start prototyping, we'd like to ask:

Does UCCL-EP currently have a libfabric transport path? I saw EFA, Broadcom, and CX7 mentioned and supported. Is EFA implemented via libfabric (efa provider), and if so, is the transport layer abstracted enough that adding a cxi provider would be mostly provider substitution?
Is any CXI / Slingshot work already underway or planned? If so, we'd be glad to test on our platform for debugging and help with performance benchmarking.
If not, would a libfabric/CXI backend be welcomed? Our intent is to keep the UCCL-EP API unchanged and map proxy operations to libfabric RMA, starting with low-latency mode. This might be useful because, as far as we know, many EU data centers actually use Slingshot because they were initially built for HPC, not exactly LLM training. If we can demonstrate performance gains, it would benefit many users.

What we'd contribute

Assuming the answer to (3) is yes, we'd plan to upstream:

A libfabric/CXI transport backend (low-latency mode first, then normal/high-throughput).
Validation on real Apertus MoE training on Alps Slingshot
Benchmarks vs. all-gather performances.

What we plan to verify first currently

The main open questions on the CXI side are CUDA HMEM registration, MR-mode handling (FI_MR_ENDPOINT / FI_MR_PROV_KEY), the write-data / immediate-data path (FI_CXI_ENABLE_WRITEDATA, cq_data_size, FI_REMOTE_CQ_DATA), and ordering between payload writes and control notifications on CXI. We plan to work through these in standalone libfabric tests before touching UCCL — starting with fi_info -p cxi -v on the actual training container, then host-memory RMA, CUDA-memory RMA, a write-data test, and a payload+control ordering stress test. Happy to share the full plan if useful.

From my understanding of the paper, UCCL-EP uses immediate-data control messages to emulate ordering semantics for EP modes. Correct me if wrong. This means the write-with-immediate number perf matters a lot which we have to test. If performance is low, we need to think of design changes.

Why we think this can be useful for both sides

For UCCL: real Slingshot/CXI deployment and validation on an open large-scale MoE workload.
For Apertus: a reusable open-source EP layer instead of an internal one-off.
For the broader community: other EU data centers that want to train or serve large MoE models on Slingshot NICs.

Finally, thanks for building UCCL-EP, which benefits many people not use ib nics

@FFGGSSJJ

At people related to this issue: @FFGGSSJJ @andresnowak @youliang-zhu @haodong98

@PanJason

@PanJason Thanks for reaching out! We are definitely interested.

We don’t currently have a generic libfabric backend in UCCL-EP, and there isn't ongoing CXI / Slingshot work on our side. That said, we would definitely welcome contributions toward a CXI / Slingshot backend and would be happy to work with you on it!

Could I confirm with you whether CXI supports AMO writes, and whether RDMA writes and AMO writes are ordered with respect to each other in send order?

If both are supported, UCCL-EP may be able to bypass the write-with-immediate path. We mainly use write-with-immediate when the network does not provide such guarantees. We have not observed write-with-immediate to be the bottleneck in practice.

cc @YangZhou1997

@MaoZiming

@MaoZiming Thanks for your feedback. On paper, CXI does support AMO, but whether our provider (HPE Slingshot 11 Cassini) supports it needs some experimentation. I am testing them now and will come back to you tmr.

@MaoZiming

@MaoZiming I was doing a bit of investigation over the weekend. Here is what I found for your question:

Support for AMO writes

Could I confirm with you whether CXI supports AMO writes

TL;DR: CXI with the implementation on our slingshot NIC supports AMO writes. The following operations are supported for both host memory (local DRAM to remote DRAM) and Device memory (local HBM to remote HBM)

libfabric op	IB verbs equivalent	Host memory	Device memory
`fi_write`	`IBV_WR_RDMA_WRITE`	Works	Works
`fi_read`	`IBV_WR_RDMA_READ`	Works	Works
`fi_atomic(..., FI_ATOMIC_WRITE)`	atomic-style remote write / set	Works	Works
`fi_atomic(..., FI_SUM)`	atomic add / fetch-add without fetch result	Works	Works
`fi_fetch_atomic(..., FI_ATOMIC_READ)`	atomic read / fetch	Works	Works
`fi_fetch_atomic(..., FI_SUM)`	fetch-and-add	Works	Works
`fi_writedata`	`IBV_WR_RDMA_WRITE_WITH_IMM`	Works	Works

Support for ordering

and whether RDMA writes and AMO writes are ordered with respect to each other in send order?

TL;DR from my results, if AMO is using fi_atomic(..., FI_ATOMIC_WRITE) then no (at least not between DRAM, works on my HBM test but I am not sure if that usage pattern is what UCCL needs).[This is wrong because of a bug in my benchmark code, which changes the src mem before the previous fi_atomic is acked from cq on the sender side] if AMO is using fi_atomic(..., FI_ATOMIC_WRITE) or fi_atomic(..., FI_SUM) then yes, at least on the same endpoint / TX context / transmit queue. This result holds empirically on our slingshot nic, which is not guaranteed by the doc

Relevant interfaces

The two basic interfaces are:

fi_write(...);                    // one-sided RMA write
fi_atomic(..., FI_ATOMIC_WRITE);  // atomic write / set
fi_atomic(..., FI_SUM);           // atomic add

The question is whether this sender-side order:

fi_write(payload)
fi_atomic(control)

implies this receiver-side visibility:

observe control
=> payload is already visible

Host-Memory Workflow

Sender:

fi_write payload[N] to remote host buffer
fi_atomic control[N] to remote host control word

Receiver:

poll/read local host control word
when control says N is ready, immediately check payload[N]

Result:

Control operation	Ordering mode	Result
`FI_ATOMIC_WRITE`	plain submission order	All passed
`FI_SUM` counter	plain submission order	All passed
`FI_ATOMIC_WRITE`	wait for matched payload completion before control (CQ completion)	All passed.
`FI_ATOMIC_WRITE`	payload `FI_DELIVERY_COMPLETE` before control	All passed.

Device-memory payload + DRAM control word

Sender:

fi_write payload[N] to remote GPU HBM
fi_atomic control[N] to remote host DRAM control word

Receiver:

polls host DRAM control_word
when control_word >= seq:
      launches CUDA kernel
      CUDA kernel reads remote_payload[seq] in local GPU HBM
      cudaMemcpy check result back to host

Result:

Payload target	Control target	Control operation	Ordering mode	Result
GPU HBM	host DRAM	`FI_ATOMIC_WRITE`	plain submission order	Passed
GPU HBM	host DRAM	`FI_SUM`	plain submission order	Passed
GPU HBM	host DRAM	`FI_ATOMIC_WRITE`	matched payload completion before control (CQ completion)	Passed.
GPU HBM	host DRAM	`FI_ATOMIC_WRITE`	payload `FI_DELIVERY_COMPLETE` before control	Passed.

Device-memory payload + control word

Sender:

fi_write payload[N] to remote GPU HBM
fi_atomic control[N] to remote GPU HBM control word

Receiver variants:

# Host-observed variant
polls GPU HBM control_word via cudaMemcpy
when control_word is observed:
      copies/checks GPU HBM payload from host side

# GPU-observed variant
launches CUDA kernel before sender starts
CUDA kernel spins on GPU HBM control_word
when control_word is observed:
      same CUDA kernel immediately reads/checks GPU HBM payload
      cudaMemcpy check result back to host

Result:

Payload target	Control target	Control operation	Ordering mode	Receiver observation	Result
GPU HBM	GPU HBM	`fi_write`	plain submission order	host observes control via CUDA copy	Passed
GPU HBM	GPU HBM	`fi_write`	plain submission order	CUDA kernel observes control	Passed
GPU HBM	GPU HBM	`FI_ATOMIC_WRITE`control	plain submission order	host observes control via CUDA copy	Passed
GPU HBM	GPU HBM	`FI_ATOMIC_WRITE`control	plain submission order	CUDA kernel observes control	Passed

Note: this device-memory control test was mainly control-style, not the full per-sequence UCCL queue protocol. FI_SUM targeting GPU HBM was tested as an atomic capability, but not yet as the full payload-write + GPU-HBM-control ordering pattern.

Conclusion

Empirically yes on Alps CXI backed by slingshot. We did not see any write/AMO reordering with any of the configuration combination shown in appendix in any of the 5 runs each of which contains 1m iterations. When the AMO/control was observed, the payload write was visible on host or GPU.

Nevertheless, I would not make it as an unqualified libfabric generic assumption.

It would also be good if you can share with me some pseudo code about how UCCL can possibly utilize this write + AMO pattern so I can do some more accurate tests. I can check the code by myself as well.

Appendix:

Different Configurations I have tested for the empirical results:

Memory:

Host DRAM only
GPU HBM payload + Host DRAM control
GPU HBM only

Receiver Validation:

Host CPU checks host DRAM payload/control
Host CPU observes host DRAM control, then launches CUDA kernel to check GPU HBM payload
Host CPU observes GPU HBM control via CUDA copy, then checks GPU HBM payload
CUDA kernel spins on GPU HBM control, then checks GPU HBM payload in-kernel

MR Layout:

Split MR: payload and control registered separately
Shared MR: payload and control in one registered region

Operation Pattern:

1 fi_write(payload) -> fi_atomic(FI_ATOMIC_WRITE control)
1 fi_write(payload) -> fi_atomic(FI_SUM control)
N fi_write(payload) -> fi_write(control)
N fi_write(payload) -> fi_atomic(FI_ATOMIC_WRITE control)

API Spelling:

fi_write + fi_atomic
fi_writemsg + fi_atomicmsg
fi_write + fi_writedata

HMEM / Registration:

No HMEM flag for host memory
2.FI_HMEM_SYSTEM for host memory
FI_HMEM_CUDA for GPU HBM

Ordering Request:

No requested ordering, tx_order=0
FI_ORDER_RMA_WAW (fi_write only)
FI_ORDER_WAW (more general for fi_write anf fi_atomic)

All the combo passed the tests on the same endpoint / TX context / transmit queue.

PS: By the doc, both FI_ORDER_RMA_WAW and FI_ORDER_WAW actually just preserves the order of operations over the same mem addresses not operations across different mem addresses but both work empirically, even without negotiation ini initialization.

Concurrency / Stress Shape:

Single pair, 2 nodes, 1 rank per node
Four concurrent cross-node GPU pairs
Depth 1, 16, 256
Payload sizes: 8 B, 64 B, 4 KiB, 1 MiB
Iterations up to 1M for small payloads

@MaoZiming

@MaoZiming I am working on porting UCCL now and I am checking the code. I have some questions:
Why do we have to use amo here to increment the tail here ? What are the issues if we replace this with a raw write of the tail to the remote? Because in my understanding, the rdma_channel_tail layout is (channel_id, src_rdma_rank) and each nvl_rank only writes to the corresponding nvl_rank, so only one lane in the sender warp of the cuda block with the specific channel_id will write to this remote?

@PanJason

hi @PanJason. We're working on the Isambard cluster in the UK, from what i can tell it has a similar shape - libfabric/CXI/Slingshot etc. We've been working on an internal experimental UCCL-EP fork (for wideEP inference, mostly) - it's pretty AI-driven, at the moment, but it is up and serving well in vLLM and SGLang benchmarks. I put our working (messy) version up here doublewordai#1

Design-wise it matches what you measured: we skip write-with-immediate entirely and use fi_write plus libfabric atomic adds for the tail/control path, leaning on the same-TX-context write→AMO ordering you measured.

would be good to compare notes or to work from either base to get libfabric/CXI support upstreamed to UCCL.

@fergusfinn

@fergusfinn Hi, we have made fi_write + fi_atomic work as well. The problem is non-ideal performance. On our configuration, 4 GPUs per node, each GPU has 1 200Gb/s Slingshot NIC, the UCCL benchmark only give 70% of the NIC bw:

The benchmark shape follows the DeepSeek-V3 pretraining-style configuration where possible.
The command we are using:

bench/test_internode.py \
  --num-tokens 4096 \
  --hidden 7168 \
  --num-topk 8 \
  --num-topk-groups <nodes capped at 4> \
  --num-experts 288 \
  --test-ll-compatibility

Benchmark configuration:

Tokens per batch: 4096.
Hidden size: 7168.
Top-k experts per token: 8.
Total experts: 288.
Local ranks / GPUs per node: 4.
NIC topology: 1 Slingshot/CXI NIC per GPU/rank, with 25 GB/s per NIC
(200 Gb/s).
Dispatch dtypes: FP8 dispatch and BF16 dispatch were benchmarked separately.
Combine dtype: BF16.
Top-k groups: 4 for the 4-node and 8-node runs; 2 for the 2-node run because only two nodes participate.

The table reports the bottleneck bandwidth and transmit latency across node-local benchmark summaries. Internode rows report RDMA bandwidth; the intranode row reports NVL bandwidth. These are fresh validation runs with proxy command tracing disabled (UCCL_PROXY_TRACE=0), not reused allocation data.

Type	FP8 Dispatch #EP	Bottleneck bandwidth & latency	BF16 Dispatch #EP	Bottleneck bandwidth & latency	Combine #EP	Bottleneck bandwidth & latency
Intranode	4	346.08 GB/s (NVL), 317.16 us	4	392.64 GB/s (NVL), 542.17 us	4	364.32 GB/s (NVL), 584.31 us
Internode	8	15.21 GB/s (RDMA), 3967 us	8	15.27 GB/s (RDMA), 7667 us	8	25.99 GB/s (RDMA), 4501 us
Internode	16	17.45 GB/s (RDMA), 6222 us	16	18.26 GB/s (RDMA), 11536 us	16	18.63 GB/s (RDMA), 11388 us
Internode	32	15.40 GB/s (RDMA), 7835 us	32	15.80 GB/s (RDMA), 14787 us	32	15.72 GB/s (RDMA), 14882 us

Did you also see the same low performance on your cluster?

We also did some porting on fi_writedata but I got mysterious ENTRY_NOT_FOUND errors from slingshot, which indicates event buffer on the NIC is overflowing. Did you ever see this?

Also, we did some porting with fi_write + fi_write but have some data corruption. This is why I was asking the question before .

Current analogous numbers:

bench/test_internode.py \
  --num-tokens 4096 \
  --hidden 7168 \
  --num-topk 8 \
  --num-topk-groups <nodes capped at 4> \
  --num-experts 288 \
  --test-ll-compatibility

Type	FP8 Dispatch #EP	Bottleneck bandwidth & latency	BF16 Dispatch #EP	Bottleneck bandwidth & latency	Combine #EP	Bottleneck bandwidth & latency
Intranode	4	347.82 GB/s (NVL), 315.58 us	4	391.53 GB/s (NVL), 543.69 us	4	360.37 GB/s (NVL), 590.71 us
Internode	8	41.59 GB/s (RDMA), 1451 us	8	43.19 GB/s (RDMA), 2708 us	8	43.63 GB/s (RDMA), 2681 us

Hey - cluster is pretty congested at the moment, so i couldn't do the larger benchmarks. The measured ceiling we get from fi_write is something like ~24.3GB/s/NIC. I think amplification over the line rate is basically because the benchmarking script here has the denominator as the kernel time. Which then inflates the bandwidth since transfers are asynchronous with regards to the kernels? when using wall time we get something more like 21GB/s/NIC for the EP8 config.

Re. perf we got quite a bit faster by posting the writes asynchronously and then queueing the AMOs until their writes completed.

We also did some porting on fi_writedata but I got mysterious ENTRY_NOT_FOUND errors from slingshot, which indicates event buffer on the NIC is overflowing. Did you ever see this?

Haven't seen this, we had some problems that i can't quite remember getting fi_writedata working (might have been this) so we dropped it. Will look into it again if i get a chance

Also, we did some porting with fi_write + fi_write but have some data corruption. This is why I was asking the question before

There's some arcane stuff in the CXI spec about ordering of fi_writes, other writes and amos with respect to each other, from what i can tell, in principle they're unordered, which could cause corruption. https://ofiwg.github.io/libfabric/main/man/fi_cxi.7.html. it could be that the empirics above are just an artifact of the small-ish payload sizes. In practise, we wait for CQ completions before posting AMOs, which i think gives ordering. we saw exactly this with immediately-posted AMOs, it was clean at small token counts, corrupted output at ≥1024 tokens. Gating the AMO on the write's CQ completion seemed to fix it. I think this could have something to do with FI_CXI_RDZV_THRESHOLD changing behaviour at different message sizes.

Thanks a lot for the performance numbers. I also ran your code here doublewordai#4 but got very poor performance. Could you please let me know which code you used to generate the numbers above? The number I got is as below:

Branch/run	FP8 dispatch	BF16 dispatch	Combine
Our current `uccl/ep`	15.33 GB/s RDMA	15.34 GB/s RDMA	32.84 GB/s RDMA
PR #4	7.32 GB/s RDMA	6.84 GB/s RDMA	32.48 GB/s RDMA

I am now checking what is going wrong here. If you can share your commit, it would great to debug. The code we used is here https://github.com/swiss-ai/uccl/tree/slingshot-dev
One more thing I would like to ask is whether you have some raw performance numbers of fi_atomic It seems on our platform the fi_atomic is extremely slow. I tried another version of batching the AMOs like what you have done but I got almost the same performance. Since we are using almost the same hardware, it would be really helpful if we could do some AB testing over the network primitives as well to check the slingshot/CXI configuration correctness.

Haven't seen this, we had some problems that i can't quite remember getting fi_writedata working (might have been this) so we dropped it. Will look into it again if i get a chance

I can provide scripts if you need for testing fi_writedata

i think gives ordering. we saw exactly this with immediately-posted AMOs, it was clean at small token counts, corrupted output at ≥1024 tokens.
We actually chose to immediately post the AMO, and so far, I have not seen the corruption you said (I assume the test_internode.py has some correctness checking? At least we did not see corruption here). The reason why we did not use the ack-then-AMO approach is also performance. I tested this technique and got quite low performance as well, roughly similar to what the table shows. I start to suspect that there are some issues with the FI_CXI_* configuration on our platform. Is it possible for you to share a list of FI_CXI_* configurations you used on you platform?

great - yes i will test out your branch!

The tuning variables we have for FI_CXI_* are:

export FI_CXI_DEFAULT_CQ_SIZE="131072"
export FI_CXI_DEFAULT_TX_SIZE="2048"
export FI_CXI_DISABLE_NON_INJECT_MSG_IDC="1"
export FI_HMEM_CUDA_USE_GDRCOPY="1"
export FI_CXI_DISABLE_HOST_REGISTER="1"
export FI_MR_CACHE_MONITOR="userfaultfd"
export FI_CXI_RDZV_PROTO="alt_read"
export FI_CXI_RDZV_THRESHOLD="0"
export FI_CXI_RDZV_GET_MIN="0"
export FI_CXI_RDZV_EAGER_SIZE="0"
export FI_CXI_RX_MATCH_MODE="hybrid"

https://docs.isambard.ac.uk/user-documentation/guides/nccl/ (from here, for reference).

I guess this could explain differences in perf numbers. Let me run w/ your branch and get back to you. (my numbers were from this branch doublewordai#1 - but it has the one you used at its tip basically, so it should have the same perf)

Happy to run any testing scripts for the raw writedata etc. versions.

Branch	FP8 Dispatch (EP8)	BF16 Dispatch (EP8)	Combine (EP8)
`doublewordai/uccl` @ `cxi-ep` (`373ce5d`)	41.62 GB/s (RDMA), 1450 us	43.70 GB/s (RDMA), 2678 us	43.94 GB/s (RDMA), 2662 us
`swiss-ai/uccl` @ `slingshot-dev` (`1da9b40`)	40.18 GB/s (RDMA), 1502 us	42.78 GB/s (RDMA), 2736 us	42.20 GB/s (RDMA), 2772 us

I think it might indeed be cluster tuning - it looks like the two branches are within a hair of one another

Ah thanks a lot for the prompt response and numbers, which narrows the investigation scope a lot.
fi_write/fi_writedata performance test code:
https://pastebin.com/RVK5QD5L
And the slurm scripts we used to test the performance:
https://pastebin.com/yUnVBESL
You can adapt from here to see whether fi_writedata works on your cluster.

I am also doing another AB test now with your FI_CXI configs and your code and see what the results will look like.

I tried with your FI_CXI_* configuration and what I got is still quite low.

Run	FP8 dispatch	BF16 dispatch	Combine
PR1 373ce5d, node 0	8.27 GB/s RDMA	7.90 GB/s RDMA	32.60 GB/s RDMA
PR1 373ce5d, node 1	8.24 GB/s RDMA	8.05 GB/s RDMA	32.89 GB/s RDMA

One thing I found not working on our cluster is FI_THREAD_ENDPOINT and my run had to fall back to FI_THREAD_DOMAIN . I guess likely this is not where the perf diff lies.

So likely this is a slingshot configuration issue. Given that we have the same hardware, a very next step is to compare the raw performance numbers of fi_* primitives such as fi_write and fi_atomic to see where the differences are coming from.

ok, so with API_KIND=writedata, i get -38 (ENOSYS, function not implemented) running your script - this is the error i found when i tried this the first time.

What version of libfabric do you have on your system? I think the reason for my lack of writedata is is that the build here is very old - its showing 1.22, whereas it looks like fi_writedata landed in libfabric a couple months ago (https://github.com/ofiwg/libfabric/blob/main/NEWS.md#v250-fri-march-20-2026).

I built 2.5.1 locally, and i get the same error you described with fi_writedata:

CQ error: err=5 (Input/output error) prov_errno=18 provider='ENTRY_NOT_FOUND' flags=0x204

Then, perf numbers for fi_write, for both libfabric versions, 'our env' is the one from above, 'default env' is the one that's defaulted in the scripts with our FI_CXI_* toggled off. Swept both slots and payload sizes because i got some weird data, but its confined to 1.22

Payload	Buffer	1.22 + our env	1.22 + default env	2.5.1 + our env	2.5.1 + default env
4 KiB	16 MiB	4.4	4.3	4.4	4.5
16 KiB	64 MiB	17.0	16.9	16.8	16.7
64 KiB	256 MiB	21.3	20.8	21.0	21.2
256 KiB	1 GiB	20.3	20.3	20.3	20.3
320 KiB	1.25 GiB	20.4	20.4	20.4	20.4
512 KiB	2 GiB	1.7	1.7	20.2	20.2
1 MiB	4 GiB	2.9	2.8	19.7	19.7

SLOTS	Buffer	1.22 + our env	1.22 + default env	2.5.1 + our env	2.5.1 + default env
4096	1 GiB	20.3	20.3	20.3	20.3
5120	1.25 GiB	20.2	20.2	20.2	20.1
6144	1.5 GiB	1.7	1.7	19.6	20.2
7168	1.75 GiB	1.7	1.2	20.1	20.1
8192	2 GiB	1.7	1.7	20.1	20.0

There's something up with large registered buffer sizes on 1.22, so at the least this has convinced me i should upgrade!

Then i reran in case libfabric version explained the perf difference on your cluster, it doesn't seem to:

Branch	libfabric	FP8 Dispatch	BF16 Dispatch	Combine
`doublewordai/uccl` @ `cxi-ep`	1.22 (system)	41.6 GB/s	43.7 GB/s	43.9 GB/s
`doublewordai/uccl` @ `cxi-ep`	2.5.1 (built)	41.5 GB/s	43.3 GB/s	44.0 GB/s
`swiss-ai/uccl` @ `slingshot-dev`	1.22 (system)	40.2 GB/s	42.8 GB/s	42.2 GB/s
`swiss-ai/uccl` @ `slingshot-dev`	2.5.1 (built)	38.6 GB/s	42.2 GB/s	41.6 GB/s

Cool thank you Fergus. Currently our cluster is down because of power outage but I can still provide you with some info.

libfabric version

2.5.0

fi_write performance

Earlier results are similar to yours 2.5.1 version, need a sweep to verify when the cluster comes back

I guess likely fi_write is not the issue, probably fi_atomic. On our cluster fi_atomic performance is very poor. I prepared a reproducer doc for you and can you also run that to align?
https://pastebin.com/P3bktrGD

ok great! glad that commit helped. I'm happy to rebase the other work i'm doing onto your branch and build & test on that - do you have an up to date public branch? I'd like to test it under load and see if it holds up. If you open it as a PR against your fork main, we can move the detailed discussion there?

Nice I willl create a branch and a PR and at you there. Then we can work together for more tests. The reason I haven't created it yet is that I encountered some errors while testing EP==64. Trying to figure out where the error is coming from and I don't want to push something that has errors on the benchmark.

@fergusfinn

@fergusfinn I updated the results above. It turned out to be the benchmark problem. The benchmark after I changed is here https://pastebin.com/DKu0vZiD. Also your PR1 failed in EP==64 case here:

num_rdma_bytes < std::numeric_limits<int>::max()

At ep/src/internode.cu:2008
Probably it is worth checking.

Now I think the code is mostly fine. I am creating the PR.

@fergusfinn

@fergusfinn I created the PR here: #997
The public branch is here https://github.com/swiss-ai/uccl/tree/main and https://github.com/swiss-ai/uccl/tree/slingshot-dev-clean which only include the basic porting code and minimum changes to the existing benchmark. I sent you the invitation.
You can pick main, and I will keep rebasing main over the UCCL main, so our PR is always in a mergeable state.
We can move our discussion there if you want.

great! i'll get that compiling and reproduce your numbers, and then run it in vLLM

Are your vLLM tests outside the UCCL tracking tree or inside? If you test vLLM on inference side, I can then test the training side.

outside - for vllm, UCCL is deepEP compatible, so there's no vLLM code changes required, and tests at the moment look like sending lots of HTTP requests.

I implemented fi_writedata() for CXI. Please let me know if I can help.

Hi Ryan, I have also implemented one version using fi_writedata. The reason why I did not use that version is that I keep getting ENTRY_NOT_FOUND errors from slingshot nic. I tried to increase the cq size on the remote side, which was proven not useful. Do you have any idea about this issue?

Hi Ryan, I have also implemented one version using fi_writedata. The reason why I did not use that version is that I keep getting ENTRY_NOT_FOUND errors from slingshot nic. I tried to increase the cq size on the remote side, which was proven not useful. Do you have any idea about this issue?

Can you please try to set: export OFI_NCCL_DISABLE_DMABUF=1 and see if it overcomes this error? There is a bug in some versions of libfabric.

p2p/cxi: simplify low-version warning to point at uccl-project#956

p2p/cxi: log libfabric runtime version, warn on <2.0 large-MR cliff (#…

The original CXI/Slingshot backend request is implemented by #997, which is merged into main. It adds the libfabric cxi transport for UCCL-EP, enabled with USE_LIBFABRIC_CXI=1, including the proxy integration and RDMA/atomic operations.

This proposal can be closed as implemented; any remaining CXI performance or shutdown problems should be tracked in their specific issues.

Navigation Menu

Repository navigation

[Proposal] libfabric-CXI backend for UCCL-EP on HPE Slingshot #956

[Proposal]: libfabric-CXI backend for UCCL-EP on HPE Slingshot

What we'd contribute

What we plan to verify first currently

Why we think this can be useful for both sides

Support for AMO writes

Support for ordering

Relevant interfaces

Host-Memory Workflow

Device-memory payload + DRAM control word

Device-memory payload + control word

Conclusion

Appendix:

Different Configurations I have tested for the empirical results:

8 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Proposal] libfabric-CXI backend for UCCL-EP on HPE Slingshot #956

Description

[Proposal]: libfabric-CXI backend for UCCL-EP on HPE Slingshot

What we'd contribute

What we plan to verify first currently

Why we think this can be useful for both sides

Activity

PanJason commented on May 21, 2026

MaoZiming commented on May 21, 2026

PanJason commented on May 21, 2026

PanJason commented on May 26, 2026

Support for AMO writes

Support for ordering

Relevant interfaces

Host-Memory Workflow

Device-memory payload + DRAM control word

Device-memory payload + control word

Conclusion

Appendix:

Different Configurations I have tested for the empirical results:

PanJason commented on Jun 9, 2026

fergusfinn commented on Jun 10, 2026

PanJason commented on Jun 10, 2026

fergusfinn commented on Jun 11, 2026

PanJason commented on Jun 11, 2026

fergusfinn commented on Jun 11, 2026

fergusfinn commented on Jun 11, 2026

PanJason commented on Jun 11, 2026

PanJason commented on Jun 11, 2026

fergusfinn commented on Jun 12, 2026

PanJason commented on Jun 12, 2026

8 remaining items

fergusfinn commented on Jun 15, 2026

PanJason commented on Jun 15, 2026

PanJason commented on Jun 15, 2026

PanJason commented on Jun 16, 2026

fergusfinn commented on Jun 16, 2026

PanJason commented on Jun 16, 2026

fergusfinn commented on Jun 17, 2026

ryanhankins commented on Jun 23, 2026

PanJason commented on Jun 24, 2026

ryanhankins commented on Jun 24, 2026

YangZhou1997 commented on Nov 8, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions