Hello @pizhenwei , I'm very interested in you proposal, and the protocol seems very novel comparing to other rdma implementation like brpc and NVMe-oF. I also have some questions about the proposal, hoping I didn't miss anything.

1. What's the mapping between QP and client/server. If many QPs per client, it's somehow a little wasteful. But if one QP per client, it may need some multiplexing mechanism, since some Redis command is blocking, a request may block the following ones.
2. Registering the memory region in batch benefits the performance, but it also involves the problem of low memory utilization. Supposing a redis-server with 1000 clients, which's normal in production environment, and reserving 1 MB memory region for every client, then 1GB is used, it's a little wasteful for memory database.
3. What about the huge request/response case, that the size of request/response is larger than the preserved memory region size, e.g. a string larger than 1MB, will the protocol support interleaving write and register?

In general I think the protocol and implementation is neat and beautiful, it deserve more attention, for the sake of research/study or production.

Hello @pizhenwei , I'm very interested in you proposal, and the protocol seems very novel comparing to other rdma implementation like brpc and NVMe-oF. I also have some questions about the proposal, hoping I didn't miss anything.

Hi, I tried to describe the deference and comparing to other protocols, please see link.

1. What's the mapping between QP and client/server. If many QPs per client, it's somehow a little wasteful. But if one QP per client, it may need some multiplexing mechanism, since some Redis command is blocking, a request may block the following ones.

Just imagine a QP(RC type) as a connection of TCP/TLS/Unix socket. If a client uses N sockets, it may need N QPs. (in fact, many sockets also waste resources in the kernel).

2. Registering the memory region in batch benefits the performance, but it also involves the problem of low memory utilization. Supposing a redis-server with 1000 clients, which's normal in production environment, and reserving 1 MB memory region for every client, then 1GB is used, it's a little wasteful for memory database.

Currently, only one memory region per QP is defined. And no strict memory region size limitation in protocol. As far as I can see in the engineering implementation:

• the server side could use a configurable size of 'RX' memory region.(the more memory used, the higher performance got. so I guess a typical size will be found during testing the real workload). I have implemented a POC version, see PR.
• the server side could use a small memory region as 'TX memory' against a large 'RX memory' of a client. (In my plan, but not implement currently)

3. What about the huge request/response case, that the size of request/response is larger than the preserved memory region size, e.g. a string larger than 1MB, will the protocol support interleaving write and register?

For example, transfer 10MB string over 1MB memory, this works like:

Register 1M memory, send 0-1MB, register 1M memory, sent 1MB-2MB ....

In general I think the protocol and implementation is neat and beautiful, it deserve more attention, for the sake of research/study or production.

Thanks!
This can be tested by repo

client:
use branch: feature-rdma-with-cli
make distclean; make BUILD_RDMA=yes -j

server:
use branch: feature-rdma
make distclean; make BUILD_RDMA=module -j

All committers have signed the CLA.

So sad, years of waiting have made me lose patience and confidence.