This is the second attempt to address #287, see also #288

Problem: The specific problem we are trying to address is to scale TURN/UDP server listeners beyond a single CPU. Currently we allocate a single global net.PacketConn per TURN/UDP listener that is shared across all clients. Since there is only a single readLoop to drain the global PacketConn, TURN/UDP listeners are practically limited to a single CPU. See more on this here.

Prior attempts: The first draft PR #288 addresses this issue by forking off a separate readloop thread per each client, which allows the server to drain each client connection on a separate CPU thread. In order to implement this in a secure way #288 introduces some fairly intrusive changes to the main TURN server implementation: there is an API change where the user must provide a Connect callback that the server calls after a new TURN allocation is created to make the per-client connection, and there is some major refactoring in internal/server as well to make this possible. Meanwhile, a security problem is still lurking in the background: without the implementation of per-user allocation quotas it is still possible for a single high-bandwidth attacker to starve the rest of the clients by creating multiple TURN allocations and consuming excess CPU.

Goal: This PR takes a completely different approach: the aim is to scale UDP/TURN listeners beyond the single-CPU limit but meanwhile (1) minimize the changes required in the existing pion/turn implementation (the PR requires absolutely no change to the server code) and (2) allow the operator to tightly control the number of CPUs allocated to each TURN/UDP listener.

Implementation: The idea is that the caller creates a fixed number of UDP listener sockets that share the same local addr:port by using the SO_REUSEPORT socket option and then initializes a separate TURN listener per each socket. Note that when multiple sockets share the same local address with SO_REUSEPORT then the kernel will load-balance packets across the sockets by the IP five-tuple hash (see here), which makes sure that each packet of a TURN allocation will be processed by the same readLoop and so the allocation manager doesn't need to be shared across CPU threads. By controlling the number of sockets, the caller can control the number of CPUs allocated to each listener. Since this requires no change at all in the TURN server code (every socket will be handled by pion/turn as a separate UDP/TURN listener), the PR contains only a simple extension of the example server (see examples/turn-server/mult-threaded-udp/main.go) to implement the caller-side changes.

The pros are simplicity and control over CPU consumption. The cons are (1) that SO_REUSEPORT is not portable (we don't see this as a problem: recall that nothing in the server knows anything about whether SO_REUSEPORT is in use, all required changes are on the caller side) and (2) performance is slightly smaller than with #288. This is because hash-based load-balancing used by the kernel to distribute TURN allocations across the CPU threads is prone to an unequal distribution of load across worker threads: this can be taken care of by implementing an adaptive load-balancing algorithm with eBPF (say, using the power-of-two choices algorithm).

Performance: Attached is a script to benchmark the naive TURN server with (mult-threaded-udp) and without (simple) the patch. Requires the turncat TURN client (you can obtain from here) in the source dir, plus iperfv2 in the PATH. On a 24-core Intel server and 8 iperf threads, the built-in simple server (./test2.sh simple 16000000) produces roughly 41 kpps (thousand packets per second) packet rate and uses about 141% CPU, while the server using 4 SO_REUSEPORT sockets and so 4 readLoop threads (./test.sh mult-threaded-udp 16000000) produces 121 kpps with 390% CPU usage. Observe that both the performance and the CPU usage is smaller than with #288, this is because of the uneven load-distribution (the difference between the least-loaded thread and the most-loaded thread can be more than 50% in our benchmarks).

Feedback appreciated.
test2.sh.gz