Just force-pushed my current version. This version is now based on the current develop branch (v13).

A lot has changed in the code, especially the way I propagate and validate signature shares between LLMQ members. In the previous version, I was directly pushing sigshare batches to intra-quorum connected nodes, without checking if these already have these shares. I did this because the classical INV system would already send around messages of at least 32 bytes size, so I assumed directly sending the shares itself would actually reduce the overhead of the INV system. Turned out it still produces too much overhead and chatter between nodes, especially as we now have switched to BLS12-381 with 96 byte signatures. With this switch, even the classical INV system would probably be better performing then what I did initially (direct push).

Instead of switching to the INV system, I implemented a specialized inventory system for signature shares. It should perform MUCH better and with much less overhead then the general purpose INV system.

Instead of announcing INV items to the other nodes, LLMQ members will now send CSigSharesInv messages to other members. It contains some information to identify the signing session to which the shares belong and a bitset that represents the shares offered. Each bit represents a share from a LLMQ member. Bit at index 10 means it's a share from member 10.

Members will now regularly (every 100ms) check which shares were announced by other members and which shares are missing locally and request these from other members. They will also check what other nodes requested and send them these shares. All the communication is leveraging the CSigSharesInv object to reduce overhead to the minimum and to request/announce as many shares as possible in one go. Shares are still sent with the CBatchedSigShares to reduce overhead even further.

Also, in the old version I extensively used the CBLSWorker to speed up things in the handling of signature shares. I removed this and will also remove the use of it in other places later. Turned out that the overhead due to moving BLS primitives around between threads destroys most of the performance gains. Especially now as we switched to BLS12-381, where the primitives got a lot larger.

Instead, I'm leveraging (single-threaded) batched verification as much as possible now. Incoming shares are put into a pending batch and then processed every 100ms. Verification is then done in sub-batches to avoid DoS attacks (an attacker sending 100 valid shares and one last invalid one). I first try to batch together from all nodes and if that succeeds, all shares are accepted. If it fails, I revert to per-node batching to figure out which node sent the invalid shares.

I'll document all this in-code when I find time, as reviewing it will otherwise not be fun.

Another change is how I determine to which nodes to announce shares to. Previously, I only sent shares to members which were determined by the intra-quorum connections selection algorithm. This however had the disadvantage that it only announced shares to outgoing connections and did not take incoming connection from other members into account. As there is no easy way to determine which incoming connection is from a LLMQ member, I implemented something like "if someone ever sends me a share belonging to my quorum, he must be interested in my shares as well, so I announce him all the shares I got from now on". It doesn't necessarily mean it's really a member of the same LLMQ, but its 99% likely.

Also updated the example dash.conf to work with the changes we did in develop

Closing this PR as the code has been merged through smaller PRs.