I should also protect the SigningVerifier, so to draft for now.

nit: unique_ptr will help, since we don't allow copies anyway and it handles free() and gets zeroed when when move happens

idk what to answer for review, but 'it does not look simplified' :)
given the stability of the existing code (as it was the main argument from Matrix), we should probably rethink the overall structure instead of pushing certain features to the current one that make it more complicated instead of the opposite