One of the reasons for this is TCP window size. Its size is dictated by socket's ReceiveBufferSize. But because client has to negotiate TCP window during TCP handshake, changing ReceiveBufferSize after a connection is established doesn't change TCP window. As an example, if I set SocketReceiveBufferSize to 100000, OS seems to ignore it and instead negotiates 327168.

If we instead set the option before establishing connection, OS does respect it and negotiates exactly 100000.