Motivation

#16787 fixed the READ_VARIABLE_HEADER too-long check on the 4.2 branch by replacing a ReplayingDecoderByteBuf.readableBytes() probe with the message size declared by the fixed header. That fix was auto-ported to 4.1 in #16838, but a later CVE-2026-44248 security merge re-introduced the broken code on 4.1, so the 4.1 branch still carries the regression:

int initialAvailableBytes = buffer.readableBytes();
...
if (initialAvailableBytes < maxBytesInMessage) {
    throw signal;   // REPLAY
} else {
    bailOut = true; // too long
}

Because MqttDecoder extends ReplayingDecoder, buffer is a ReplayingDecoderByteBuf whose readableBytes() returns Integer.MAX_VALUE - readerIndex rather than the bytes actually buffered. With the default maxBytesInMessage of 8092, the initialAvailableBytes < maxBytesInMessage check is therefore effectively always false. So when decodeVariableHeader asks for a REPLAY because the variable header has not fully arrived yet, the decoder takes the bailOut branch and rejects the message with a TooLongFrameException instead of waiting for the rest — a valid message whose variable header is split across reads is dropped.

Modification

Re-apply the #16787 fix on 4.1: drop the initialAvailableBytes / readableBytes() probe and decide based on bytesRemainingBeforeVariableHeader (the remaining length declared by the fixed header). A genuinely oversized message is still rejected by the existing bytesRemainingBeforeVariableHeader > maxBytesInMessage check; an incomplete one now correctly REPLAYs.

Result

A message whose variable header arrives in chunks is decoded once complete, instead of being rejected as too long, while declared-oversize messages still fail with TooLongFrameException. The two regression tests from #16787 are ported here. All codec-mqtt tests pass locally.

Note: this targets 4.1 only and intentionally carries no cherry-pick label — 4.2 already has the fix via #16787. Found while addressing @chrisvest's review comment on #16813 about the same readableBytes() antipattern.