Skip to content

Auto-port 4.1: Fix MQTT decoder size check after variable header replay#16838

Merged
chrisvest merged 1 commit into
4.1from
auto-port-pr-16787-to-4.1
May 20, 2026
Merged

Auto-port 4.1: Fix MQTT decoder size check after variable header replay#16838
chrisvest merged 1 commit into
4.1from
auto-port-pr-16787-to-4.1

Conversation

@netty-project-bot
Copy link
Copy Markdown
Contributor

Auto-port of #16787 to 4.1
Cherry-picked commit: 72df658


Problem

The MQTT decoder can reject valid packets after the CVE-2026-44248 fix when several MQTT packets are present in the same cumulation buffer. If the current packet's variable header needs a replay, the decoder compares the total readable bytes in the buffer against maxBytesInMessage, so later packets can make the current in-limit packet look too large.

Root Cause

READ_VARIABLE_HEADER recorded buffer.readableBytes() before decoding the variable header. That value is the cumulation's total readable bytes, not the current MQTT packet's declared remaining length. When decodeVariableHeader throws Signal.REPLAY, the too-large decision must be based on bytesRemainingBeforeVariableHeader for the current packet.

Fix

  • Use bytesRemainingBeforeVariableHeader when deciding whether to swallow Signal.REPLAY and raise TooLongFrameException.
  • Continue replaying when the current packet is within maxBytesInMessage, even if the cumulation contains additional bytes for following packets.

Tests Added

Change Point Test
Variable-header replay uses the current packet size instead of total cumulation bytes for the too-long check testPublishMessageIncompleteVariableHeaderDoesNotUseCumulationSizeForTooLongCheck() verifies an incomplete in-limit PUBLISH variable header with extra cumulated PINGREQ packets does not emit an invalid message
Oversized current packets still fail during variable-header replay testPublishMessageIncompleteVariableHeaderStillFailsWhenCurrentPacketTooLarge() verifies an incomplete PUBLISH whose own remaining length exceeds maxBytesInMessage still emits a TooLongFrameException

Impact

This restores decoding for valid in-limit MQTT packets batched in the same buffer while preserving the CVE fix: packets whose declared remaining length exceeds maxBytesInMessage still fail with TooLongFrameException even if variable-header decoding requests replay.

Fixes #16776

## Problem

The MQTT decoder can reject valid packets after the CVE-2026-44248 fix
when several MQTT packets are present in the same cumulation buffer. If
the current packet's variable header needs a replay, the decoder
compares the total readable bytes in the buffer against
`maxBytesInMessage`, so later packets can make the current in-limit
packet look too large.

## Root Cause

`READ_VARIABLE_HEADER` recorded `buffer.readableBytes()` before decoding
the variable header. That value is the cumulation's total readable
bytes, not the current MQTT packet's declared remaining length. When
`decodeVariableHeader` throws `Signal.REPLAY`, the too-large decision
must be based on `bytesRemainingBeforeVariableHeader` for the current
packet.

## Fix

- Use `bytesRemainingBeforeVariableHeader` when deciding whether to
swallow `Signal.REPLAY` and raise `TooLongFrameException`.
- Continue replaying when the current packet is within
`maxBytesInMessage`, even if the cumulation contains additional bytes
for following packets.

## Tests Added

| Change Point | Test |
|-------------|------|
| Variable-header replay uses the current packet size instead of total
cumulation bytes for the too-long check |
`testPublishMessageIncompleteVariableHeaderDoesNotUseCumulationSizeForTooLongCheck()`
verifies an incomplete in-limit PUBLISH variable header with extra
cumulated PINGREQ packets does not emit an invalid message |
| Oversized current packets still fail during variable-header replay |
`testPublishMessageIncompleteVariableHeaderStillFailsWhenCurrentPacketTooLarge()`
verifies an incomplete PUBLISH whose own remaining length exceeds
`maxBytesInMessage` still emits a `TooLongFrameException` |

## Impact

This restores decoding for valid in-limit MQTT packets batched in the
same buffer while preserving the CVE fix: packets whose declared
remaining length exceeds `maxBytesInMessage` still fail with
`TooLongFrameException` even if variable-header decoding requests
replay.

Fixes #16776

(cherry picked from commit 72df658)
@chrisvest chrisvest added this to the 4.1.134.Final milestone May 20, 2026
@chrisvest chrisvest enabled auto-merge (squash) May 20, 2026 18:33
@chrisvest chrisvest merged commit 30f8f28 into 4.1 May 20, 2026
18 checks passed
@chrisvest chrisvest deleted the auto-port-pr-16787-to-4.1 branch May 20, 2026 20:43
chrisvest pushed a commit that referenced this pull request Jun 5, 2026
### Motivation

#16787 fixed the `READ_VARIABLE_HEADER` too-long check on the **4.2**
branch by replacing a `ReplayingDecoderByteBuf.readableBytes()` probe
with the message size declared by the fixed header. That fix was
auto-ported to 4.1 in #16838, **but a later CVE-2026-44248 security
merge re-introduced the broken code on 4.1**, so the 4.1 branch still
carries the regression:

```java
int initialAvailableBytes = buffer.readableBytes();
...
if (initialAvailableBytes < maxBytesInMessage) {
    throw signal;   // REPLAY
} else {
    bailOut = true; // too long
}
```

Because `MqttDecoder extends ReplayingDecoder`, `buffer` is a
`ReplayingDecoderByteBuf` whose `readableBytes()` returns
`Integer.MAX_VALUE - readerIndex` rather than the bytes actually
buffered. With the default `maxBytesInMessage` of `8092`, the
`initialAvailableBytes < maxBytesInMessage` check is therefore
effectively always false. So when `decodeVariableHeader` asks for a
`REPLAY` because the variable header has not fully arrived yet, the
decoder takes the `bailOut` branch and rejects the message with a
`TooLongFrameException` instead of waiting for the rest — a valid
message whose variable header is split across reads is dropped.

### Modification

Re-apply the #16787 fix on 4.1: drop the `initialAvailableBytes` /
`readableBytes()` probe and decide based on
`bytesRemainingBeforeVariableHeader` (the remaining length declared by
the fixed header). A genuinely oversized message is still rejected by
the existing `bytesRemainingBeforeVariableHeader > maxBytesInMessage`
check; an incomplete one now correctly `REPLAY`s.

### Result

A message whose variable header arrives in chunks is decoded once
complete, instead of being rejected as too long, while declared-oversize
messages still fail with `TooLongFrameException`. The two regression
tests from #16787 are ported here. All `codec-mqtt` tests pass locally.

Note: this targets **4.1 only** and intentionally carries no cherry-pick
label — 4.2 already has the fix via #16787. Found while addressing
@chrisvest's review comment on #16813 about the same `readableBytes()`
antipattern.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants