Skip to content

MQTT: Make the decodeProperties early-REPLAY check actually fire#16813

Merged
chrisvest merged 1 commit into
netty:4.1from
daguimu:fix/mqtt-decode-properties-replay
Jun 5, 2026
Merged

MQTT: Make the decodeProperties early-REPLAY check actually fire#16813
chrisvest merged 1 commit into
netty:4.1from
daguimu:fix/mqtt-decode-properties-replay

Conversation

@daguimu
Copy link
Copy Markdown
Contributor

@daguimu daguimu commented May 14, 2026

Motivation:

The CVE-2026-44248 fix in 82f47fa added a guard at the top of decodeProperties to trigger an early REPLAY when the cumulation buffer did not yet have the full properties block:

if (buffer.readableBytes() < totalPropertiesLength) {
    buffer.readSlice(totalPropertiesLength);
}

Because MqttDecoder extends ReplayingDecoder, the buffer passed to decodeProperties is a ReplayingDecoderByteBuf whose readableBytes() returns Integer.MAX_VALUE - readerIndex rather than the actual number of bytes available (see ReplayingDecoderByteBuf.readableBytes()). The if-condition is therefore false in practice and the readSlice() call never executes, so the early-REPLAY optimization is effectively dead code. When the properties block arrives in chunks the decoder still falls back to partial parsing followed by a mid-loop REPLAY, which defeats the optimization's intent on slow streams.

Modification:

Replace the broken readableBytes() check with a getByte() probe at buffer.readerIndex() + totalPropertiesLength - 1. ReplayingDecoderByteBuf.checkIndex throws Signal.REPLAY when index + 1 > writerIndex — i.e. exactly when the cumulation buffer does not yet hold the full properties block. The call has no side effect on readerIndex, so subsequent parsing is unchanged once the data arrives. A totalPropertiesLength > 0 guard skips the probe when there are no properties.

Result:

The early-REPLAY guard now actually fires when properties data is incomplete, restoring the CVE-2026-44248 fix's intent of avoiding repeated partial-properties parsing on slow streams. No semantic change on the happy path; all 107 existing codec-mqtt tests pass locally.

Note: this is complementary to #16787, which addresses the outer variable-header REPLAY handling broken by the same CVE patch (same buffer.readableBytes() antipattern in a ReplayingDecoder context). Both fixes are independent and can land in either order; together they restore the optimization the CVE-2026-44248 fix originally aimed for.

Motivation:

The CVE-2026-44248 fix added a guard at the top of decodeProperties
to trigger an early REPLAY when the cumulation buffer did not yet have
the full properties block:

    if (buffer.readableBytes() < totalPropertiesLength) {
        buffer.readSlice(totalPropertiesLength);
    }

Because MqttDecoder extends ReplayingDecoder, the buffer passed to
decodeProperties is a ReplayingDecoderByteBuf whose readableBytes()
returns Integer.MAX_VALUE - readerIndex rather than the actual number
of bytes available. The if-condition is false in practice and the
readSlice() call never executes, so the early-REPLAY optimization is
effectively dead code. When the properties block arrives in chunks the
decoder still falls back to partial parsing followed by a mid-loop
REPLAY, defeating the optimization's intent on slow streams.

Modification:

Replace the broken readableBytes() check with a getByte() probe at
buffer.readerIndex() + totalPropertiesLength - 1. The
ReplayingDecoderByteBuf.checkIndex implementation throws Signal.REPLAY
when index + 1 > writerIndex - i.e. exactly when the cumulation buffer
does not yet hold the full properties block. The call has no side effect
on readerIndex, so subsequent parsing is unchanged once the data arrives.
A totalPropertiesLength > 0 guard skips the probe when there are no
properties.

Result:

The early-REPLAY guard now actually fires when properties data is
incomplete, restoring the CVE-2026-44248 fix's intent of avoiding
repeated partial-properties parsing on slow streams. No semantic change
on the happy path; all existing codec-mqtt tests pass.
@normanmaurer normanmaurer requested a review from chrisvest May 18, 2026 14:10
@chrisvest chrisvest added this to the 4.1.135.Final milestone Jun 1, 2026
@chrisvest chrisvest added needs-cherry-pick-4.2 This PR should be cherry-picked to 4.2 once merged. needs-cherry-pick-5.0 This PR should be cherry-picked to 5.0 once merged. labels Jun 1, 2026
@chrisvest chrisvest modified the milestones: 4.1.135.Final, 4.1.136.Final Jun 2, 2026
@chrisvest chrisvest removed the needs-cherry-pick-5.0 This PR should be cherry-picked to 5.0 once merged. label Jun 4, 2026
@chrisvest
Copy link
Copy Markdown
Member

@daguimu We also use readableBytes on MqttDecoder.java:100 which is likewise problematic. Perhaps we need a protected method in ReplayingDecoder to bypass the ReplayingDecoderByteBuf.readableBytes() implementation.

@daguimu
Copy link
Copy Markdown
Contributor Author

daguimu commented Jun 5, 2026

Good catch — you're right that line 100 has the same ReplayingDecoderByteBuf.readableBytes() problem. It turns out that's actually a regression rather than a new issue: #16787 already fixed exactly this on 4.2 (and it was auto-ported to 4.1 in #16838), but the later CVE-2026-44248 security merge re-introduced the broken initialAvailableBytes = buffer.readableBytes() on 4.1. So 4.1 currently carries the regression while 4.2 is fine.

I've opened #16916 to re-apply the #16787 fix on 4.1 (drop the readableBytes() probe, decide on the declared bytesRemainingBeforeVariableHeader instead, plus the two regression tests from #16787). I kept it out of this PR on purpose: this one is labelled needs-cherry-pick-4.2, and folding the line-100 change in here would collide with #16787 when it ports to 4.2. #16916 targets 4.1 only and carries no cherry-pick label.

On the "protected method to bypass ReplayingDecoderByteBuf.readableBytes()" idea — for line 100 we don't actually need the buffer's readable count at all; the declared size (bytesRemainingBeforeVariableHeader) is the right signal, which is what #16787/#16916 use. For decodeProperties here, the method is static so it can't reach actualReadableBytes(), hence the getByte() probe. Happy to introduce a shared accessor instead if you'd prefer a uniform approach.

chrisvest pushed a commit that referenced this pull request Jun 5, 2026
### Motivation

#16787 fixed the `READ_VARIABLE_HEADER` too-long check on the **4.2**
branch by replacing a `ReplayingDecoderByteBuf.readableBytes()` probe
with the message size declared by the fixed header. That fix was
auto-ported to 4.1 in #16838, **but a later CVE-2026-44248 security
merge re-introduced the broken code on 4.1**, so the 4.1 branch still
carries the regression:

```java
int initialAvailableBytes = buffer.readableBytes();
...
if (initialAvailableBytes < maxBytesInMessage) {
    throw signal;   // REPLAY
} else {
    bailOut = true; // too long
}
```

Because `MqttDecoder extends ReplayingDecoder`, `buffer` is a
`ReplayingDecoderByteBuf` whose `readableBytes()` returns
`Integer.MAX_VALUE - readerIndex` rather than the bytes actually
buffered. With the default `maxBytesInMessage` of `8092`, the
`initialAvailableBytes < maxBytesInMessage` check is therefore
effectively always false. So when `decodeVariableHeader` asks for a
`REPLAY` because the variable header has not fully arrived yet, the
decoder takes the `bailOut` branch and rejects the message with a
`TooLongFrameException` instead of waiting for the rest — a valid
message whose variable header is split across reads is dropped.

### Modification

Re-apply the #16787 fix on 4.1: drop the `initialAvailableBytes` /
`readableBytes()` probe and decide based on
`bytesRemainingBeforeVariableHeader` (the remaining length declared by
the fixed header). A genuinely oversized message is still rejected by
the existing `bytesRemainingBeforeVariableHeader > maxBytesInMessage`
check; an incomplete one now correctly `REPLAY`s.

### Result

A message whose variable header arrives in chunks is decoded once
complete, instead of being rejected as too long, while declared-oversize
messages still fail with `TooLongFrameException`. The two regression
tests from #16787 are ported here. All `codec-mqtt` tests pass locally.

Note: this targets **4.1 only** and intentionally carries no cherry-pick
label — 4.2 already has the fix via #16787. Found while addressing
@chrisvest's review comment on #16813 about the same `readableBytes()`
antipattern.
@chrisvest chrisvest merged commit 2d781e2 into netty:4.1 Jun 5, 2026
39 checks passed
@netty-project-bot
Copy link
Copy Markdown
Contributor

Auto-port PR for 4.2: #16919

@github-actions github-actions Bot removed the needs-cherry-pick-4.2 This PR should be cherry-picked to 4.2 once merged. label Jun 5, 2026
@chrisvest
Copy link
Copy Markdown
Member

Thanks for finding these. Good to have this straightened out again.

chrisvest pushed a commit that referenced this pull request Jun 5, 2026
…ually fire (#16919)

Auto-port of #16813 to 4.2
Cherry-picked commit: 2d781e2

---
Motivation:

The CVE-2026-44248 fix in 82f47fa added
a guard at the top of `decodeProperties` to trigger an early REPLAY when
the cumulation buffer did not yet have the full properties block:

```java
if (buffer.readableBytes() < totalPropertiesLength) {
    buffer.readSlice(totalPropertiesLength);
}
```

Because `MqttDecoder` extends `ReplayingDecoder`, the buffer passed to
`decodeProperties` is a `ReplayingDecoderByteBuf` whose
`readableBytes()` returns `Integer.MAX_VALUE - readerIndex` rather than
the actual number of bytes available (see
`ReplayingDecoderByteBuf.readableBytes()`). The `if`-condition is
therefore false in practice and the `readSlice()` call never executes,
so the early-REPLAY optimization is effectively dead code. When the
properties block arrives in chunks the decoder still falls back to
partial parsing followed by a mid-loop REPLAY, which defeats the
optimization's intent on slow streams.

Modification:

Replace the broken `readableBytes()` check with a `getByte()` probe at
`buffer.readerIndex() + totalPropertiesLength - 1`.
`ReplayingDecoderByteBuf.checkIndex` throws `Signal.REPLAY` when `index
+ 1 > writerIndex` — i.e. exactly when the cumulation buffer does not
yet hold the full properties block. The call has no side effect on
`readerIndex`, so subsequent parsing is unchanged once the data arrives.
A `totalPropertiesLength > 0` guard skips the probe when there are no
properties.

Result:

The early-REPLAY guard now actually fires when properties data is
incomplete, restoring the CVE-2026-44248 fix's intent of avoiding
repeated partial-properties parsing on slow streams. No semantic change
on the happy path; all 107 existing `codec-mqtt` tests pass locally.

Note: this is complementary to #16787, which addresses the *outer*
variable-header REPLAY handling broken by the same CVE patch (same
`buffer.readableBytes()` antipattern in a ReplayingDecoder context).
Both fixes are independent and can land in either order; together they
restore the optimization the CVE-2026-44248 fix originally aimed for.

Co-authored-by: Guimu <[email protected]>
@daguimu daguimu deleted the fix/mqtt-decode-properties-replay branch June 7, 2026 13:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants