Skip to content

Netty Issue: GZIP trailer corruption with large payloads after upgrading from 4.2.9 to 4.2.11 #16656

@nsnmurthyk

Description

@nsnmurthyk

Title

ZipException: Corrupt GZIP trailer for large GZIP-compressed HTTP request bodies after upgrading from 4.2.9.Final to 4.2.11.Final

Netty version

  • Affected: 4.2.11.Final (also likely 4.2.10.Final)
  • Working: 4.2.9.Final

Java version

  • OpenJDK 21 (also observed on JDK 17)

Deployment environment

  • Kubernetes (containerized microservices)
  • Linux containers

Summary

After upgrading Netty from 4.2.9.Final to 4.2.11.Final, we started observing java.util.zip.ZipException: Corrupt GZIP trailer errors when processing large GZIP-compressed HTTP POST request bodies (≥ 3MB uncompressed). Smaller payloads (< 1MB) are not affected. The exact same client payloads work without any issues on 4.2.9.Final.

The issue is intermittent — it does not reproduce on every request, but is consistently observed under production load with larger payloads. Reverting to 4.2.9.Final immediately resolves the issue.

Stack trace

java.io.UncheckedIOException: java.util.zip.ZipException: Corrupt GZIP trailer
    at java.base/java.io.BufferedReader$1.hasNext(BufferedReader.java:684)
    at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132)
    at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1939)
    at java.base/java.util.stream.ReferencePipeline$Head.forEachOrdered(ReferencePipeline.java:772)
    ...
Caused by: java.util.zip.ZipException: Corrupt GZIP trailer
    at java.base/java.util.zip.GZIPInputStream.readTrailer(GZIPInputStream.java:257)
    at java.base/java.util.zip.GZIPInputStream.read(GZIPInputStream.java:151)
    at org.jboss.resteasy.plugins.interceptors.encoding.GZIPDecodingInterceptor$FinishableGZIPInputStream.read(GZIPDecodingInterceptor.java:74)
    at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:350)
    at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:393)
    at java.base/sun.nio.cs.StreamDecoder.lockedRead(StreamDecoder.java:217)
    at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:171)
    at java.base/java.io.InputStreamReader.read(InputStreamReader.java:188)
    at java.base/java.io.BufferedReader.fill(BufferedReader.java:160)
    at java.base/java.io.BufferedReader.implReadLine(BufferedReader.java:370)
    at java.base/java.io.BufferedReader.readLine(BufferedReader.java:347)
    at java.base/java.io.BufferedReader.readLine(BufferedReader.java:436)
    at java.base/java.io.BufferedReader$1.hasNext(BufferedReader.java:681)
    ...

The outer exception is dispatched by RESTEasy's Netty4 integration:

org.jboss.resteasy.spi.UnhandledException: java.io.UncheckedIOException: java.util.zip.ZipException: Corrupt GZIP trailer
    at org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:344)
    ...
    at org.jboss.resteasy.plugins.server.netty.RequestHandler.channelRead0(RequestHandler.java:51)
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    ...

Request processing flow

  1. Client sends HTTP POST with Content-Encoding: gzip header and a GZIP-compressed JSON body
  2. Netty receives the request through its pipeline (HttpServerCodecHttpObjectAggregator)
  3. RESTEasy Netty4 adapter (resteasy-netty4) reads the request body from the Netty ByteBuf as an InputStream
  4. A JAX-RS ContainerRequestFilter wraps the entity stream with GZIPInputStream for decompression
  5. A downstream filter reads the decompressed body via BufferedReader.lines()
  6. GZIPInputStream.readTrailer() fails with Corrupt GZIP trailer — the last 8 bytes (CRC32 + ISIZE) are corrupted

Key observations

Observation Detail
Affected payload size ≥ 1MB uncompressed (compressed size ~200KB+)
Unaffected payload size < 1MB uncompressed
Frequency Intermittent under production load, not reproducible on localhost
4.2.9.Final Works correctly — zero errors
4.2.11.Final Consistently produces errors for large payloads
Client payload Verified correct — same clients, same payloads, same compression logic
Increasing decompression buffer size No effect (tried up to 12MB)
Reverting to 4.2.9.Final Immediately resolves the issue

Suspected root cause

The following PRs merged between 4.2.9 and 4.2.11 changed buffer handling and are the most likely cause:

PR Change Milestone
#15736 AdaptiveByteBuf.setBytes optimization — new absolutePut path 4.2.10
#16058 Allocation-free AdaptiveByteBuf.setBytes(byte[]) using root parent NIO buffer 4.2.10
#15741 Adaptive allocator thread-local performance overhaul 4.2.11
#16053 Buddy allocation for large buffers 4.2.11

These changes affect how Netty copies data internally within AdaptiveByteBuf. If there is an off-by-one error or offset miscalculation in the setBytes/absolutePut fast path, the last bytes of the GZIP stream (the 8-byte trailer containing CRC32 and ISIZE) could be silently corrupted as the request body passes through Netty's HttpObjectAggregator pipeline — before it reaches the application layer.

The fact that:

  • Only large payloads are affected (requiring multiple buffer allocations/copies)
  • The corruption is in the GZIP trailer (the last 8 bytes of the stream)
  • The issue is intermittent (depends on buffer allocation patterns)
  • It does not reproduce on localhost (single-buffer pass, no real TCP fragmentation)

...strongly suggests a buffer boundary/offset issue in the adaptive allocator's large buffer handling path.

Minimal reproduction context

The issue occurs when:

  1. A Netty-based HTTP server receives a GZIP-compressed POST body ≥ 3MB uncompressed
  2. The body is read from the ByteBuf as an InputStream (e.g., via ByteBufInputStream)
  3. The InputStream is wrapped with java.util.zip.GZIPInputStream for decompression
  4. The decompressed content is read lazily (e.g., via BufferedReader.lines())
  5. Under real network conditions (TCP fragmentation, multiple Netty read cycles)

We were unable to reproduce in a standalone unit test on localhost because the data arrives in a single contiguous buffer, bypassing the multi-chunk aggregation path where the corruption occurs.

Please check and share your comments on this observation

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions