Skip to content

Fix codec pipeline Delta (or DoubleDelta) + Gorilla#45615

Merged
rschu1ze merged 3 commits intomasterfrom
fix-gorilla
Jan 26, 2023
Merged

Fix codec pipeline Delta (or DoubleDelta) + Gorilla#45615
rschu1ze merged 3 commits intomasterfrom
fix-gorilla

Conversation

@rschu1ze
Copy link
Copy Markdown
Member

@rschu1ze rschu1ze commented Jan 25, 2023

Fixes #45195

Codec combinations (Delta, Gorilla) and (DoubleDelta, Gorilla) (and possibly other codec pipelines where Gorilla is not the first codec) corrupt data. In case of Delta, the reason is that the first codec writes a 11 byte header (containing the method, the item width etc.) and then compresses each input item to an equally wide output item. Gorilla then receives an input byte stream with an unaligned number of bytes. There exists code to store the unaligned bytes as is (*) but the code did not include them in the output stream byte size. This led to the situation that up to 7 bytes at the end of the compressed data (in case of double-precision FP values) were not read, creating random decompressed output data.

(*) Gorilla deals with Delta-compressed inputs in a weird way: The delta header is chopped into 3+8 bytes (for double-precision floats), 3 bytes are copied as is into the output, 8 bytes are treated as first floating point value. That aside, Gorilla has its own mechanism of handling deltas between values and the combination Delta/DoubleDelta and Gorilla should be outlawed / made "suspicious". This will be done separately.

(also including some unrelated smaller improvements)

Changelog category (leave one):

  • Bug Fix (user-visible misbehavior in official stable or prestable release)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

  • Fix data corruption when codecs Delta or DoubleDelta are combined with codec Gorilla.

@robot-ch-test-poll2 robot-ch-test-poll2 added the pr-bugfix Pull request with bugfix, not backported by default label Jan 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-bugfix Pull request with bugfix, not backported by default

Projects

None yet

Development

Successfully merging this pull request may close these issues.

data corruption on delta+gorilla+lz4 codec pipeline

3 participants