feat: limit decimal encoding to values allowed by schema #565
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Goal of this PR
When encoding
fixed.decimal, there are 2 possible errors which manifest when attempting to decode:the number of digits exceed the precision specified in the schema -- this module decodes these properly because the precision is ignored, but the official
org.apache.avroJava decoder errors withDECIMAL_PRECISION_EXCEEDS_MAX_PRECISIONthe number of bytes used for encoding exceeds the fixed size specified in the schema -- this shifts subsequent bytes, so the rest of the data cannot be decoded to proper values. The official
org.apache.avroJava decoder errors for various malformed data as it tries to decode the wrong bytes.This PR fixes both of these failure modes by checking the actual precision of the value being encoded for
fixed.decimalandbytes.decimalagainst the precision in the schema, and also the number of bytes being written forfixed.decimal.I also fixed the max precision calculation in
schema_parse.goto match the one in the Avro specification. The original formula was close, but off by one in some cases (e.g. size=3 should allow precision=7 but only allows 6, size=5 should allow 12 but only allows 11, etc).And I had to update some unit tests because
1734/5is346.80, which needsprecision=5to encode ifscale=2.How did I test it?
I added unit tests that expect encoding failure for these cases, which fail without the fixes and pass with the fixes.