Repository navigation
fix(athena): preserve decimal precision with the Arrow backend - #3497
Merged
kukushking merged 4 commits intoOct 25, 2026
Merged
Conversation
kukushking
approved these changes
Oct 25, 2026
kukushking
left a comment
Collaborator
There was a problem hiding this comment.
Thanks, looks good. I restructured tests since it mixed unit and integration tests and replaced unit test monkeypatching with slightly cleaner patch obects.
Contributor
Author
|
Thanks for reviewing and merging this, and for splitting the unit and integration tests. That makes the coverage much clearer. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Feature or Bugfix
Detail
dtype_backend="pyarrow", regular Athena CSV results and managed query results parsedecimal(p,s)columns asdouble[pyarrow]. This can silently change values, for example123456789012345.67to123456789012345.69, or raiseArrowInvalidfor integers outside binary64's exact range.Decimalconverter, then convert them todecimal128(p, s)[pyarrow]using Athena's declared precision and scale. This preserves exact values across chunked reads, nulls, all-null and empty results. Explicitcategoriesrequests still take precedence.double[pyarrow]to Arrow decimal. The defaultnumpy_nullablebehavior, global type mapping, CTAS/UNLOAD reads, cached-result reads and write paths are unchanged.Validation:
read_sql_queryAPI. The tests use the real result metadata and CSV parsing code, with AWS calls and their boundary wrappers mocked. Against the base, 24 fail and 28 pass; all 52 pass with the fix.uv lock --checkpass. Focused mypy on the two changed modules reports the same 10 errors as the base. Doc8 reports the same existing carriage-return errors on both trees.Relates
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.