Implement arrow_json encoder for Decimal128 & Decimal256#6606
Implement arrow_json encoder for Decimal128 & Decimal256#6606tustvold merged 5 commits intoapache:masterfrom
Conversation
This reverts commit dfe8edb.
peasee
left a comment
peasee
left a comment
There was a problem hiding this comment.
Looking good, thanks for working on this!
arrow-json/src/writer/encoder.rs
Outdated
|
|
||
| impl<'a> Encoder for Decimal128Encoder<'a> { | ||
| fn encode(&mut self, idx: usize, out: &mut Vec<u8>) { | ||
| let formatted = self.array.value_as_string(idx); |
There was a problem hiding this comment.
FWIW you can avoid this allocation by using https://docs.rs/arrow-cast/latest/arrow_cast/display/struct.ArrayFormatter.html
There was a problem hiding this comment.
Let me try this
There was a problem hiding this comment.
I tried this, but it encoded them as strings not numbers, and I don't see any option that is relevant.
DataType::Decimal128(_, _) | DataType::Decimal256(_, _)=> {
let options = FormatOptions::new();
let formatter = ArrayFormatter::try_new(array, &options)?;
(Box::new(formatter) as _, array.nulls().cloned())
}assertion `left == right` failed
left: [Some(Object {"decimal": Number(12.34)}), Some(Object {"decimal": Number(56.78)}), Some(Object {"decimal": Number(90.12)}), None]
right: [Some(Object {"decimal": String("12.34")}), Some(Object {"decimal": String("56.78")}), Some(Object {"decimal": String("90.12")}), None]I'm not sure why though, it seems to call the same format_decimal method that value_as_string does?
There was a problem hiding this comment.
Yes you will need to keep the custom encoder as a newtype as unlike the implementation for ArrayFormatter above, you don't want to write the quote characters
3 participants
Which issue does this PR close?
Closes #6605
Rationale for this change
Makes the arrow_json crate able to serialize more record batches.
What changes are included in this PR?
Implements a Decimal128Encoder and Decimal256Encoder and uses that within the
make_encoder_implfunction for the Decimal128 & Decimal256 types.Are there any user-facing changes?
More record batches will be able to be serialized to JSON, and this is backwards compatible.