Skip to content

Refactor arrow-cast decimal casting to unify the rescale logic used in Parquet variant casts #8670

@liamzwbao

Description

@liamzwbao

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Refactor decimal casting in arrow-cast to converge on the same rescale logic used by the parquet variant cast

Describe the solution you'd like

Originally post by @scovich in #8552 (comment)

  • rework convert_to_smaller_scale_decimal and convert_to_bigger_or_equal_scale_decimal

    • no longer take array or cast_options as input
    • return Ok((f, is_infallible_cast) which corresponds to the return type Result<(impl Fn(I::Native) -> Option<O::Native>, bool), ArrowError>
  • define a new generic apply_decimal_cast function helper

    • it takes as input array, cast_options and the (impl Fn, bool) pair produced by a convert_to_xxx_scale_decimal helper
    • it handles the three ways of applying f to an array
  • rework cast_decimal_to_decimal and cast_decimal_to_decimal_same_type to call those functions (see below)

  • rescale_decimal would be the single-row equivalent of cast_decimal_to_decimal, returning Option<O::Native>

  • The decimal builder's constructor calls validate_decimal_precision_and_scale and fails on error, so we don't need to validate on a per-row basis.

cast_decimal_to_decimal

let array: PrimitiveArray<O> = if input_scale > output_scale {
    let (f, is_infallible_cast) = convert_to_smaller_scale_decimal(...)?;
    apply_decimal_cast(array, cast_options, f, is_infallible)?
} else {
    let (f, is_infallible_cast) = convert_to_bigger_or_equal_scale_decimal(...)?;
    apply_decimal_cast(array, cast_options, f, is_infallible)?
}

rescale_decimal

if input_scale > output_scale {
    let (f, _) = convert_to_smaller_scale_decimal(...)?;
    f(integer)
} else {
    let (f, _) = convert_to_bigger_or_equal_scale_decimal(...)?;
    f(integer)
}

Describe alternatives you've considered

Additional context

Metadata

Metadata

Assignees

Labels

arrowChanges to the arrow crateenhancementAny new improvement worthy of a entry in the changelogparquet-variantparquet-variant* crates

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions