Reduce allocations when reading small-precision decimals from ArrowColumnVector

XMLWordPrintableJSON

      ArrowColumnVector.DecimalAccessor converts each non-null Decimal128 value
      through Decimal.apply(vector.getObject(rowId), precision, scale).

      Arrow Java 19.0.0 constructs a temporary byte array and BigInteger when
      materializing the BigDecimal. This also happens for source precision 1–18,
      whose valid unscaled integers fit in a signed long.

      A small-precision accessor can read the native Decimal128 words, check
      sign extension, and construct BigDecimal from the unscaled long and source
      scale. Values that do not fit in a long retain the full-width conversion.
      Spark still applies the requested precision and scale, and the result
      retains BigDecimal-backed Decimal behavior for checked integer casts.

      ArrowCachedBatchSerializer already reads native long words for compact
      decimals when writing directly to UnsafeRow. This change applies the same
      decoding approach to the general ArrowColumnVector interface.

      A prototype based on master 15ec0e55b5d5faa9b7844978eddfec696f95f778,
      using Arrow Java 19.0.0 and Corretto Java 17.0.16, measured:

      • decimal(18,2) -> decimal(18,2): 192 -> 96 allocated bytes per read.
      • decimal(18,2) -> decimal(18,1): 256 -> 160 allocated bytes per read.
      • decimal(38,2) -> decimal(38,2): 200 -> 200 allocated bytes per read.

      The allocation comparison used the original and modified ArrowColumnVector
      classes in separate JVMs, with two runs per version and five measured
      samples per scenario. Input batches contained 4,096 alternating positive
      and negative values; returned Decimal objects were retained in an array.

      Related Vortex connector discussion:
      https://github.com/vortex-data/vortex/issues/9837

            Assignee:
            Peifeng Li
            Reporter:
            Peifeng Li
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: