Repository navigation
branch-4.1:[chore](thirdparty) Upgrade Apache Arrow to 24.0.0(#66221) - #66480
Merged
Gabriel39 merged 4 commits intoAug 6, 2026
Merged
Conversation
### What problem does this PR solve? Issue Number: None Related PR: apache#66221 Problem Summary: Doris used Apache Arrow 17.0.0, which blocked consuming newer Arrow fixes and APIs. Upgrade the C++ third-party dependency to 24.0.0, align xsimd with Arrow 24, port the Paimon row-group, INT96, and Parquet LZO compatibility patches, add Arrow Compute to the static dependency graph, and migrate removed Arrow and Parquet APIs while preserving Decimal128 semantics. ### Release note None ### Check List (For Author) - Test: Manual test - Rebuilt the Arrow and Paimon C++ third-party dependencies from the selected sources and patches - Built ASAN BE with `BUILD_TYPE=ASAN ./build.sh --be -j 16` - Ran `thirdparty/test/arrow-paimon-lifecycle-test.sh` - Ran `build-support/check-format.sh` - Ran changed-line clang-tidy for `arrow_schema_util.cpp` - BE C++ unit test sources were updated but their test targets were not built or run - Behavior changed: No - Does this need documentation: No
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
hubgeter
changed the title
Contributor
Author
|
run buildall |
Contributor
Author
|
run buildall |
### What problem does this PR solve? Issue Number: None Related PR: apache#66221, apache#66480 Problem Summary: The Arrow and Paimon prebuilt fingerprints hashed whole shared build scripts. Unrelated differences between master and branch-4.1 therefore rejected a shared prebuilt even when the selected sources, patches, cache, and build recipe were identical. Compute component fingerprints from explicit semantic inputs and schema versions instead. Accept the current master-generated component markers only for the matching new semantic fingerprints, while retaining exact version and artifact-closure validation. ### Release note None ### Check List (For Author) - Test: Manual test - thirdparty/test/arrow-paimon-lifecycle-test.sh - Bash syntax, ShellCheck, shfmt, and git diff checks - Local Doris compilation was not run per request - Behavior changed: No - Does this need documentation: No
Contributor
Author
|
run buildall |
### What problem does this PR solve? Issue Number: None Related PR: apache#66221, apache#66480 Problem Summary: The branch-4.1 BE UT runner uses the install-only thirdparty tree embedded in the shared build image. That image predates the Arrow 24 component closure and does not contain libarrow_compute.a, while the external TeamCity wrapper skips downloading the shared automation prebuilt for branch-4.1. Validate the build-image Arrow/Paimon installation from run-be-ut.sh and refresh it from the shared automation asset when stale. Downloaded archives are validated before replacing the existing installation. ### Release note None ### Check List (For Author) - Test: Unit Test - bash thirdparty/test/arrow-paimon-lifecycle-test.sh - bash -n and shellcheck for the changed shell scripts - Full BE compilation was not run for this shell-only CI repair - Behavior changed: Yes. Linux x86_64 BE UT in the official build image refreshes an incompatible Arrow/Paimon prebuilt before compiling. - Does this need documentation: No
Contributor
Author
|
run buildall |
### What problem does this PR solve? Issue Number: None Related PR: apache#66221 Problem Summary: Arrow 24 removed ParquetVersion::PARQUET_2_0, causing the BE UT target to fail while compiling parquet_expr_test.cpp. Use PARQUET_2_6, which is supported by Arrow 24 and matches the other Parquet writer tests. ### Release note None ### Check List (For Author) - Test: No local compile or unit test was run per request; build-support/check-format.sh and git diff --check passed. - Behavior changed: No - Does this need documentation: No
Contributor
|
run buildall |
yiguolei
approved these changes
Aug 6, 2026
Contributor
|
PR approved by at least one committer and no changes requested. |
github-actions
Bot
added
the
approved
Contributor
|
PR approved by anyone and no changes requested. |
buu-nguyen
pushed a commit
to thealtoclef/doris
that referenced
this pull request
Aug 28, 2026
The iceberg table-function schema converter called the generic arrow::decimal() factory, which the bundled Arrow headers no longer provide (DecimalType is now backed by the typed decimal128()/decimal256() forms), so source builds of the BE failed in be/src/format/table/iceberg/arrow_schema_util.cpp. Switch to arrow::decimal128() — ICEBERG decimals are always 128-bit. Hunk picked from the upstream Apache Arrow 24.0.0 thirdparty upgrade: branch-4.1 f0e0080 [chore](thirdparty) Upgrade Apache Arrow to 24.0.0 (apache#66480) (backport of main 5baf045 / PR apache#66221) Only this single API-migration line is taken; the full Arrow 17 -> 24 thirdparty bump is NOT part of this commit and 4.1.3 still pins Arrow 17.0.0. On a stock thirdparty build (Arrow 17) arrow::decimal() still exists, so this hunk is only required for source builds against newer bundled Arrow headers; decimal128() itself exists in both versions, making it safe either way. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Closed
16 tasks
buu-nguyen
pushed a commit
to thealtoclef/doris
that referenced
this pull request
Oct 15, 2026
The iceberg table-function schema converter called the generic arrow::decimal() factory, which the bundled Arrow headers no longer provide (DecimalType is now backed by the typed decimal128()/decimal256() forms), so source builds of the BE failed in be/src/format/table/iceberg/arrow_schema_util.cpp. Switch to arrow::decimal128() — ICEBERG decimals are always 128-bit. Hunk picked from the upstream Apache Arrow 24.0.0 thirdparty upgrade: branch-4.1 f0e0080 [chore](thirdparty) Upgrade Apache Arrow to 24.0.0 (apache#66480) (backport of main 5baf045 / PR apache#66221) Only this single API-migration line is taken; the full Arrow 17 -> 24 thirdparty bump is NOT part of this commit and 4.1.3 still pins Arrow 17.0.0. On a stock thirdparty build (Arrow 17) arrow::decimal() still exists, so this hunk is only required for source builds against newer bundled Arrow headers; decimal128() itself exists in both versions, making it safe either way. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
bp #66221
Problem Summary: Doris used Apache Arrow 17.0.0, which blocked consuming newer Arrow fixes and APIs. Upgrade the C++ third-party dependency to 24.0.0, align xsimd with Arrow 24, port the Paimon row-group, INT96, and Parquet LZO compatibility patches, add Arrow Compute to the static dependency graph, and migrate removed Arrow and Parquet APIs while preserving Decimal128 semantics.
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)