[feature](paimon) support read paimon blob type by zhangstar333 · Pull Request #69658 · apache/doris

zhangstar333 · 2026-10-27T03:11:07Z

What problem does this PR solve?

Problem Summary:

mysql> desc blob_with_id;
+-------+--------+------+------+---------+-------+
| Field | Type   | Null | Key  | Default | Extra |
+-------+--------+------+------+---------+-------+
| id    | bigint | Yes  | true | NULL    |       |
| image | FILE   | Yes  | true | NULL    |       |
+-------+--------+------+------+---------+-------+
2 rows in set (0.00 sec)

mysql> select * from blob_with_id;
+------+--------------------------------------------------------------------------------------------+
| id   | image                                                                                      |
+------+--------------------------------------------------------------------------------------------+
|    1 | {"uri":null,"offset":null,"size":null,"content_type":null,"checksum":null,"inline":"AQID"} |
|    2 | {"uri":null,"offset":null,"size":null,"content_type":null,"checksum":null,"inline":"BAUG"} |
+------+--------------------------------------------------------------------------------------------+
2 rows in set (0.62 sec)

mysql> select * from blob_with_id@options("blob-as-descriptor"="true"); // return blob descriptor without data
+------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id   | image                                                                                                                                                                                                      |
+------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|    1 | {"uri":"file:/mnt/disk2/zhangsida/test_paimon/test_blob.db/blob_with_id/bucket-0/data-c94d7440-e3f3-49c9-8735-3e0dbf96e1df-1.blob","offset":4,"size":3,"content_type":null,"checksum":null,"inline":null}  |
|    2 | {"uri":"file:/mnt/disk2/zhangsida/test_paimon/test_blob.db/blob_with_id/bucket-0/data-c94d7440-e3f3-49c9-8735-3e0dbf96e1df-1.blob","offset":23,"size":3,"content_type":null,"checksum":null,"inline":null} |
+------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
2 rows in set (0.47 sec)

mysql> select cast(fl_get_binary(image) as string) from blob_table;
+--------------------------------------------+
| cast(fl_get_binary(image) as string) |
+--------------------------------------------+
| h                                          |
| hello world                                |
+--------------------------------------------+
2 rows in set (0.68 sec)


mysql> set paimon_blob_max_size = 2; default is 10M
Query OK, 0 rows affected (0.00 sec)

mysql> select id,cast(fl_get_binary(image) as string) from blob_table;
ERROR 1105 (HY000): errCode = 2, detailMessage = (10.26.20.3)[INVALID_ARGUMENT]Paimon BLOB column 'image' inline size 11 bytes exceeds paimon_blob_max_size=2 bytes. Increase paimon_blob_max_size or use blob-as-descriptor=true
mysql>

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

### What problem does this PR solve? Problem Summary: ``` mysql> desc blob_with_id; +-------+--------+------+------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-------+--------+------+------+---------+-------+ | id | bigint | Yes | true | NULL | | | image | FILE | Yes | true | NULL | | +-------+--------+------+------+---------+-------+ 2 rows in set (0.00 sec) mysql> select * from blob_with_id; +------+--------------------------------------------------------------------------------------------+ | id | image | +------+--------------------------------------------------------------------------------------------+ | 1 | {"uri":null,"offset":null,"size":null,"content_type":null,"checksum":null,"inline":"AQID"} | | 2 | {"uri":null,"offset":null,"size":null,"content_type":null,"checksum":null,"inline":"BAUG"} | +------+--------------------------------------------------------------------------------------------+ 2 rows in set (0.62 sec) mysql> select * from blob_with_id@options("blob-as-descriptor"="true"); // return blob descriptor without data +------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | id | image | +------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 1 | {"uri":"file:/mnt/disk2/zhangsida/test_paimon/test_blob.db/blob_with_id/bucket-0/data-c94d7440-e3f3-49c9-8735-3e0dbf96e1df-1.blob","offset":4,"size":3,"content_type":null,"checksum":null,"inline":null} | | 2 | {"uri":"file:/mnt/disk2/zhangsida/test_paimon/test_blob.db/blob_with_id/bucket-0/data-c94d7440-e3f3-49c9-8735-3e0dbf96e1df-1.blob","offset":23,"size":3,"content_type":null,"checksum":null,"inline":null} | +------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2 rows in set (0.47 sec) mysql> select cast(element_at(image,'inline') as string) from blob_table; +--------------------------------------------+ | cast(element_at(image,'inline') as string) | +--------------------------------------------+ | h | | hello world | +--------------------------------------------+ 2 rows in set (0.68 sec) mysql> set paimon_blob_max_size = 2; default is 10M Query OK, 0 rows affected (0.00 sec) mysql> select id,cast(element_at(image,'inline') as string) from blob_table; ERROR 1105 (HY000): errCode = 2, detailMessage = (10.26.20.3)[INVALID_ARGUMENT]Paimon BLOB column 'image' inline size 11 bytes exceeds paimon_blob_max_size=2 bytes. Increase paimon_blob_max_size or use blob-as-descriptor=true mysql> ``` ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

hello-stephen · 2026-10-27T03:11:12Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

zhangstar333 · 2026-10-27T03:11:32Z

/review

github-actions

Static review of all 48 changed paths at 9b03e19 found two P2 issues: JNI descriptor/null/empty FILE values leak native allocations, and real Parquet ROW values lose the FileIO needed for decoding. Both findings are inline. There are no existing inline findings to retain or duplicate, and no additional user-provided focus.

Critical checkpoints:

Goal and proof: Paimon BLOB maps to the canonical six-child FILE through native managed companions or JNI. Most intended paths are implemented; the two findings leave allocation cleanup and nested interoperability incomplete. Added tests were inspected, not executed.
Scope and clarity: Physical BLOB framing, per-column lookup, table materialization, and JNI transport are separated; changes remain focused on this feature.
Concurrency and locks: New state is scanner/split-local, cached table options are copied, and no new threads, locks, lock ordering, or heavy locked work were introduced.
Lifecycle and initialization: Native readers own copied metadata and release per-split files/indexes; Java buffers are copied before release. The zero-length allocation exception is reported inline. No cross-translation-unit static initialization dependency was found.
Configuration: paimon_blob_max_size is a session setting, defaults to 10 MiB, validates 1..Integer.MAX_VALUE, and reaches both reader paths per query without restart. Descriptor output is exempt.
Compatibility: The reader consumes Paimon 1.4.2 framing without changing stored data. The new-FE/old-BE concern was dismissed because the supported upgrade order requires all BEs before FEs. See the Doris upgrade procedure.
Thrift: TQueryOptions and TPaimonFileDesc definitions were checked on HEAD, master, branch-4.2, branch-4.1, branch-4.0, and branch-3.1. New IDs 237 and 21-23 are unused and optional; older preexisting ID differences are outside this PR.
Parallel paths: Native/JNI, V1 rejection, Rust exclusion for projected FILE, fallback/system tables, Parquet versus ORC anchors, and nested projections were traced. The real nested JNI representation is the second finding.
Special conditions: Native eligibility excludes unsupported merge/layout/delete/external-storage combinations. Full range validation distinguishes missing nullable fields from holes; inline limits precede payload reads, while descriptor mode reads metadata only.
Test coverage: BE framing/corruption/ownership/range tests, Java ABI/projection/limit tests, and Spark-backed native/JNI regressions cover many NULL, empty, binary, filtering, schema, snapshot, compaction and rollover cases. Production native allocator cleanup and actual vectorized ROW fixtures are missing for the reported paths.
Expected results: Both golden files and fixture SQL were inspected; static payload length/hash and rollover NULL-count arithmetic agreed. SQL results and golden-generation provenance were not independently executed or established.
Observability: The materialization timer, tracing-reader footer/index/payload accounting, and contextual stream errors preserve useful attribution; no additional logging/metric defect was established.
Transactions and persistence: No EditLog, transaction commit, master-failover, or persistence-writing logic changes. Snapshot-selected DataSplit metadata remains authoritative.
Writes and crash behavior: No new Doris data-write or atomic publication path is introduced. Reader cleanup, errors, cancellation, split reset and EOF were inspected; the reported native allocation leak survives cleanup.
FE/BE variables: The size option is forwarded through query options and JNI scanner parameters and read directly by native materialization; no missing active read-path handoff was found.
Performance and memory: Native indexes/readers are reused and unprojected companions open lazily; large native buffers use tracked PODArray. The JNI zero-length ownership bug is actionable. No performance measurements were run.
Other correctness checks: Original file positions survive filtering/splits; FILE predicates wait for final materialization; virtual BLOB mappings cannot take COUNT(column)/MIN/MAX metadata shortcuts; SDK range/bunch selection and historical field IDs preserve the accepted native layouts.
Remaining limits: Static inspection only: no builds, tests, benchmarks, source edits, or submodule initialization. No unresolved review candidate or unreviewed changed path remains; runtime behavior and performance are unvalidated.

Review converged after 2 round(s).

zhangstar333 · 2026-10-27T03:41:20Z

run buildall

zhangstar333 · 2026-10-27T07:45:32Z

run buildall

hello-stephen · 2026-10-27T10:02:35Z

BE UT Coverage Report

Increment line coverage 87.42% (396/453) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	61.81% (27287/44147)
Line Coverage	46.50% (284689/612272)
Region Coverage	42.13% (225426/535094)
Branch Coverage	43.72% (105053/240312)

hello-stephen · 2026-10-27T10:41:36Z

FE UT Coverage Report

Increment line coverage 12.08% (18/149) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-10-27T11:32:21Z

BE Regression && UT Coverage Report

Increment line coverage 87.42% (396/453) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	75.29% (32326/42934)
Line Coverage	59.73% (362800/607377)
Region Coverage	56.35% (302098/536145)
Branch Coverage	57.38% (137713/240009)

hello-stephen · 2026-10-27T11:44:45Z

FE Regression Coverage Report

Increment line coverage 22.16% (115/519) 🎉
Increment coverage report
Complete coverage report

### What problem does this PR solve? Issue Number: None Related PR: apache#69658 Problem Summary: Reserve the Paimon BLOB descriptor fields from branch-4.1 on master so future changes do not reuse their field IDs. Add TPaimonBlobFileDesc and TPaimonFileDesc fields 21-23 with the same types and optional semantics as branch-4.1. Reserve TQueryOptions.238 for paimon_blob_max_size with its 10 MiB default. Field 237 already means spill_analytic_sink_mem_limit_bytes on master; the matching branch-4.1 change moves paimon_blob_max_size to 238. This reserves protocol definitions without enabling Paimon BLOB support. ### Release note None ### Check List (For Author) - Test: Manual test: C++ and Java Thrift generation passed; field IDs, types and optional semantics checked against branch-3.1, branch-4.0, branch-4.1 and branch-4.2; git diff --check passed. No BE/FE build or runtime tests for this protocol reservation. - Behavior changed: No - Does this need documentation: No

### What problem does this PR solve? Issue Number: None Related PR: apache#69658, apache#68500, apache#61325, apache#65553 Problem Summary: Release branches have been allocating thrift ids and enum values on their own, and those ids later collide with master. Comparing branch-4.0/4.1/4.2/master shows, for example: - TQueryOptions 237 is paimon_blob_max_size on branch-4.1 (apache#69658, merged directly to branch-4.1) and spill_analytic_sink_mem_limit_bytes on master (apache#68500); both are i64, so a mixed cluster silently reads one as the other. - TRLTaskTxnCommitAttachment 12 is firstErrorMsg on 4.0/4.1/4.2 but kinesisRLTaskProgress on master, because the backport of apache#65553 renumbered firstErrorMsg into the gap left by the unpicked apache#61325. - TSchemaTableType values 62-72 differ between 4.0, 4.1/4.2 and master. The existing rule only says an id must mean the same thing everywhere; it does not say where an id is allocated, so each branch picks the next free number. This adds the rule to gensrc/AGENTS.md (summarized in the root AGENTS.md and the code-review skill): a new id or enum value is added on master first, a release-only feature still reserves its id on master, picks keep master's numbers, enum values are written explicitly, and a release-branch PR that adds an id master does not have is not mergeable. ### Release note None ### Check List (For Author) - Test: No need to test (documentation-only change) - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <[email protected]>

### What problem does this PR solve? Related PR: apache#69658 Problem Summary: branch-4.1 uses TQueryOptions field 237 for paimon_blob_max_size, while master uses the same i64 field for spill_analytic_sink_mem_limit_bytes. A branch-4.1 FE sending a 32 MiB blob limit silently replaces the intended 64 MiB analytic spill threshold. The branch-4.1 BLOB scan metadata also lacks matching definitions on master. Align the BLOB Thrift contract with branch-4.1: - Keep optional i64 paimon_blob_max_size at field 237 with its 10 MiB default; move the analytic spill option to unused field 238 with its 64 MiB default. Existing analytic accessors and FE setters keep their names. - Add TPaimonBlobFileDesc with optional path, file_size, first_row_id, row_count and field_id at the same field IDs 1 through 5. - Append optional TPaimonFileDesc fields 21 blob_files, 22 anchor_first_row_id and 23 anchor_row_count. Preserve master's existing field 20 original_file_path. These changes align and reserve the protocol definitions. Paimon BLOB materialization and FILE execution support remain separate changes. The anchor fields describe the complete Parquet anchor's logical row range; byte splits share the same origin, rather than using their first returned row. The field assignments were checked against current branch-3.1, branch-4.0, branch-4.1, branch-4.2 and master definitions. The branch-4.1 contract stays unchanged. This replaces the field-1024 approach in closed PR apache#70027. ### Release note Align BLOB query and scan metadata with branch-4.1. Master builds using the previous analytic spill field 237 are not wire compatible with this correction: deploy matching FE/BE builds when replacing those master builds. This is not a guarantee of general cross-version query-plan compatibility. ### Check List (For Author) - Test: Java/C++/Python Thrift code generation; Binary/Compact protocol checks using actual generated branch-4.1 and modified-master types: reproduce the query-option collision, preserve the 32 MiB blob limit in both directions, retain the 64 MiB spill default, round-trip the separate 128 MiB spill limit, and verify the old-master boundary; round-trip all BLOB descriptor members and anchor metadata in both directions, preserve master's field 20, and check missing optional fields. git diff --check; SelectDB revert tree equality. Full FE/BE builds and SQL regression were not run for this IDL-only change. - Behavior changed: Yes. Master field 237 represents the Paimon blob limit; the analytic spill limit uses 238, with unchanged defaults. Optional BLOB scan metadata uses the same IDs and types as branch-4.1. - Does this need documentation: No. The rollout boundary is documented above.

zhangstar333 requested a review from yiguolei as a code owner October 27, 2026 03:11

github-actions Bot reviewed Oct 27, 2026

View reviewed changes

Comment thread ...ava-extensions/paimon-connector/src/main/java/org/apache/doris/paimon/PaimonColumnValue.java Resolved

Comment thread ...-java-extensions/java-common/src/main/java/org/apache/doris/common/jni/vec/VectorColumn.java Resolved

update review

755a5dd

zhangstar333 added 2 commits October 27, 2026 15:40

update case

ecb6fe3

update blob large payload

255630b

yiguolei approved these changes Oct 28, 2026

View reviewed changes

zhangstar333 merged commit da8c623 into apache:branch-4.1 Oct 28, 2026
35 of 36 checks passed

HappenLee mentioned this pull request Nov 3, 2026

[fix](paimon) Align BLOB Thrift fields with branch-4.1 #70073

Open

Navigation Menu

Repository navigation

[feature](paimon) support read paimon blob type - #69658

[feature](paimon) support read paimon blob type#69658
zhangstar333 merged 4 commits into
apache:branch-4.1from
zhangstar333:branch-4.1-blob

zhangstar333 commented Oct 27, 2026 •

edited

Loading

Uh oh!

hello-stephen commented Oct 27, 2026

Uh oh!

zhangstar333 commented Oct 27, 2026

Uh oh!

github-actions Bot left a comment

Uh oh!

Uh oh!

Uh oh!

zhangstar333 commented Oct 27, 2026

Uh oh!

zhangstar333 commented Oct 27, 2026

Uh oh!

hello-stephen commented Oct 27, 2026

Uh oh!

hello-stephen commented Oct 27, 2026

Uh oh!

hello-stephen commented Oct 27, 2026

Uh oh!

hello-stephen commented Oct 27, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

zhangstar333 commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Oct 27, 2026

Uh oh!

zhangstar333 commented Oct 27, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

zhangstar333 commented Oct 27, 2026

Uh oh!

zhangstar333 commented Oct 27, 2026

Uh oh!

hello-stephen commented Oct 27, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Oct 27, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Oct 27, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Oct 27, 2026

FE Regression Coverage Report

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

zhangstar333 commented Oct 27, 2026 •

edited

Loading