Repository navigation
Commit be71c25
[fix](catalog) catalog-SPI abstraction-mismatch review: paimon time-travel crash, partition/schema fidelity, SPI convergence & dead-code cleanup (#65970)
### What problem does this PR solve?
Issue Number: #65185
Related PR: #65893, #65733, #65502, #65473, #65329
Problem Summary:
Part of the catalog-SPI migration tracked in #65185, which decouples the
built-in
connectors (hive / iceberg / hudi / paimon) from FE core into loadable
plugins.
An independent clean-room review of the migrated catalog SPI raised 28
abstraction-mismatch
findings (data/predicate/schema/partition/stats/write granularity). Each
was re-verified
against the current tree in a two-stage adversarial pass (independent
investigation →
red-team). Most collapsed on inspection: 2 were refuted (#5, #17), 4
were already fixed
upstream (#3, #4, #6, #28-core), and the bulk of the rest were dead
code, intentional
legacy parity, or estimate-only skew. This PR lands the genuinely
actionable subset, plus
a few adjacent connector runtime/build fixes discovered alongside. The
verification
workspace (the survey, seven per-category analyses, the rollup, and
per-fix design/summary
notes) is included under `plan-doc/catalog-spi-mismatch/`.
Each fix is a standalone commit; connector changes are connector-local
(fe-core keeps
shrinking, per the migration's "fe-core only sheds source-specific code"
rule).
**1. Correctness — user-visible fixes**
- **paimon time-travel read across a rename no longer crashes BE (#2).**
For a
plugin-driven read that pins a historical MVCC snapshot with a
*distinct* schema
(`FOR VERSION/TIME AS OF`, `@branch/@tag`),
`PluginDrivenScanNode.buildColumnHandles`
now resolves the connector column handles at the *pinned* schema via a
new opt-in
`ConnectorTableOps.getColumnHandles(session, handle,
ConnectorMvccSnapshot)` overload
(capability `supportsColumnHandleSnapshotPin`, default false,
implemented only by
paimon). Previously a mixed projection (a surviving column plus a
renamed one) resolved
handles against the *latest* schema, silently dropped the renamed
column's handle, and
still sent it to BE as a scan slot absent from paimon's field-id
dictionary — a BE
`StructNode` `std::out_of_range`/SIGABRT. This is a **crash fix**, not a
wrong-results
fix. iceberg was already immune (it rebuilds its field-id dictionary
from the full
pinned schema) and is untouched. A capability-gated fail-loud guard now
throws a
`UserException` (surfaced through the split-assignment error channel) if
a bound column
is present in the pinned schema but still lacks a handle, instead of
degrading into a BE
crash.
- **hudi decimal partition predicate no longer prunes matching rows
(#1).**
`HudiConnectorMetadata.extractLiteralValue` now renders a `BigDecimal`
literal via
`stripTrailingZeros().toPlainString()` (gated on `instanceof
BigDecimal`; all other
literals unchanged). Before, a `decimal(8,4)` predicate rendered as
`"1.0000"` and never
string-equalled the Hive-canonical stored partition value `"1"`, so the
partition was
pruned and its rows silently dropped from the result. Mirrors the
rendering hive already
uses.
- **paimon `partition_values()` TVF handles DATE/null (#14).**
`PaimonConnectorMetadata` now puts the already-rendered spec (DATE
formatted via
`DateTimeUtils.formatDate`, genuine null normalized to the Hive
default-partition
sentinel) into the `ConnectorPartitionInfo` value map, aligning it with
hive/iceberg
which already carry decoded canonical strings. Before, the raw paimon
spec carried DATE
as an epoch-day string, which the active TVF feeder parsed and threw on
— failing the
whole `partition_values()` query — and rendered null as the internal
sentinel string
rather than SQL NULL.
**2. CBO estimate accuracy — plans only, results unchanged (#15)**
- Time-travel row-count is now estimated at the pinned snapshot instead
of the latest
table. `ConnectorStatisticsOps` gains an opt-in
`getTableStatistics(session, handle, ConnectorMvccSnapshot)` overload;
`StatementContext.getVersionedSnapshot` exposes the single versioned pin
for a table;
`PluginDrivenExternalTable.getRowCount` computes the count at that
snapshot (bypassing
and not polluting the latest-keyed cross-statement cache) and falls back
to the cached
latest estimate when a table cannot count at the snapshot or there is no
statement
context. iceberg reads the pinned snapshot's summary
(`table.snapshot(id)` vs
`currentSnapshot()`); paimon sums the pinned split row counts. This only
affects the
cardinality fed to join reorder / build-side selection; query results
were correct before
and after.
**3. User-signed-off enhancements (#10, #12, #16)**
- **iceberg column write default (#10).**
`IcebergConnectorMetadata.parseSchema` now
carries `field.writeDefault()` through the new
`IcebergSchemaUtils.writeDefaultToDorisString` (reusing the read-side
human-string form;
returns null for missing defaults, complex, and binary-like types). A v3
write default
now survives a schema refresh, so `DESCRIBE` shows it and an INSERT that
omits the column
applies it instead of writing NULL. This only populates FE `Column`
metadata and is
orthogonal to the read-default (initialDefault) BE path from #65502.
- **paimon nested STRUCT field comment (#12).** A `COMMENT` on a field
nested inside a
STRUCT column is now carried through both the paimon write path (4-arg
`DataField`) and
read path (`ConnectorType.structOf(..., comments)`), and the shared
`ConnectorColumnConverter.convertStructType` now threads per-field
comment and
nullability (connectors that leave them unset are byte-identical to
before, so only
paimon changes). The nested comment is visible via `SHOW CREATE TABLE`
and the on-disk
`$schemas.fields`; `DESCRIBE` omits nested-field comments for every
table by design, so
it is not asserted there.
- **SqlCache quiet-window gate uses a real wall clock (#16).**
`CacheAnalyzer` now computes
the quiet-window as `now - max(latestPartitionUpdateMillis)` sourced
from a new
wall-clock accessor
(`MTMVRelatedTableIf.getNewestUpdateTimeMillisForCache`), separate
from the raw version token used for BE PCache staleness. iceberg fills
it from
`last_updated_at` micros/1000 while its monotonic version marker stays
in micros
(`ConnectorMvccPartitionView` renamed to drop the misleading `Millis`
unit; pure rename,
no behavior change). Before, iceberg's ~1.7e15-micros token clamped
`now` via `Math.max`
and permanently suppressed SqlCache for a quiet iceberg table (and any
query joining one).
This is a **cache-eligibility (availability) fix**; staleness is still
guarded
independently by the unchanged version-token equality check.
**4. Connector-SPI shape convergence — refactor, no behavior change (#7,
#8, #9)**
- **#8** lifts source-specific methods off the shared transaction
contracts into narrow
opt-in capability interfaces: `RewriteCapableTransaction` (iceberg
compaction rewrite)
and `WriteBlockAllocatingConnectorTransaction` (maxcompute block
allocation) replace the
methods previously declared on `ConnectorTransaction`, and the
maxcompute write-block
concept is removed from the generic fe-core `Transaction` interface
(new `WriteBlockAllocatingTransaction extends Transaction`). Call sites
(`FrontendServiceImpl`, `ConnectorRewriteDriver`,
`PluginDrivenTransactionManager`) now
`instanceof`-gate the capability. Transaction-id granularity,
commit/rollback,
update-count, and every generic method are unchanged; the user-facing
error strings are
byte-identical.
- **#7 / #9** are javadoc-only corrections: narrow
`ConnectorWriteHandle.getWriteContext`
to "the static partition spec" (its only producer), and rewrite the
`ConnectorBucketSpec`
javadoc to drop the dead `iceberg_bucket`/`hive_hash` algorithm names
and document the
real table-level `DISTRIBUTE BY` values consumed by hive. No code
change.
**5. Dead-code & comment cleanup — no behavior change (#21, #23, #25,
#28)**
- Remove the never-constructed, never-read `ConnectorDeleteFile` (#21)
and
`ConnectorDomain`/`ConnectorRange` pushdown abstraction (#23) from
connector-api, with
their vestigial accessors (`ConnectorScanRange.getDeleteFiles`,
`ConnectorFilterConstraint.getColumnDomains`); iceberg merge-on-read
deletes use their
own untouched path.
- Add `equals`/`hashCode`/`toString` to `ConnectorMvccSnapshot` for
parity with its
value-object siblings (#25; additive, no current Map/Set-key usage).
- Reword stale "dormant until SPI_READY_TYPES" comments in the hive
connector now that the
HMS→SPI flip is complete (#28; comment text only, verified no
non-comment lines changed).
**6. Adjacent connector runtime/build fixes**
- **paimon-scanner classpath**: bundle `hive-common` +
`hive-shims-common` (pinned to
`${hive.version}`) and `hive-standalone-metastore` into the
paimon-scanner assembly so
HMS-backed Paimon reads link — fixing BE `NoClassDefFoundError` on
`org/apache/hadoop/hive/conf/HiveConf` and
`.../metastore/api/NoSuchObjectException`
after #65733 slimmed the shared BE classpath. Build/packaging only.
- **preload-extensions classpath**: restore `net.java.dev.jna:jna` /
`jna-platform`
`5.13.0`, fixing all trino-connector cases failing with
`NoSuchMethodError com.sun.jna.Native.load` (oshi/Udev) after fe-common
went trino-free
and the older hadoop-hdfs jna won the parent-first classpath.
Build/packaging only.
- **iceberg flat MODIFY COLUMN**: a `MODIFY COLUMN` that omits `COMMENT`
now preserves the
field's existing doc instead of clearing it (threads a
`commentSpecified` flag, parity
with the nested path / legacy ops), and a complex→primitive type change
is now rejected
with a clean `Modify column type from complex to primitive is not
supported` message
instead of leaking a raw iceberg type-diff error (symmetric with the
existing
primitive→complex guard; the operation was already unsupported).
**7. Review workspace (docs, `plan-doc/`)**
- Add `plan-doc/catalog-spi-mismatch/`: the abstraction-mismatch survey,
seven per-category
analyses (A–G), the verification rollup, HANDOFF/TASKLIST, and per-fix
design/summary
notes. `plan-doc/` is already tracked on this migration branch.
Findings verified and intentionally **not** changed (refuted,
already-fixed upstream, or
intentional legacy parity — e.g.
#3/#4/#5/#6/#11/#13/#17/#18/#19/#22/#24) are recorded with
evidence in `plan-doc/catalog-spi-mismatch/analysis-00-rollup.md`. The
`ReaderType`
unification (#27) is left as an independent design task.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. <!-- Explain the behavior change -->
- paimon time-travel mixed-projection read across a rename no longer
crashes BE (#2).
- hudi decimal partition predicate no longer prunes matching rows (#1).
- paimon `partition_values()` TVF handles DATE/null (#14).
- iceberg column write default is surfaced for DESCRIBE and
INSERT-omitted fill (#10).
- paimon nested STRUCT field comment carried through create/read (SHOW
CREATE TABLE) (#12).
- iceberg tables become SqlCache-eligible (availability; staleness still
guarded) (#16).
- iceberg flat MODIFY COLUMN preserves an omitted comment / rejects
complex→primitive.
- Time-travel CBO row-count estimate is snapshot-accurate (plans only;
results unchanged) (#15).
- Refactor/cleanup items (#7/#8/#9/#21/#23/#25/#28) and the classpath
fixes carry no logic change.
- Does this need documentation?
- [x] No.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
---------
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>1 parent f9cbd1b commit be71c25
103 files changed
Lines changed: 4284 additions & 712 deletions
File tree
- fe
- be-java-extensions
- paimon-scanner
- preload-extensions
- fe-connector
- fe-connector-api/src
- main/java/org/apache/doris/connector/api
- ddl
- handle
- mvcc
- pushdown
- scan
- test/java/org/apache/doris/connector/api/mvcc
- fe-connector-hive/src
- main/java/org/apache/doris/connector/hive
- test/java/org/apache/doris/connector/hive
- fe-connector-hms/src/main/java/org/apache/doris/connector/hms
- fe-connector-hudi/src
- main/java/org/apache/doris/connector/hudi
- test/java/org/apache/doris/connector/hudi
- fe-connector-iceberg/src
- main/java/org/apache/doris/connector/iceberg
- test/java/org/apache/doris/connector/iceberg
- fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute
- fe-connector-paimon/src
- main/java/org/apache/doris/connector/paimon
- test/java/org/apache/doris/connector/paimon
- fe-core/src
- main/java/org/apache/doris
- datasource
- connector/converter
- mvcc
- plugin
- scan
- mtmv
- nereids
- trees/plans/commands/execute
- qe/cache
- service
- transaction
- test/java/org/apache/doris
- connector/fake
- datasource
- connector/converter
- mvcc
- qe/cache
- service
- transaction
- plan-doc/catalog-spi-mismatch
- regression-test/suites/external_table_p0
- iceberg
- paimon
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
56 | 56 | | |
57 | 57 | | |
58 | 58 | | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
59 | 124 | | |
60 | 125 | | |
61 | 126 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
74 | 74 | | |
75 | 75 | | |
76 | 76 | | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
77 | 99 | | |
78 | 100 | | |
79 | 101 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| 21 | + | |
21 | 22 | | |
22 | 23 | | |
23 | 24 | | |
| |||
35 | 36 | | |
36 | 37 | | |
37 | 38 | | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
38 | 59 | | |
39 | 60 | | |
40 | 61 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
136 | 136 | | |
137 | 137 | | |
138 | 138 | | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
139 | 176 | | |
140 | 177 | | |
141 | 178 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
25 | | - | |
26 | | - | |
| 25 | + | |
| 26 | + | |
27 | 27 | | |
28 | | - | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
29 | 36 | | |
30 | | - | |
31 | | - | |
32 | | - | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
33 | 40 | | |
34 | 41 | | |
35 | 42 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
23 | | - | |
24 | 23 | | |
25 | 24 | | |
26 | 25 | | |
| |||
70 | 69 | | |
71 | 70 | | |
72 | 71 | | |
73 | | - | |
74 | | - | |
75 | | - | |
76 | | - | |
77 | | - | |
78 | | - | |
79 | | - | |
80 | | - | |
81 | | - | |
82 | | - | |
83 | | - | |
84 | | - | |
85 | | - | |
86 | | - | |
87 | | - | |
88 | | - | |
89 | | - | |
90 | | - | |
91 | | - | |
92 | | - | |
93 | | - | |
94 | | - | |
95 | | - | |
96 | | - | |
97 | | - | |
98 | 72 | | |
99 | 73 | | |
100 | 74 | | |
| |||
127 | 101 | | |
128 | 102 | | |
129 | 103 | | |
130 | | - | |
131 | | - | |
132 | | - | |
133 | | - | |
134 | | - | |
135 | | - | |
136 | | - | |
137 | | - | |
138 | | - | |
139 | | - | |
140 | | - | |
141 | | - | |
142 | | - | |
143 | | - | |
144 | | - | |
145 | | - | |
146 | | - | |
147 | | - | |
148 | | - | |
149 | | - | |
150 | | - | |
151 | | - | |
152 | | - | |
153 | | - | |
154 | | - | |
155 | | - | |
156 | | - | |
157 | 104 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
32 | | - | |
33 | | - | |
34 | | - | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
| |||
45 | 45 | | |
46 | 46 | | |
47 | 47 | | |
48 | | - | |
49 | | - | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
50 | 55 | | |
51 | 56 | | |
52 | 57 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
0 commit comments