Skip to content

Commit 1604b80

Browse files
morningmanclaude
andcommitted
[fix](catalog) catalog-SPI abstraction-mismatch review: paimon time-travel crash, partition/schema fidelity, SPI convergence & dead-code cleanup (#65970)
### What problem does this PR solve? Issue Number: #65185 Related PR: #65893, #65733, #65502, #65473, #65329 Problem Summary: Part of the catalog-SPI migration tracked in #65185, which decouples the built-in connectors (hive / iceberg / hudi / paimon) from FE core into loadable plugins. An independent clean-room review of the migrated catalog SPI raised 28 abstraction-mismatch findings (data/predicate/schema/partition/stats/write granularity). Each was re-verified against the current tree in a two-stage adversarial pass (independent investigation → red-team). Most collapsed on inspection: 2 were refuted (#5, #17), 4 were already fixed upstream (#3, #4, #6, #28-core), and the bulk of the rest were dead code, intentional legacy parity, or estimate-only skew. This PR lands the genuinely actionable subset, plus a few adjacent connector runtime/build fixes discovered alongside. The verification workspace (the survey, seven per-category analyses, the rollup, and per-fix design/summary notes) is included under `plan-doc/catalog-spi-mismatch/`. Each fix is a standalone commit; connector changes are connector-local (fe-core keeps shrinking, per the migration's "fe-core only sheds source-specific code" rule). **1. Correctness — user-visible fixes** - **paimon time-travel read across a rename no longer crashes BE (#2).** For a plugin-driven read that pins a historical MVCC snapshot with a *distinct* schema (`FOR VERSION/TIME AS OF`, `@branch/@tag`), `PluginDrivenScanNode.buildColumnHandles` now resolves the connector column handles at the *pinned* schema via a new opt-in `ConnectorTableOps.getColumnHandles(session, handle, ConnectorMvccSnapshot)` overload (capability `supportsColumnHandleSnapshotPin`, default false, implemented only by paimon). Previously a mixed projection (a surviving column plus a renamed one) resolved handles against the *latest* schema, silently dropped the renamed column's handle, and still sent it to BE as a scan slot absent from paimon's field-id dictionary — a BE `StructNode` `std::out_of_range`/SIGABRT. This is a **crash fix**, not a wrong-results fix. iceberg was already immune (it rebuilds its field-id dictionary from the full pinned schema) and is untouched. A capability-gated fail-loud guard now throws a `UserException` (surfaced through the split-assignment error channel) if a bound column is present in the pinned schema but still lacks a handle, instead of degrading into a BE crash. - **hudi decimal partition predicate no longer prunes matching rows (#1).** `HudiConnectorMetadata.extractLiteralValue` now renders a `BigDecimal` literal via `stripTrailingZeros().toPlainString()` (gated on `instanceof BigDecimal`; all other literals unchanged). Before, a `decimal(8,4)` predicate rendered as `"1.0000"` and never string-equalled the Hive-canonical stored partition value `"1"`, so the partition was pruned and its rows silently dropped from the result. Mirrors the rendering hive already uses. - **paimon `partition_values()` TVF handles DATE/null (#14).** `PaimonConnectorMetadata` now puts the already-rendered spec (DATE formatted via `DateTimeUtils.formatDate`, genuine null normalized to the Hive default-partition sentinel) into the `ConnectorPartitionInfo` value map, aligning it with hive/iceberg which already carry decoded canonical strings. Before, the raw paimon spec carried DATE as an epoch-day string, which the active TVF feeder parsed and threw on — failing the whole `partition_values()` query — and rendered null as the internal sentinel string rather than SQL NULL. **2. CBO estimate accuracy — plans only, results unchanged (#15)** - Time-travel row-count is now estimated at the pinned snapshot instead of the latest table. `ConnectorStatisticsOps` gains an opt-in `getTableStatistics(session, handle, ConnectorMvccSnapshot)` overload; `StatementContext.getVersionedSnapshot` exposes the single versioned pin for a table; `PluginDrivenExternalTable.getRowCount` computes the count at that snapshot (bypassing and not polluting the latest-keyed cross-statement cache) and falls back to the cached latest estimate when a table cannot count at the snapshot or there is no statement context. iceberg reads the pinned snapshot's summary (`table.snapshot(id)` vs `currentSnapshot()`); paimon sums the pinned split row counts. This only affects the cardinality fed to join reorder / build-side selection; query results were correct before and after. **3. User-signed-off enhancements (#10, #12, #16)** - **iceberg column write default (#10).** `IcebergConnectorMetadata.parseSchema` now carries `field.writeDefault()` through the new `IcebergSchemaUtils.writeDefaultToDorisString` (reusing the read-side human-string form; returns null for missing defaults, complex, and binary-like types). A v3 write default now survives a schema refresh, so `DESCRIBE` shows it and an INSERT that omits the column applies it instead of writing NULL. This only populates FE `Column` metadata and is orthogonal to the read-default (initialDefault) BE path from #65502. - **paimon nested STRUCT field comment (#12).** A `COMMENT` on a field nested inside a STRUCT column is now carried through both the paimon write path (4-arg `DataField`) and read path (`ConnectorType.structOf(..., comments)`), and the shared `ConnectorColumnConverter.convertStructType` now threads per-field comment and nullability (connectors that leave them unset are byte-identical to before, so only paimon changes). The nested comment is visible via `SHOW CREATE TABLE` and the on-disk `$schemas.fields`; `DESCRIBE` omits nested-field comments for every table by design, so it is not asserted there. - **SqlCache quiet-window gate uses a real wall clock (#16).** `CacheAnalyzer` now computes the quiet-window as `now - max(latestPartitionUpdateMillis)` sourced from a new wall-clock accessor (`MTMVRelatedTableIf.getNewestUpdateTimeMillisForCache`), separate from the raw version token used for BE PCache staleness. iceberg fills it from `last_updated_at` micros/1000 while its monotonic version marker stays in micros (`ConnectorMvccPartitionView` renamed to drop the misleading `Millis` unit; pure rename, no behavior change). Before, iceberg's ~1.7e15-micros token clamped `now` via `Math.max` and permanently suppressed SqlCache for a quiet iceberg table (and any query joining one). This is a **cache-eligibility (availability) fix**; staleness is still guarded independently by the unchanged version-token equality check. **4. Connector-SPI shape convergence — refactor, no behavior change (#7, #8, #9)** - **#8** lifts source-specific methods off the shared transaction contracts into narrow opt-in capability interfaces: `RewriteCapableTransaction` (iceberg compaction rewrite) and `WriteBlockAllocatingConnectorTransaction` (maxcompute block allocation) replace the methods previously declared on `ConnectorTransaction`, and the maxcompute write-block concept is removed from the generic fe-core `Transaction` interface (new `WriteBlockAllocatingTransaction extends Transaction`). Call sites (`FrontendServiceImpl`, `ConnectorRewriteDriver`, `PluginDrivenTransactionManager`) now `instanceof`-gate the capability. Transaction-id granularity, commit/rollback, update-count, and every generic method are unchanged; the user-facing error strings are byte-identical. - **#7 / #9** are javadoc-only corrections: narrow `ConnectorWriteHandle.getWriteContext` to "the static partition spec" (its only producer), and rewrite the `ConnectorBucketSpec` javadoc to drop the dead `iceberg_bucket`/`hive_hash` algorithm names and document the real table-level `DISTRIBUTE BY` values consumed by hive. No code change. **5. Dead-code & comment cleanup — no behavior change (#21, #23, #25, #28)** - Remove the never-constructed, never-read `ConnectorDeleteFile` (#21) and `ConnectorDomain`/`ConnectorRange` pushdown abstraction (#23) from connector-api, with their vestigial accessors (`ConnectorScanRange.getDeleteFiles`, `ConnectorFilterConstraint.getColumnDomains`); iceberg merge-on-read deletes use their own untouched path. - Add `equals`/`hashCode`/`toString` to `ConnectorMvccSnapshot` for parity with its value-object siblings (#25; additive, no current Map/Set-key usage). - Reword stale "dormant until SPI_READY_TYPES" comments in the hive connector now that the HMS→SPI flip is complete (#28; comment text only, verified no non-comment lines changed). **6. Adjacent connector runtime/build fixes** - **paimon-scanner classpath**: bundle `hive-common` + `hive-shims-common` (pinned to `${hive.version}`) and `hive-standalone-metastore` into the paimon-scanner assembly so HMS-backed Paimon reads link — fixing BE `NoClassDefFoundError` on `org/apache/hadoop/hive/conf/HiveConf` and `.../metastore/api/NoSuchObjectException` after #65733 slimmed the shared BE classpath. Build/packaging only. - **preload-extensions classpath**: restore `net.java.dev.jna:jna` / `jna-platform` `5.13.0`, fixing all trino-connector cases failing with `NoSuchMethodError com.sun.jna.Native.load` (oshi/Udev) after fe-common went trino-free and the older hadoop-hdfs jna won the parent-first classpath. Build/packaging only. - **iceberg flat MODIFY COLUMN**: a `MODIFY COLUMN` that omits `COMMENT` now preserves the field's existing doc instead of clearing it (threads a `commentSpecified` flag, parity with the nested path / legacy ops), and a complex→primitive type change is now rejected with a clean `Modify column type from complex to primitive is not supported` message instead of leaking a raw iceberg type-diff error (symmetric with the existing primitive→complex guard; the operation was already unsupported). **7. Review workspace (docs, `plan-doc/`)** - Add `plan-doc/catalog-spi-mismatch/`: the abstraction-mismatch survey, seven per-category analyses (A–G), the verification rollup, HANDOFF/TASKLIST, and per-fix design/summary notes. `plan-doc/` is already tracked on this migration branch. Findings verified and intentionally **not** changed (refuted, already-fixed upstream, or intentional legacy parity — e.g. #3/#4/#5/#6/#11/#13/#17/#18/#19/#22/#24) are recorded with evidence in `plan-doc/catalog-spi-mismatch/analysis-00-rollup.md`. The `ReaderType` unification (#27) is left as an independent design task. ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [x] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [x] Yes. <!-- Explain the behavior change --> - paimon time-travel mixed-projection read across a rename no longer crashes BE (#2). - hudi decimal partition predicate no longer prunes matching rows (#1). - paimon `partition_values()` TVF handles DATE/null (#14). - iceberg column write default is surfaced for DESCRIBE and INSERT-omitted fill (#10). - paimon nested STRUCT field comment carried through create/read (SHOW CREATE TABLE) (#12). - iceberg tables become SqlCache-eligible (availability; staleness still guarded) (#16). - iceberg flat MODIFY COLUMN preserves an omitted comment / rejects complex→primitive. - Time-travel CBO row-count estimate is snapshot-accurate (plans only; results unchanged) (#15). - Refactor/cleanup items (#7/#8/#9/#21/#23/#25/#28) and the classpath fixes carry no logic change. - Does this need documentation? - [x] No. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> --------- Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
1 parent 741c446 commit 1604b80

103 files changed

Lines changed: 4284 additions & 712 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎fe/be-java-extensions/paimon-scanner/pom.xml‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,71 @@ under the License.
5656
<artifactId>paimon-hive-connector-3.1</artifactId>
5757
</dependency>
5858

59+
<!--
60+
Hive runtime classes required by the Paimon Hive connector on the BE side.
61+
paimon-hive-connector-3.1 marks hive-exec as `provided`, and only bundles
62+
relocated hive classes under org.apache.paimon.shade.*; the real
63+
org.apache.hadoop.hive.conf.HiveConf (referenced by
64+
org.apache.paimon.hive.SerializableHiveConf when a HMS-backed Paimon table
65+
is deserialized, and by HiveCatalog.createHiveConf) is NOT packaged.
66+
It used to be provided accidentally by java-udf's hive-catalog-shade on the
67+
shared BE classpath, but #65733 replaced that with the slim hive-udf-shade,
68+
so paimon-scanner must now carry HiveConf itself. Version is pinned to
69+
${hive.version} (3.1.x) to match the Hive 3.1 the connector was built against.
70+
The transitive trees (jetty/orc/ant/log4j bridges/hadoop/...) are pruned:
71+
HiveConf's construction closure lives entirely inside hive-common, shims/Utils
72+
(used only by the lazy HiveConf#getUser) lives in hive-shims-common, and hadoop
73+
is already on BE's shared classpath.
74+
-->
75+
<dependency>
76+
<groupId>org.apache.hive</groupId>
77+
<artifactId>hive-common</artifactId>
78+
<version>${hive.version}</version>
79+
<exclusions>
80+
<exclusion>
81+
<groupId>*</groupId>
82+
<artifactId>*</artifactId>
83+
</exclusion>
84+
</exclusions>
85+
</dependency>
86+
<dependency>
87+
<groupId>org.apache.hive.shims</groupId>
88+
<artifactId>hive-shims-common</artifactId>
89+
<version>${hive.version}</version>
90+
<exclusions>
91+
<exclusion>
92+
<groupId>*</groupId>
93+
<artifactId>*</artifactId>
94+
</exclusion>
95+
</exclusions>
96+
</dependency>
97+
<!--
98+
For the same reason (hive-catalog-shade no longer on the shared classpath),
99+
paimon-scanner must also carry the Hive metastore API classes. When BE
100+
deserializes an HMS-backed Paimon table, org.apache.paimon.hive.HiveCatalogLoader
101+
links org.apache.paimon.hive.HiveCatalog, which references a broad slice of
102+
org.apache.hadoop.hive.metastore.api.* (Table, Database, Partition,
103+
StorageDescriptor, SerDeInfo, FieldSchema, NoSuchObjectException,
104+
UnknownTableException, ...). In Hive 3.x that whole thrift-generated `api`
105+
package lives in hive-standalone-metastore (the `hive-metastore` artifact is a
106+
thin shim and does NOT contain it). Transitives are pruned like the two jars
107+
above: the api classes only extend org.apache.thrift.* (provided by java-common's
108+
libthrift on the shared classpath, exactly as under the old hive-catalog-shade)
109+
and java stdlib; datanucleus/orc/hadoop/etc. are only needed by the metastore
110+
server/client, which BE never runs (it reads the already-resolved table).
111+
-->
112+
<dependency>
113+
<groupId>org.apache.hive</groupId>
114+
<artifactId>hive-standalone-metastore</artifactId>
115+
<version>${hive.version}</version>
116+
<exclusions>
117+
<exclusion>
118+
<groupId>*</groupId>
119+
<artifactId>*</artifactId>
120+
</exclusion>
121+
</exclusions>
122+
</dependency>
123+
59124
<dependency>
60125
<groupId>org.apache.paimon</groupId>
61126
<artifactId>paimon-format</artifactId>

‎fe/be-java-extensions/preload-extensions/pom.xml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,28 @@ under the License.
7474
<artifactId>commons-lang</artifactId>
7575
<scope>runtime</scope>
7676
</dependency>
77+
<!-- Runtime-only: the BE trino-connector scanner's oshi (pulled via trino-main) initializes
78+
com.sun.jna.platform.linux.Udev, which calls com.sun.jna.Native.load(String, Class) (jna >= 5.0).
79+
JNI scanners load through JniScannerClassLoader, a parent-first URLClassLoader whose parent is the
80+
BE system classpath; that classpath (bin/start_be.sh) places preload-extensions ahead of
81+
lib/hadoop_hdfs, whose bundled jna is < 5.0. jna used to reach this shared preload classpath
82+
transitively via fe-common -> trino-main -> oshi, so the modern jna shadowed the hadoop copy; after
83+
ca840c9dbea made fe-common trino-free it was evicted, and the scanner's own bundled jna cannot win
84+
(parent-first resolves Native from the hadoop jar first) -> NoSuchMethodError. Restore it here (the
85+
shared parent of all JNI scanners), mirroring the commons-lang / fastutil-core restorations. Pin
86+
both core and platform to 5.13.0 (what oshi 6.4.5 pulls and what fe-core pins inline). -->
87+
<dependency>
88+
<groupId>net.java.dev.jna</groupId>
89+
<artifactId>jna</artifactId>
90+
<version>5.13.0</version>
91+
<scope>runtime</scope>
92+
</dependency>
93+
<dependency>
94+
<groupId>net.java.dev.jna</groupId>
95+
<artifactId>jna-platform</artifactId>
96+
<version>5.13.0</version>
97+
<scope>runtime</scope>
98+
</dependency>
7799
<dependency>
78100
<groupId>org.apache.arrow</groupId>
79101
<artifactId>arrow-memory-unsafe</artifactId>

‎fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorStatisticsOps.java‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
package org.apache.doris.connector.api;
1919

2020
import org.apache.doris.connector.api.handle.ConnectorTableHandle;
21+
import org.apache.doris.connector.api.mvcc.ConnectorMvccSnapshot;
2122

2223
import java.util.Collections;
2324
import java.util.List;
@@ -35,6 +36,26 @@ default Optional<ConnectorTableStatistics> getTableStatistics(
3536
return Optional.empty();
3637
}
3738

39+
/**
40+
* Returns table statistics AS OF {@code snapshot} &mdash; the row count at the pinned snapshot, for a
41+
* time-travel read (FOR VERSION/TIME AS OF, {@code @branch}/{@code @tag}).
42+
*
43+
* <p>The default ignores the snapshot and returns the latest statistics via
44+
* {@link #getTableStatistics(ConnectorSession, ConnectorTableHandle)}. WHY this exists: the table-level
45+
* row count feeds the CBO. Without a snapshot dimension the optimizer estimates a time-travel query's
46+
* cardinality from the LATEST snapshot while the scan reads the pinned one, skewing join reorder / build
47+
* side selection (estimate-only &mdash; results stay correct). A connector that can count at a snapshot
48+
* overrides this (mirrors the {@code getTableSchema}/{@code getColumnHandles} snapshot overloads). fe-core
49+
* only calls it for a genuine versioned read, so the shared latest-keyed row-count cache is untouched for
50+
* plain reads.</p>
51+
*/
52+
default Optional<ConnectorTableStatistics> getTableStatistics(
53+
ConnectorSession session,
54+
ConnectorTableHandle handle,
55+
ConnectorMvccSnapshot snapshot) {
56+
return getTableStatistics(session, handle);
57+
}
58+
3859
/**
3960
* Returns per-column statistics the connector can serve WITHOUT a table scan — the query-planner
4061
* column-statistics fast path, consulted on a stats-cache miss (fe-core's

‎fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorTableOps.java‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -136,6 +136,43 @@ default Map<String, ConnectorColumnHandle> getColumnHandles(
136136
"getColumnHandles not implemented");
137137
}
138138

139+
/**
140+
* Returns a name-to-handle map for all columns AT {@code snapshot.getSchemaId()} &mdash; the
141+
* columns as of the pinned snapshot, for time-travel reads under schema evolution.
142+
*
143+
* <p>The default ignores the snapshot and returns the latest columns via
144+
* {@link #getColumnHandles(ConnectorSession, ConnectorTableHandle)}. WHY this exists: the generic
145+
* scan node builds column handles BEFORE it pins the snapshot, so without a snapshot-aware overload
146+
* it can only key handles by the LATEST names. A time-travel query binds its slots to the PINNED
147+
* (old) names, so after a RENAME the renamed column's slot misses the latest-keyed map and is
148+
* silently dropped &mdash; crashing BE with a field-id dictionary miss on connectors whose native
149+
* projection is name/ordinal-driven (paimon). A connector that supports schema-at-snapshot
150+
* overrides this to key handles by the pinned names (mirrors the
151+
* {@link #getTableSchema(ConnectorSession, ConnectorTableHandle, ConnectorMvccSnapshot)} split) and
152+
* declares {@link #supportsColumnHandleSnapshotPin(ConnectorSession)}.</p>
153+
*/
154+
default Map<String, ConnectorColumnHandle> getColumnHandles(
155+
ConnectorSession session, ConnectorTableHandle handle,
156+
ConnectorMvccSnapshot snapshot) {
157+
return getColumnHandles(session, handle);
158+
}
159+
160+
/**
161+
* Whether {@link #getColumnHandles(ConnectorSession, ConnectorTableHandle, ConnectorMvccSnapshot)}
162+
* resolves handles AT the pinned snapshot's schema (i.e. keys them by the pinned names).
163+
*
164+
* <p>Only a connector that returns {@code true} is subject to the generic node's fail-loud check
165+
* that every bound column present in the pinned schema has a handle: for such a connector a missing
166+
* pinned column is a genuine bug (the connector promised pinned handles but dropped one) and must
167+
* fail with a clear error rather than being silently dropped into a BE crash. A connector that
168+
* returns {@code false} keeps the legacy latest-keyed handles and recovers from the drop by its own
169+
* means (e.g. iceberg rebuilds its field-id dictionary from the full pinned schema), so it is left
170+
* on the unchanged silent-skip path.</p>
171+
*/
172+
default boolean supportsColumnHandleSnapshotPin(ConnectorSession session) {
173+
return false;
174+
}
175+
139176
/** Lists all table names within the given database. */
140177
default List<String> listTableNames(ConnectorSession session,
141178
String dbName) {

‎fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/ConnectorBucketSpec.java‎

Lines changed: 13 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -22,14 +22,21 @@
2222
import java.util.Objects;
2323

2424
/**
25-
* Bucket / distribution specification carried by
26-
* {@link ConnectorCreateTableRequest}.
25+
* TABLE-LEVEL hash/random distribution ({@code DISTRIBUTE BY ...}) carried by
26+
* {@link ConnectorCreateTableRequest}; {@code null} when the user wrote no {@code DISTRIBUTE BY}.
2727
*
28-
* <p>{@code algorithm} is a connector-known string. Common values:</p>
28+
* <p>This is NOT the carrier for iceberg's per-column {@code bucket(num, column)} transform: that is a
29+
* partition-spec transform written inside {@code PARTITIONED BY} and resolved by
30+
* {@code IcebergSchemaBuilder.buildPartitionSpec}, never routed through this spec. Iceberg has no
31+
* whole-table hash/random distribution and rejects {@code DISTRIBUTE BY} outright
32+
* ({@code IcebergConnectorMetadata.rejectDistribution}). Today only the hive connector consumes this spec.</p>
33+
*
34+
* <p>{@code algorithm} is a connector-known string, produced by
35+
* {@code CreateTableInfoToConnectorRequestConverter}:</p>
2936
* <ul>
30-
* <li>{@code "hive_hash"} — Hive-compatible 32-bit hash.</li>
31-
* <li>{@code "iceberg_bucket"} — Iceberg bucket transform.</li>
32-
* <li>{@code "doris_default"} — Doris CRC32 distribution.</li>
37+
* <li>{@code "doris_default"} — hash distribution ({@code DISTRIBUTE BY HASH}).</li>
38+
* <li>{@code "doris_random"} — random distribution ({@code DISTRIBUTE BY RANDOM}); hive rejects it
39+
* (hive external tables support hash bucketing only).</li>
3340
* </ul>
3441
*/
3542
public final class ConnectorBucketSpec {

‎fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/handle/ConnectorTransaction.java‎

Lines changed: 0 additions & 53 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,6 @@
2020
import org.apache.doris.connector.api.pushdown.ConnectorPredicate;
2121

2222
import java.io.Closeable;
23-
import java.util.Set;
2423

2524
/**
2625
* A connector-managed transaction that scopes one or more write operations.
@@ -70,31 +69,6 @@ default void addCommitData(byte[] commitFragment) {
7069
// no-op: connectors that participate in writes override this
7170
}
7271

73-
/**
74-
* Whether this transaction allocates write block ranges through a write-time
75-
* BE&rarr;FE callback. Only connectors with a stateful write session that
76-
* hands out block ids (e.g. maxcompute) return {@code true}.
77-
*/
78-
default boolean supportsWriteBlockAllocation() {
79-
return false;
80-
}
81-
82-
/**
83-
* Allocates a contiguous range of write block ids for the given write
84-
* session, returning the first allocated id. Called from the BE&rarr;FE RPC
85-
* path during a write.
86-
*
87-
* <p>Only invoked when {@link #supportsWriteBlockAllocation()} returns
88-
* {@code true}; the default throws.</p>
89-
*
90-
* @param writeSessionId opaque connector-defined write session identifier
91-
* @param count number of block ids to allocate
92-
* @return the first allocated block id
93-
*/
94-
default long allocateWriteBlockRange(String writeSessionId, long count) {
95-
throw new UnsupportedOperationException("write block allocation not supported");
96-
}
97-
9872
/** Returns the number of rows affected by the write(s) bound to this transaction. */
9973
default long getUpdateCnt() {
10074
return 0;
@@ -127,31 +101,4 @@ default void applyWriteConstraint(ConnectorPredicate targetOnlyFilter) {
127101
default String profileLabel() {
128102
return "EXTERNAL";
129103
}
130-
131-
/**
132-
* Compaction rewrite ({@code rewrite_data_files}): registers the original data files this transaction
133-
* will atomically replace, by their RAW file paths. The engine rewrite driver hands the connector the
134-
* neutral {@code String} paths from its bin-packed groups (fe-core cannot traffic in connector-native
135-
* file objects across the wall); the connector resolves them back to its own file objects — e.g. by
136-
* re-deriving from the table at the transaction's pinned snapshot — and removes them at {@link #commit()}.
137-
*
138-
* <p>Default throws: only rewrite-capable connectors override it.</p>
139-
*
140-
* @param dataFilePaths the raw paths of the source data files to replace
141-
*/
142-
default void registerRewriteSourceFiles(Set<String> dataFilePaths) {
143-
throw new UnsupportedOperationException("rewrite source file registration not supported");
144-
}
145-
146-
/**
147-
* Compaction rewrite: the number of new compacted data files this transaction added, available only
148-
* AFTER {@link #commit()} (the count is materialized from the BE-reported commit fragments during commit).
149-
* Feeds the procedure's {@code added_data_files_count} result column — the one rewrite-result statistic
150-
* the engine driver cannot compute from its planning groups.
151-
*
152-
* <p>Default throws: only rewrite-capable connectors override it.</p>
153-
*/
154-
default int getRewriteAddedDataFilesCount() {
155-
throw new UnsupportedOperationException("rewrite statistics not supported");
156-
}
157104
}

‎fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/handle/ConnectorWriteHandle.java‎

Lines changed: 10 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -29,9 +29,9 @@
2929
* {@link org.apache.doris.connector.api.write.ConnectorWritePlanProvider#planWrite}.
3030
*
3131
* <p>Carries the engine-resolved facts about a single DML write: the target
32-
* table handle, the column list, whether it is an OVERWRITE, and a free-form
33-
* write context (static partition spec, write path, etc.). The connector reads
34-
* these to build its Thrift data sink.</p>
32+
* table handle, the column list, whether it is an OVERWRITE, and the static
33+
* partition spec ({@link #getWriteContext}). The connector reads these to build
34+
* its Thrift data sink.</p>
3535
*/
3636
public interface ConnectorWriteHandle {
3737

@@ -45,8 +45,13 @@ public interface ConnectorWriteHandle {
4545
boolean isOverwrite();
4646

4747
/**
48-
* Free-form write context: static partition spec, write path, and other
49-
* connector-defined keys carried from the bound sink to {@code planWrite}.
48+
* The static partition spec (partition column name -&gt; value) for a statically partitioned write,
49+
* carried from the bound sink to {@code planWrite}; an EMPTY map when the write is not statically
50+
* partitioned. Despite the "write context" name (once envisioned as a free-form bag), the only content
51+
* the sole producer ({@code PluginDrivenTableSink.bindDataSink} -&gt;
52+
* {@code PluginDrivenInsertCommandContext.getStaticPartitionSpec}) ever populates is the static
53+
* partition spec, and the write providers (hive/iceberg/maxcompute) all consume it as such (iceberg
54+
* ships it verbatim as {@code TDataSink.static_partition_values}).
5055
*/
5156
Map<String, String> getWriteContext();
5257

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
package org.apache.doris.connector.api.handle;
19+
20+
import java.util.Set;
21+
22+
/**
23+
* Narrow opt-in capability for a {@link ConnectorTransaction} that supports the compaction
24+
* {@code rewrite_data_files} procedure (only iceberg today).
25+
*
26+
* <p>Kept OFF {@link ConnectorTransaction} so the shared transaction contract carries no source-specific
27+
* methods: the engine rewrite driver ({@code ConnectorRewriteDriver}) checks {@code instanceof
28+
* RewriteCapableTransaction} before it calls, turning "unsupported" from a runtime throw into a type
29+
* mismatch. A connector whose transaction is not rewrite-capable simply does not implement this.</p>
30+
*/
31+
public interface RewriteCapableTransaction {
32+
33+
/**
34+
* Compaction rewrite ({@code rewrite_data_files}): registers the original data files this transaction
35+
* will atomically replace, by their RAW file paths. The engine rewrite driver hands the connector the
36+
* neutral {@code String} paths from its bin-packed groups (fe-core cannot traffic in connector-native
37+
* file objects across the wall); the connector resolves them back to its own file objects — e.g. by
38+
* re-deriving from the table at the transaction's pinned snapshot — and removes them at
39+
* {@code ConnectorTransaction#commit()}.
40+
*
41+
* @param dataFilePaths the raw paths of the source data files to replace
42+
*/
43+
void registerRewriteSourceFiles(Set<String> dataFilePaths);
44+
45+
/**
46+
* Compaction rewrite: the number of new compacted data files this transaction added, available only
47+
* AFTER {@code ConnectorTransaction#commit()} (the count is materialized from the BE-reported commit
48+
* fragments during commit). Feeds the procedure's {@code added_data_files_count} result column — the one
49+
* rewrite-result statistic the engine driver cannot compute from its planning groups.
50+
*/
51+
int getRewriteAddedDataFilesCount();
52+
}

0 commit comments

Comments
 (0)