Repository navigation
Commit c954090
[opt](catalog) fe-connector: unify the connector cache foundation, memoize per-statement hot paths, fix iceberg case-insensitive column DDL (#66029)
### What problem does this PR solve?
Issue Number: #65185
Problem Summary:
Part of the catalog-SPI migration (#65185). Review round 19, targeting
`branch-catalog-spi`.
Three themes plus one bug fix.
**1. Connector cache foundation unification (`fe-connector-cache` /
`fe-connector-api`)**
Several connectors had each grown their own copy of the same two caching
shapes. This round factors
them out instead of copying them a fourth time:
- `ConnectorPartitionViewCache` is generalized into a reusable
`ConnectorMetadataCache`, so a
connector can cache a derived metadata view without re-implementing the
loader / invalidation
plumbing.
- The per-statement table memo that iceberg, paimon and hudi each
open-coded is extracted into a
shared `ConnectorStatementScopes` helper in `fe-connector-api`.
- The "`ttl <= 0` disables the cache" contract is folded into
`CacheSpec.ofConnectorTtl` rather than
being re-derived at every call site.
- Statement-scope namespace constants are pushed DOWN into each
connector (jdbc's lands on
`JdbcConnectorMetadata`); `fe-connector-api` keeps only the neutral
`resolveInStatement` plus the
key convention, so the API stays source-agnostic. Namespaces are
prefixed with `getType()`, which
makes cross-connector collisions impossible by construction, and each
connector reflectively walks
its own `*_NAMESPACE` constants to assert the prefix, so the check
extends itself.
- A fe-core test guards that the connector per-statement scope is reset
across prepared executions of
`ExecuteCommand`.
**2. Round-1 hot-path memoization (perf)**
Removes repeated remote round-trips and repeated local CPU within a
single statement:
| connector | change |
| --- | --- |
| hudi | cache the HMS client (fresh listing + REFRESH flush); memoize
latest schema + instant per statement |
| maxcompute | memoize the resolved table handle per statement |
| es | hoist per-scan metadata state, memoize schema, and share ONE
index-mapping fetch per statement across the schema and scan paths |
| paimon | route `SHOW PARTITIONS` / `partition_values()` through the
partition-view cache |
| jdbc | dedup the remote column fetch per statement across the read and
write paths |
Some of these save remote IO and some save only local CPU (repeated
rendering / parsing / object
construction behind an already-warm lower-level cache); both are counted
as real wins here.
**3. Build-gate cleanup**
`tools/check-authz-cache-sharding.sh` (and its self-test) are removed. A
shell/regex gate cannot decide
Java boolean semantics — two adversarial review rounds produced 10 and
11 false positives/negatives
respectively, and a false positive is the worse failure mode because it
blocks legitimate builds. The
invariant it guarded is already proven by runtime behavior tests
(`IcebergConnectorCacheTest`) and is
now stated as an ATTN comment at the call sites. Static gates stay for
existence/prefix/marker
invariants only (e.g. `check-connector-imports`).
**4. Fix: iceberg now rejects case-insensitive column-name collisions on
the top-level ALTER ops**
A port gap from #65329. Upstream `IcebergMetadataOps` called
`validateNoCaseInsensitiveSiblingCollision` from five sites — flat ADD,
nested ADD, the ADD COLUMNS
batch, flat RENAME and nested RENAME — but the connector port carried
only the two NESTED sites.
Top-level column DDL never reaches that class: the fe-core bridge
short-circuits every non-nested
`ColumnPath` back to the flat SPI, and the connector short-circuits it
again, so the flat ops went
straight to `UpdateSchema` with no validation.
Iceberg could not catch it either: `SchemaUpdate` sets `caseSensitive =
true`, so `findField("id")`
does not see an existing `Id`, while Doris column names are
case-insensitive. Against a Spark-created
table holding `Id INT, Label STRING, Info STRUCT<Metric:INT,
Label:STRING>`:
- `ALTER TABLE t ADD COLUMN id STRING` silently committed a second
top-level column differing from
`Id` only in case;
- `ALTER TABLE t RENAME COLUMN label TO ...` failed with iceberg's raw
`Cannot rename missing column: label`, because the real field is
`Label`.
The three top-level ops now use the validators the nested arm already
had:
- `addColumn` rejects a name colliding case-insensitively with an
existing field.
- `addColumns` does the same per column and additionally rejects two
names of one request that
collide with each other; the whole batch is validated before anything is
staged, so a partly-valid
request commits nothing.
- `renameColumn` resolves the SOURCE name case-insensitively, lets a
pure re-casing rename through
via the field-identity escape, and restates identifier-field paths
through the existing
`applyRenameColumn`.
`validateNoCaseInsensitiveSiblingCollision` becomes package-private; its
`parentPath.isEmpty()`
top-level arm had already been written but was unreachable, which is
what flagged the missing call
sites.
Known remaining gaps in the same flat arm, deliberately left out of this
PR and not exercised by any
current test: flat `dropColumn` and `modifyColumn` still resolve names
case-sensitively, the `AFTER`
position reference is not resolved case-insensitively, and
`reorderColumns` neither canonicalizes
names nor rejects duplicates.
### Release note
Fixed an Iceberg external-catalog issue where `ALTER TABLE ... ADD
COLUMN` and `RENAME COLUMN` ignored
case-insensitive collisions with existing column names, so a column
differing from an existing one
only in case could be added; renaming a column whose name was typed in a
different case than the
underlying Iceberg field also failed.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
The iceberg fix is covered by the existing suite
`external_table_p0/iceberg/test_iceberg_nested_schema_evolution_spark_doris_interop.groovy`
(lines 273-297), which was failing on this branch; four of its
assertions after line 273 had never
been reached before. New unit tests: 7 cases in
`CatalogBackedIcebergCatalogOpsColumnEvolutionTest`
against a real `InMemoryCatalog` with a mixed-case fixture, verified to
fail on the pre-fix code with
the exact CI symptoms. The cache and perf work adds unit tests per
connector
(`HudiConnectorHmsCacheTest`, `HudiStatementMemoTest`,
`MaxComputeConnectorMetadataHandleMemoTest`,
`JdbcConnectorMetadataTest`, `JdbcWritePlanProviderTest`,
`IcebergStatementScopeTest`,
`PaimonConnectorMetadataPartitionViewCacheTest`,
`ConnectorStatementScopeTest`, ...).
- Behavior changed:
- [x] Yes.
Iceberg `ALTER TABLE ... ADD COLUMN` / `ADD COLUMN (...)` / `RENAME
COLUMN` now fail with an explicit
"conflicts with existing Iceberg field '<name>' (case-insensitive)"
message where they previously
succeeded (or failed with iceberg's raw message). A top-level `RENAME
COLUMN` whose source name
differs only in case from the underlying Iceberg field now succeeds
instead of failing.
- Does this need documentation?
- [x] No.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>1 parent 565d31e commit c954090
102 files changed
Lines changed: 7223 additions & 614 deletions
File tree
- fe
- fe-connector
- fe-connector-api/src
- main/java/org/apache/doris/connector/api
- test/java/org/apache/doris/connector/api
- fe-connector-cache/src
- main/java/org/apache/doris/connector/cache
- test/java/org/apache/doris/connector/cache
- fe-connector-es/src
- main/java/org/apache/doris/connector/es
- test/java/org/apache/doris/connector/es
- fe-connector-hive/src
- main/java/org/apache/doris/connector/hive
- test/java/org/apache/doris/connector/hive
- fe-connector-hudi/src
- main/java/org/apache/doris/connector/hudi
- test/java/org/apache/doris/connector/hudi
- fe-connector-iceberg/src
- main/java/org/apache/doris/connector/iceberg
- action
- test/java/org/apache/doris/connector/iceberg
- fe-connector-jdbc/src
- main/java/org/apache/doris/connector/jdbc
- test/java/org/apache/doris/connector/jdbc
- fe-connector-maxcompute/src
- main/java/org/apache/doris/connector/maxcompute
- test/java/org/apache/doris/connector/maxcompute
- fe-connector-paimon/src
- main/java/org/apache/doris/connector/paimon
- test/java/org/apache/doris/connector/paimon
- fe-core/src/test/java/org/apache/doris/connector
- plan-doc
- connector-cache-unification
- connectors
- data
- designs
- perf-hotpath-es
- designs
- perf-hotpath-hudi
- designs
- perf-hotpath-jdbc
- perf-hotpath-maxcompute
- designs
- perf-hotpath-paimon
- tools
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
67 | 67 | | |
68 | 68 | | |
69 | 69 | | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
70 | 87 | | |
71 | 88 | | |
72 | 89 | | |
| |||
0 commit comments