You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 0bdb7b6
Browse filesBrowse the repository at this point in the historyBrowse files
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: apache#65564, apache#66313, apache#66417
Problem Summary:
Master port of the remaining external-regression gaps diagnosed on the
branch-4.0 pipeline (apache#66417). apache#66313 already carried the keytab and
baseline
fixes to master; this PR carries the rest. All three defects were
verified to
exist on master by inspection — master shares the same code and the same
lightweight Kerberos environment (apache#65564, which apache#65939 backported to
4.0).
**1. Kerberos HDFS writes fail in the p2 suites (SASL client property
missing)**
apache#65564 created `kerberos/conf/hdfs-site.xml.tpl` with
`dfs.data.transfer.protection=authentication`. A client that does not
set the
same property sends a plain `writeBlock` op, `SaslDataTransferServer`
closes the
socket, the client excludes the only DataNode, and the write fails with
could only be written to 0 of the 1 minReplication nodes.
There are 1 datanode(s) running and 1 node(s) are excluded in this
operation.
(through iceberg this surfaces as `IllegalArgumentException:
Self-suppression
not permitted`). Metadata operations only reach the NameNode and the
HMS, so
`CREATE DATABASE` / `CREATE TABLE` still succeed and only the write
fails.
apache#65564 set the client property in the four `external_table_p0/kerberos`
suites
but not in the p2 suites that talk to the same kerberized HDFS. On the
branch-4.0 pipeline this turned `hive_on_hms_and_dlf` and
`iceberg_on_hms_and_filesystem_and_dlf` red on every run; master carries
the
identical omission.
Fix: set `dfs.data.transfer.protection` in every property block that
points at
the kerberized HDFS. A scan of `:8520`/`:8620` across the whole suite
tree finds
seven such blocks still missing it: two each in `hive_on_hms_and_dlf`,
`iceberg_on_hms_and_filesystem_and_dlf` and `test_paimon_hms_catalog`,
plus one
in `hdfs_all_test` (currently returns early because
`refactor_params_hdfs_kerberos_test` is unset — it would fail the moment
it is
enabled). The kerberos block of `test_information_schema_timezone` has
the same
gap but is disabled by an unrelated TODO, so it is left alone.
**2. `test_paimon_hms_catalog`: `Unknown database 'hdfs_db'`**
The lightweight Kerberos environment starts an empty metastore —
`schematool
-initSchema` and nothing else. The environment it replaced uploaded
`paimon_data` into HDFS and registered `hdfs_db` through
`create_paimon_hive_table.hql` on every start; `paimon_data/hdfs_db.db/`
is
still in the tree but the compose file never mounted it. This restores
the
fixture on `kerberos1` only (it owns metastore 9583, the only one the
suite
talks to), runs it before `DORIS_KERBEROS_READY` so nothing races the
suites,
and aborts the container on failure instead of handing out an
environment
silently missing `hdfs_db`.
Two deliberate differences from the branch-4.0 sibling commit:
- **`hdfs_db` only, no `ali_db`/OSS.** Master's pre-apache#65564 HQL
provisioned
exactly `hdfs_db`; the `oss_hms_kerberos` case that reads `ali_db`
through
the kerberized metastore exists only on branch-4.0, master's `ali_db`
cases
go through the Hive3 metastore, and the golden output here carries only
the
`hdfs_kerberos` / `hdfs_new_kerberos` blocks. This keeps the
jindo/aliyun
jars and OSS credentials out of the kerberos stack.
- **The fixture is gated on the container role (`HOST`).** Both kerberos
containers run the same entrypoint with the same rendered env switch,
but
only `kerberos1` carries the `sql/`/`paimon_data/`/`auxlib/` mounts —
without
the gate, `set -e` would abort `kerberos2` the moment `enablePaimonHms`
is
on. (The branch-4.0 sibling needs the same gate; a follow-up on apache#66417
will
add it.)
**3. `IcebergConnectorMetadata`: report the root cause of failed
metadata commits**
`TableMetadataParser.internalWrite` writes the metadata JSON inside
try-with-resources, and `DFSOutputStream` throws the same exception
instance
from `write` and from `close` once its pipeline has failed, so
`Throwable.addSuppressed(this)` replaces the failure with
`IllegalArgumentException: Self-suppression not permitted`. The FE log
keeps the
chain, but the message returned over the MySQL protocol is built from
`e.getMessage()` and carries only the wrapper.
This switches all nineteen metadata-commit paths — `createTable`, the
column/schema updates (including the nested-column and comment
variants), the
branch/tag operations, and the partition-field operations — to
`ExceptionUtils.getRootCauseMessage(e)`. The legacy `IcebergMetadataOps`
already reported the root cause for the partition-field trio; the
connector
rewrite lost that, so this also restores it. Read paths and the
drop/rename
operations cannot hit self-suppression and keep `e.getMessage()`.
**Intentionally not ported from apache#66417:** the file-cache de-flake.
Master's
`test_file_cache_features` (rewritten in apache#66313's lineage) already waits
for
`disk_resource_limit_mode` / `need_evict_cache_in_advance` to clear
before
returning, which closes the residual-state window at its source.
**Verification (local):** `fe-connector-iceberg` compiles and passes
checkstyle
(`mvn -pl fe-connector/fe-connector-iceberg package`); the four changed
suites
compile with the offline groovy compiler; `bash -n` passes on both shell
scripts; the compose/env/conf templates render with no leftover
placeholders,
the rendered XML parses, kerberos1 gets exactly the three new mounts
while
kerberos2 is unchanged; the settings resolution and the entrypoint
role-gate
were exercised for all switch/role combinations. The containers
themselves need
the CI environment and have not been run.
### Release note
None
### Check List (For Author)
- Test
- [x] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
---------
Signed-off-by: morningman <[email protected]>
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java
0 commit comments