Skip to content

Commit 042e613

Browse files
authored
[test](regression) Complete Paimon read/write P0 coverage (#66065)
### What problem does this PR solve? Problem Summary: The Paimon P0 regression suite did not distinguish the four primary-key merge engines and did not verify that every unsupported Doris data-write shape leaves both data and snapshots unchanged. This PR: - adds Parquet and ORC coverage for deduplicate, partial-update, aggregation, and first-row tables under automatic and forced-JNI reader routing; - verifies dynamic-bucket cross-partition deduplication; - covers rejected INSERT, INSERT SELECT, INSERT OVERWRITE, UPDATE, DELETE, and MERGE statements with pre/post row and snapshot invariants; - records the Paimon CTAS metadata-atomicity defect as an isolated opt-in negative regression; and - documents the Paimon/Iceberg read/write P0 coverage matrix and why no additional Iceberg P0 case is needed for the currently supported surface.
1 parent 74227a8 commit 042e613

6 files changed

Lines changed: 626 additions & 0 deletions

File tree

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
-- This file is automatically generated. You should know what you did if you want to edit this
2+
-- !auto_parquet_deduplicate --
3+
1 11 new-1
4+
2 22 new-2
5+
3 30 only-3
6+
7+
-- !auto_parquet_partial_update --
8+
1 15 2 filled-1
9+
2 30 4 base-2
10+
11+
-- !auto_parquet_aggregation --
12+
1 7 35 new-1
13+
2 9 300 new-2
14+
15+
-- !auto_parquet_first_row --
16+
1 10 first-1
17+
2 20 first-2
18+
3 30 first-3
19+
20+
-- !auto_parquet_engine_aggregates --
21+
63 51 16 60
22+
23+
-- !auto_orc_deduplicate --
24+
1 11 new-1
25+
2 22 new-2
26+
3 30 only-3
27+
28+
-- !auto_orc_partial_update --
29+
1 15 2 filled-1
30+
2 30 4 base-2
31+
32+
-- !auto_orc_aggregation --
33+
1 7 35 new-1
34+
2 9 300 new-2
35+
36+
-- !auto_orc_first_row --
37+
1 10 first-1
38+
2 20 first-2
39+
3 30 first-3
40+
41+
-- !auto_orc_engine_aggregates --
42+
63 51 16 60
43+
44+
-- !auto_dynamic_cross_partition --
45+
1 new-part 11
46+
2 stable-part 20
47+
48+
-- !auto_dynamic_old_partition --
49+
0
50+
51+
-- !auto_dynamic_new_partition --
52+
1
53+
54+
-- !forced_jni_parquet_deduplicate --
55+
1 11 new-1
56+
2 22 new-2
57+
3 30 only-3
58+
59+
-- !forced_jni_parquet_partial_update --
60+
1 15 2 filled-1
61+
2 30 4 base-2
62+
63+
-- !forced_jni_parquet_aggregation --
64+
1 7 35 new-1
65+
2 9 300 new-2
66+
67+
-- !forced_jni_parquet_first_row --
68+
1 10 first-1
69+
2 20 first-2
70+
3 30 first-3
71+
72+
-- !forced_jni_parquet_engine_aggregates --
73+
63 51 16 60
74+
75+
-- !forced_jni_orc_deduplicate --
76+
1 11 new-1
77+
2 22 new-2
78+
3 30 only-3
79+
80+
-- !forced_jni_orc_partial_update --
81+
1 15 2 filled-1
82+
2 30 4 base-2
83+
84+
-- !forced_jni_orc_aggregation --
85+
1 7 35 new-1
86+
2 9 300 new-2
87+
88+
-- !forced_jni_orc_first_row --
89+
1 10 first-1
90+
2 20 first-2
91+
3 30 first-3
92+
93+
-- !forced_jni_orc_engine_aggregates --
94+
63 51 16 60
95+
96+
-- !forced_jni_dynamic_cross_partition --
97+
1 new-part 11
98+
2 stable-part 20
99+
100+
-- !forced_jni_dynamic_old_partition --
101+
0
102+
103+
-- !forced_jni_dynamic_new_partition --
104+
1
105+
Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
-- This file is automatically generated. You should know what you did if you want to edit this
2+
-- !before_rows --
3+
1 10 base-1
4+
2 20 base-2
5+
6+
-- !before_snapshots --
7+
1
8+
9+
-- !after_rows --
10+
1 10 base-1
11+
2 20 base-2
12+
13+
-- !after_snapshots --
14+
1
15+
Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
<!--
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing,
13+
software distributed under the License is distributed on an
14+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
KIND, either express or implied. See the License for the
16+
specific language governing permissions and limitations
17+
under the License.
18+
-->
19+
20+
# Paimon / Iceberg Read and Write P0 Coverage
21+
22+
## Scope
23+
24+
This matrix compares the supported Doris surface with the format capabilities documented by
25+
[Apache Paimon](https://paimon.apache.org/docs/1.0/), its
26+
[ecosystem matrix](https://paimon.apache.org/docs/master/ecosystem/), and Iceberg's
27+
[write](https://iceberg.apache.org/docs/latest/spark-writes/) and
28+
[evolution](https://iceberg.apache.org/docs/latest/evolution/) documentation. A format feature is a Doris P0 contract
29+
only when Doris supports it. Unsupported write or format boundaries require deterministic negative
30+
coverage and no metadata or data mutation.
31+
32+
Doris currently documents Paimon data access as read-only. Master also supports Paimon catalog,
33+
database and table metadata creation/deletion, but it has no Paimon data sink. Iceberg supports read,
34+
DDL, INSERT, INSERT OVERWRITE, CTAS and, for compatible V2/V3 tables, DELETE, UPDATE and MERGE INTO.
35+
36+
## Audit result
37+
38+
The master inventory before this change contained 41 Paimon suites and 145 Iceberg suites, including
39+
35 Iceberg write suites and four dedicated Iceberg DML suites. Iceberg P0 is complete for the Doris-
40+
supported read/write surface: its existing matrices cover file-format versions, delete encodings,
41+
evolution, time travel, DDL, distributed writes, row-level DML, failure atomicity and unsupported
42+
boundaries.
43+
44+
Paimon P0 was incomplete. No suite selected a `merge-engine`, and write rejection was not checked
45+
across all DML shapes with pre/post data and snapshot invariants. PM01-PM03 below close those gaps.
46+
PM04 exposes one remaining product defect as an opt-in negative regression: failed Paimon CTAS leaves
47+
target metadata. Streaming writes, overwrite, delete/update and merge listed as unsupported by the
48+
Paimon ecosystem matrix are boundary tests rather than positive Doris P0 contracts.
49+
50+
## Risks
51+
52+
| ID | Risk | Source | Impact | Priority |
53+
| --- | --- | --- | --- | --- |
54+
| R01 | Append and primary-key tables are planned with the wrong merge semantics | Black box: Paimon table models | Silent wrong results | P0 |
55+
| R02 | Deduplicate, partial-update, aggregation and first-row produce the same result for duplicate keys | White box: LSM sorted-run merge | Silent wrong results | P0 |
56+
| R03 | Fixed and dynamic bucket tables expose duplicate or stale cross-partition keys | Paimon data distribution | Silent wrong results | P0 |
57+
| R04 | Automatic and forced-JNI routing disagree on Paimon merged rows, or native raw-file reads disagree where conversion is supported | Doris split routing | Query correctness | P0 |
58+
| R05 | A rejected Paimon write creates a snapshot, changes data or leaves a CTAS table | Doris read-only boundary | Data or metadata mutation | P0 |
59+
| R06 | Iceberg schema or partition evolution binds old field/spec IDs during current or historical reads | Iceberg evolution | Silent wrong results | P0 |
60+
| R07 | Position/equality deletes or V3 deletion vectors are applied to the wrong file or snapshot | Iceberg row-level deletes | Deleted data visible or live data lost | P0 |
61+
| R08 | Iceberg writes lose rows, route them to the wrong transform, or publish partial failed commits | Doris distributed Iceberg sink | Data loss or corruption | P0 |
62+
| R09 | Snapshot, tag or branch reads/writes leak schema or data across references | Both formats | Historical data corruption | P0 |
63+
| R10 | Unsupported format, DML mode or CTAS mutates state before rejection | Capability boundary | Partial commits or orphan metadata | P0 |
64+
65+
## Feature matrix
66+
67+
| Format | Capability | P0 status | Main suites |
68+
| --- | --- | --- | --- |
69+
| Paimon | Append table, partitioned table, primitive and nested types | Covered | `test_paimon_catalog`, `test_paimon_partition_table`, `test_paimon_full_schema_change` |
70+
| Paimon | Primary-key deduplicate, partial-update, aggregation and first-row | Covered | `test_paimon_merge_engine_matrix` |
71+
| Paimon | Fixed bucket, dynamic bucket and cross-partition update | Covered | `test_paimon_merge_engine_matrix`, `test_paimon_partition_pk_delete_refs` |
72+
| Paimon | Parquet/ORC and mixed-format reads; JNI/native parity | Covered | `test_paimon_merge_engine_matrix`, `paimon_tb_mix_format`, `test_paimon_cpp_reader` |
73+
| Paimon | Snapshot/timestamp/tag/branch and incremental modes | Covered | `paimon_time_travel`, `paimon_incr_read`, `test_paimon_schema_time_travel_matrix` |
74+
| Paimon | Schema evolution, partition-key restrictions and historical schema binding | Covered | `test_paimon_schema_time_travel_matrix`, `test_paimon_partition_mutation_atomicity` |
75+
| Paimon | Deletion vectors, upsert/delete visibility and data/system tables | Covered | `test_paimon_deletion_vector`, `paimon_data_system_table`, `paimon_system_table` |
76+
| Paimon | Catalog/database/table create and drop | Covered | `test_create_paimon_table` |
77+
| Paimon | Doris data write-back | Negative boundary covered | `test_paimon_write_boundary` |
78+
| Paimon | Failed CTAS metadata atomicity | Isolated known-bug regression | `test_paimon_ctas_atomicity_negative` |
79+
| Iceberg | V1/V2/V3, Parquet/ORC, position/equality deletes and deletion vectors | Covered | `test_iceberg_position_delete`, `test_iceberg_equality_delete`, `test_iceberg_deletion_vector` |
80+
| Iceberg | Schema, partition and sort-order evolution | Covered | `test_iceberg_schema_time_travel_matrix`, `test_iceberg_partition_evolution_format_scanner`, `iceberg_schema_change_ddl` |
81+
| Iceberg | Snapshot/timestamp/tag/branch reads and reference actions | Covered | `test_iceberg_time_travel`, `iceberg_query_tag_branch`, `test_iceberg_schema_ref_actions_matrix` |
82+
| Iceberg | INSERT, INSERT OVERWRITE, static/hybrid partition and CTAS | Covered | `write/test_iceberg_write_insert`, `write/test_iceberg_write_overwrite_evolution`, `write/test_iceberg_write_ctas_format_boundary` |
83+
| Iceberg | DELETE, UPDATE and MERGE in supported MOR modes | Covered | `dml/test_iceberg_update_delete_advanced`, `dml/test_iceberg_merge_into_advanced`, `write/test_iceberg_write_dml_modes_evolution` |
84+
| Iceberg | Distributed/concurrent commits, failure atomicity and Spark interoperability | Covered | `write/test_iceberg_write_concurrent_merge_invariants`, `write/test_iceberg_write_overwrite_atomicity` |
85+
| Iceberg | System tables, views, caches and catalog variants | Covered | `test_iceberg_sys_table`, `test_iceberg_view_query_p0`, `test_iceberg_table_cache`, catalog-specific suites |
86+
| Iceberg | COW row DML, tag writes and unsupported file writes | Negative boundary covered | `write/test_iceberg_write_dml_modes_evolution`, `write/test_iceberg_write_branch_dml_boundary`, `write/test_iceberg_write_ctas_format_boundary` |
87+
88+
Detailed schema/time-travel and Iceberg write combinations are maintained in
89+
`iceberg_paimon_schema_time_travel_coverage.md` and `iceberg/write/ICEBERG_WRITE_P0_COVERAGE.md`.
90+
91+
## Added test design
92+
93+
| Case | Goal | Risks | Dimension | Preconditions | Load | Expected |
94+
| --- | --- | --- | --- | --- | --- | --- |
95+
| PM01 | Distinguish all four primary-key merge engines | R01, R02, R04 | Functional, correctness, compatibility | Paimon Parquet/ORC tables | Duplicate keys across several commits | Each engine returns its documented merged row under automatic and forced-JNI routing |
96+
| PM02 | Validate dynamic-bucket cross-partition deduplication | R03, R04 | Correctness | Primary key excludes partition key, bucket=-1 | Move one key between partitions | Exactly one current row remains in the new partition |
97+
| PM03 | Preserve the Paimon read-only boundary | R05, R10 | Negative, atomicity | Existing Paimon PK table | VALUES, SELECT, OVERWRITE, UPDATE, DELETE, MERGE | Every statement fails before a snapshot or data change |
98+
| PM04 | Reject or roll back Paimon CTAS atomically | R05, R10 | Isolated negative, atomicity | Paimon catalog with no target table | CREATE TABLE AS SELECT | Desired contract: the command fails and no target table remains; current bug leaves the table |
99+
100+
Every P0 risk maps to at least one deterministic positive, boundary, or isolated known-bug
101+
regression. Catalog authentication and cloud storage permutations stay in their existing connector
102+
suites because they do not add format semantics to this matrix.
Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
suite("test_paimon_ctas_atomicity_negative",
19+
"p0,external,paimon,external_docker,external_docker_paimon") {
20+
String enabled = context.config.otherConfigs.get("enablePaimonTest")
21+
if (enabled == null || !enabled.equalsIgnoreCase("true")) {
22+
logger.info("disable paimon test")
23+
return
24+
}
25+
26+
// CTAS currently creates Paimon metadata before discovering that no Paimon data sink exists.
27+
// Keep this opt-in until the product rejects or rolls back the statement atomically.
28+
String knownBugEnabled = context.config.otherConfigs.get("enablePaimonKnownBugTest")
29+
if (knownBugEnabled == null || !knownBugEnabled.equalsIgnoreCase("true")) {
30+
logger.info("skip isolated Paimon known-bug regression")
31+
return
32+
}
33+
34+
String minioPort = context.config.otherConfigs.get("iceberg_minio_port")
35+
String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
36+
String catalogName = "test_paimon_ctas_atomicity_negative"
37+
String dbName = "paimon_ctas_atomicity_negative_db"
38+
39+
sql """drop catalog if exists ${catalogName}"""
40+
sql """
41+
create catalog ${catalogName} properties (
42+
'type'='paimon',
43+
'warehouse'='s3://warehouse/wh',
44+
's3.endpoint'='http://${externalEnvIp}:${minioPort}',
45+
's3.access_key'='admin',
46+
's3.secret_key'='password',
47+
's3.path.style.access'='true',
48+
'meta.cache.paimon.table.ttl-second'='0'
49+
)
50+
"""
51+
52+
try {
53+
spark_paimon_multi """
54+
create database if not exists paimon.${dbName};
55+
drop table if exists paimon.${dbName}.ctas_target;
56+
"""
57+
58+
sql """switch ${catalogName}"""
59+
sql """use ${dbName}"""
60+
61+
// A failed CTAS must not leave metadata that makes a retry fail with TABLE ALREADY EXISTS.
62+
test {
63+
sql """
64+
create table ctas_target engine=paimon
65+
as select cast(1 as int) as id, cast('candidate' as string) as payload
66+
"""
67+
exception "PaimonExternalCatalog"
68+
}
69+
assertEquals(0, (sql """show tables like 'ctas_target'""").size())
70+
} finally {
71+
spark_paimon """drop table if exists paimon.${dbName}.ctas_target"""
72+
sql """drop catalog if exists ${catalogName}"""
73+
}
74+
}

0 commit comments

Comments
 (0)