Skip to content

Commit 318fe23

Browse files
morningmansuxiaogang223Gabriel39claude
committed
[chore](paimon) Upgrade the docker Spark fixture to Paimon 1.4.2
### What problem does this PR solve? Issue Number: None Related PR: #66346, #66536, #67023 Problem Summary: Branch-4.1 #66346 upgraded Paimon to 1.4.2 on both sides of the regression tests: the Doris FE/BE dependency, and the Spark connector of the Iceberg docker environment, which creates the Paimon fixtures and reads back what Doris writes. Master took the dependency part with #67395 but kept the docker Spark on Paimon 1.3.1, so the Paimon suites check Doris against data written by an older Paimon than the one Doris runs. test_paimon_write_external_paths also had to drop its weight-robin case: Paimon 1.3.1 does not know that strategy, and Spark fails on such a table with "Could not parse value 'weight-robin' for key 'data-file.external-paths.strategy'". The jars cannot be upgraded alone. Paimon 1.4.2 parses a present Variant shredding schema property as JSON, so the insert that run13.sql makes after setting the property to '' fails with "No content to map due to end-of-input", and the Spark entrypoint stops there. This change brings the docker environment to the branch-4.x state: - run-thirdparties-docker.sh downloads iceberg_data_spark40_paimon142.zip, the package branch-4.x uses. It is iceberg_data_spark40.zip with paimon-s3 and paimon-spark replaced by their 1.4.2 builds; the other 3881 entries have the same size and CRC. - iceberg.yaml.tpl mounts paimon-s3-1.4.2.jar and paimon-spark-4.0_2.13-1.4.2.jar into the Spark container. - run13.sql removes the property with UNSET TBLPROPERTIES before writing the unshredded file of variant_mixed_su (#66536). - test_paimon_write_external_paths gets back its weight-robin case from branch-4.x (#67023). - test_paimon_partition_table.out is regenerated: Paimon 1.4.2 writes smaller Parquet files for the run07.sql tables, so only the file sizes of SHOW PARTITIONS change, to the values branch-4.x has. ### Release note None ### Check List (For Author) - Test: Regression test - Rebuilt the Iceberg docker environment from scratch with the new package and ran the suites under external_table_p0/paimon on a local cluster. The suites that need the Hive docker environment, cloud credentials or Linux were not run locally. - Behavior changed: No - Does this need documentation: No Co-authored-by: Socrates <[email protected]> Co-authored-by: Gabriel <[email protected]> Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 8b67d53 commit 318fe23

6 files changed

Lines changed: 64 additions & 31 deletions

File tree

‎docker/thirdparties/docker-compose/iceberg/iceberg.yaml.tpl‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,8 +35,8 @@ services:
3535
- ./spark-defaults.conf:/opt/spark/conf/spark-defaults.conf
3636
- ./data/input/jars/iceberg-aws-bundle-1.10.1.jar:/opt/spark/jars/iceberg-aws-bundle-1.10.1.jar
3737
- ./data/input/jars/iceberg-spark-runtime-4.0_2.13-1.10.1.jar:/opt/spark/jars/iceberg-spark-runtime-4.0_2.13-1.10.1.jar
38-
- ./data/input/jars/paimon-s3-1.3.1.jar:/opt/spark/jars/paimon-s3-1.3.1.jar
39-
- ./data/input/jars/paimon-spark-4.0-1.3.1.jar:/opt/spark/jars/paimon-spark-4.0-1.3.1.jar
38+
- ./data/input/jars/paimon-s3-1.4.2.jar:/opt/spark/jars/paimon-s3-1.4.2.jar
39+
- ./data/input/jars/paimon-spark-4.0_2.13-1.4.2.jar:/opt/spark/jars/paimon-spark-4.0_2.13-1.4.2.jar
4040
environment:
4141
- AWS_ACCESS_KEY_ID=admin
4242
- AWS_SECRET_ACCESS_KEY=password

‎docker/thirdparties/docker-compose/iceberg/scripts/create_preinstalled_scripts/paimon/run13.sql‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -63,8 +63,10 @@ tblproperties (
6363

6464
insert into variant_mixed_su values
6565
(1, date '2026-06-01', parse_json('{"name":"alice","age":18,"layout":"shredded"}'));
66-
alter table variant_mixed_su set tblproperties (
67-
'parquet.variant.shreddingSchema' = ''
66+
-- Paimon 1.4.2 parses every present shredding schema as JSON, so remove the property
67+
-- before writing unshredded files.
68+
alter table variant_mixed_su unset tblproperties (
69+
'parquet.variant.shreddingSchema'
6870
);
6971
insert into variant_mixed_su values
7072
(2, date '2026-07-01', parse_json('{"name":"bob","age":30,"layout":"unshredded"}'));

‎docker/thirdparties/run-thirdparties-docker.sh‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1443,15 +1443,15 @@ start_iceberg() {
14431443
cd "${ICEBERG_DIR}" || exit 1
14441444
rm -f iceberg_data*.zip
14451445
if [[ "${HOST_OS}" == "Darwin" ]]; then
1446-
curl -fL -o iceberg_data_spark40.zip \
1447-
"https://${s3BucketName}.${s3Endpoint}/regression/datalake/pipeline_data/iceberg_data_spark40.zip"
1446+
curl -fL -o iceberg_data_spark40_paimon142.zip \
1447+
"https://${s3BucketName}.${s3Endpoint}/regression/datalake/pipeline_data/iceberg_data_spark40_paimon142.zip"
14481448
else
14491449
wget -P "${ROOT}/docker-compose/iceberg" \
1450-
"https://${s3BucketName}.${s3Endpoint}/regression/datalake/pipeline_data/iceberg_data_spark40.zip"
1450+
"https://${s3BucketName}.${s3Endpoint}/regression/datalake/pipeline_data/iceberg_data_spark40_paimon142.zip"
14511451
fi
1452-
host_admin_cmd unzip iceberg_data_spark40.zip
1452+
host_admin_cmd unzip iceberg_data_spark40_paimon142.zip
14531453
host_admin_cmd mv iceberg_data data
1454-
host_admin_cmd rm -rf iceberg_data_spark40.zip
1454+
host_admin_cmd rm -rf iceberg_data_spark40_paimon142.zip
14551455
)
14561456
else
14571457
echo "${ICEBERG_DIR}/data exist, continue !"
Lines changed: 19 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1,30 +1,30 @@
11
-- This file is automatically generated. You should know what you did if you want to edit this
22
-- !show_partition_sales_by_date --
3-
sale_date=2024-01-15 sale_date 2 2216 1
4-
sale_date=2024-01-16 sale_date 2 4217 2
5-
sale_date=2024-01-17 sale_date 1 2118 1
3+
sale_date=2024-01-15 sale_date 2 2100 1
4+
sale_date=2024-01-16 sale_date 2 3985 2
5+
sale_date=2024-01-17 sale_date 1 2002 1
66

77
-- !show_partition_sales_by_region --
8-
region=China-Beijing region 1 2425 1
9-
region=Japan-Tokyo region 1 2420 1
10-
region=USA-California region 1 2454 1
8+
region=China-Beijing region 1 2295 1
9+
region=Japan-Tokyo region 1 2290 1
10+
region=USA-California region 1 2324 1
1111

1212
-- !show_partition_sales_by_date_region --
13-
sale_date=2024-01-15/region=China-Beijing sale_date,region 1 2627 1
14-
sale_date=2024-01-15/region=Japan-Tokyo sale_date,region 1 2614 1
15-
sale_date=2024-01-15/region=USA-California sale_date,region 1 2655 1
16-
sale_date=2024-01-16/region=China-Shanghai sale_date,region 1 2636 1
17-
sale_date=2024-01-16/region=Japan-Osaka sale_date,region 1 2636 1
18-
sale_date=2024-01-16/region=USA-New York sale_date,region 1 2643 1
13+
sale_date=2024-01-15/region=China-Beijing sale_date,region 1 2485 1
14+
sale_date=2024-01-15/region=Japan-Tokyo sale_date,region 1 2472 1
15+
sale_date=2024-01-15/region=USA-California sale_date,region 1 2513 1
16+
sale_date=2024-01-16/region=China-Shanghai sale_date,region 1 2494 1
17+
sale_date=2024-01-16/region=Japan-Osaka sale_date,region 1 2494 1
18+
sale_date=2024-01-16/region=USA-New York sale_date,region 1 2501 1
1919

2020
-- !show_partition_events_by_hour --
21-
hour_partition=2024-01-15-10 hour_partition 2 2361 1
22-
hour_partition=2024-01-15-11 hour_partition 2 4515 2
23-
hour_partition=2024-01-15-14 hour_partition 2 2371 1
21+
hour_partition=2024-01-15-10 hour_partition 2 2245 1
22+
hour_partition=2024-01-15-11 hour_partition 2 4283 2
23+
hour_partition=2024-01-15-14 hour_partition 2 2255 1
2424

2525
-- !show_partition_logs_by_date_hierarchy --
26-
year_val=2024/month_val=1/day_val=15 year_val,month_val,day_val 2 2841 1
27-
year_val=2024/month_val=1/day_val=16 year_val,month_val,day_val 2 5323 2
28-
year_val=2024/month_val=1/day_val=17 year_val,month_val,day_val 1 2658 1
29-
year_val=2024/month_val=2/day_val=1 year_val,month_val,day_val 1 2686 1
26+
year_val=2024/month_val=1/day_val=15 year_val,month_val,day_val 2 2701 1
27+
year_val=2024/month_val=1/day_val=16 year_val,month_val,day_val 2 5042 2
28+
year_val=2024/month_val=1/day_val=17 year_val,month_val,day_val 1 2517 1
29+
year_val=2024/month_val=2/day_val=1 year_val,month_val,day_val 1 2546 1
3030

‎regression-test/data/external_table_p0/paimon/write/test_paimon_write_external_paths.out‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,9 +32,18 @@ p2 4 4
3232
p3 5 4
3333
p3 6 3
3434

35+
-- !external_weight_robin --
36+
1 weight-1
37+
2 weight-2
38+
3 weight-3
39+
4 weight-4
40+
5 weight-5
41+
6 weight-6
42+
3543
-- !external_specific_fs --
3644
1 specific-1
3745
2 specific-2
3846

3947
-- !external_default_path --
4048
1 default-path
49+

‎regression-test/suites/external_table_p0/paimon/write/test_paimon_write_external_paths.groovy‎

Lines changed: 25 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -59,9 +59,19 @@ suite("test_paimon_write_external_paths", "p0,external,paimon") {
5959
'data-file.external-paths.strategy' = 'round-robin'
6060
);
6161
62-
-- The Spark fixture uses Paimon 1.3.1, which cannot create a table with the
63-
-- weight-robin strategy introduced in Paimon 1.4. Re-enable that strategy
64-
-- after the fixture has a compatible Spark connector.
62+
DROP TABLE IF EXISTS paimon.${dbName}.t_weight_robin;
63+
CREATE TABLE paimon.${dbName}.t_weight_robin (
64+
id INT, payload STRING
65+
) USING paimon
66+
TBLPROPERTIES (
67+
'primary-key' = 'id',
68+
'bucket' = '1',
69+
'write-only' = 'true',
70+
'target-file-size' = '1 kb',
71+
'data-file.external-paths' = '${pathRoot}/weight-a,${pathRoot}/weight-b',
72+
'data-file.external-paths.strategy' = 'weight-robin',
73+
'data-file.external-paths.weights' = '1,1'
74+
);
6575
6676
DROP TABLE IF EXISTS paimon.${dbName}.t_specific_fs;
6777
CREATE TABLE paimon.${dbName}.t_specific_fs (
@@ -197,6 +207,18 @@ suite("test_paimon_write_external_paths", "p0,external,paimon") {
197207
assertDorisSparkRows("external_round_robin_changed", "t_round_robin",
198208
"pt, id, length(payload)", "ORDER BY pt, id")
199209

210+
(1..6).each { id ->
211+
sql """INSERT INTO t_weight_robin VALUES (${id}, 'weight-${id}')"""
212+
}
213+
def weightedFiles = dataFiles("t_weight_robin")
214+
assertFalse(weightedFiles.isEmpty())
215+
assertTrue(weightedFiles.every {
216+
it.startsWith("${pathRoot}/weight-a/") ||
217+
it.startsWith("${pathRoot}/weight-b/")
218+
})
219+
assertDorisSparkRows("external_weight_robin", "t_weight_robin",
220+
"id, payload", "ORDER BY id")
221+
200222
sql """INSERT INTO t_specific_fs VALUES (1, 'specific-1')"""
201223
sql """INSERT INTO t_specific_fs VALUES (2, 'specific-2')"""
202224
def specificFiles = dataFiles("t_specific_fs")

0 commit comments

Comments
 (0)