Skip to content

Commit 1032755

Browse files
morningmanclaude
andcommitted
[chore](docker) Re-extract the iceberg fixture when it lacks a mounted jar
### What problem does this PR solve? Issue Number: None Related PR: #66346 Problem Summary: Review follow-up. run-thirdparties-docker.sh extracts the iceberg fixture archive into docker-compose/iceberg/data once and reuses that directory on later runs. After this PR switches the archive and the mounted Paimon jars to 1.4.2, an environment that still has the data directory from the old archive skips the download, the 1.4.2 jars it mounts do not exist, Compose creates empty directories in their place, and Spark cannot load the Paimon catalog. CI starts from a clean environment and is not affected. start_iceberg now checks every jar iceberg.yaml mounts from data/input/jars before deciding whether to download. If one is missing (or is a directory Compose created), the stale data directory is removed and the current archive is extracted again, the same as a fresh install. Later jar changes in iceberg.yaml are picked up the same way. ### Release note None ### Check List (For Author) - Test: Manual test (bash -n; the new check run on four layouts: all jars present, a data directory from the old archive, a jar path Compose created as a directory, and no data directory) - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 318fe23 commit 1032755

1 file changed

Lines changed: 13 additions & 0 deletions

File tree

‎docker/thirdparties/run-thirdparties-docker.sh‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1437,6 +1437,19 @@ start_iceberg() {
14371437
register_stack_metadata "iceberg" "${ROOT}/docker-compose/iceberg/iceberg.yaml" "${ROOT}/docker-compose/iceberg/iceberg.env"
14381438
compose_down_stack "${ROOT}/docker-compose/iceberg/iceberg.yaml" "${ROOT}/docker-compose/iceberg/iceberg.env" --remove-orphans
14391439
if [[ "${STOP}" -ne 1 ]]; then
1440+
# data/ is extracted from the fixture archive once and then reused. A data/ from an older archive lacks
1441+
# jars that iceberg.yaml now mounts, and Compose would create empty directories in their place, so
1442+
# start over from the current archive.
1443+
if [[ -d "${ICEBERG_DIR}/data" ]]; then
1444+
local jar
1445+
for jar in $(grep -o '\./data/input/jars/[^:]*\.jar' "${ICEBERG_DIR}/iceberg.yaml"); do
1446+
if [[ ! -f "${ICEBERG_DIR}/${jar#./}" ]]; then
1447+
echo "${ICEBERG_DIR}/${jar#./} is missing, ${ICEBERG_DIR}/data is from an older archive"
1448+
host_admin_cmd rm -rf "${ICEBERG_DIR:?}/data"
1449+
break
1450+
fi
1451+
done
1452+
fi
14401453
if [[ ! -d "${ICEBERG_DIR}/data" ]]; then
14411454
echo "${ICEBERG_DIR}/data does not exist"
14421455
(

0 commit comments

Comments
 (0)