Repository navigation
Open-API: Stage REST fixture runtime JARs for Docker - #16214
Merged
Merged
Conversation
kevinjqliu
force-pushed
the
kevinjqliu/open-api
branch
from
May 6, 2026 22:06
9261ef1 to
f2ee703
Compare
kevinjqliu
changed the title
kevinjqliu
marked this pull request as ready for review
May 7, 2026 01:49
Contributor
Author
|
this is ready for first round of review. Please take a look at once those are good, please check |
Member
|
I'm wondering whether we need all those jars and bundles for docker image and integration tests in iceberg-python and iceberg-rust. |
kevinjqliu
force-pushed
the
kevinjqliu/open-api
branch
from
June 7, 2026 23:40
adfc237 to
31831d0
Compare
kevinjqliu
force-pushed
the
kevinjqliu/open-api
branch
from
June 8, 2026 00:49
40d608b to
5d3b794
Compare
kevinjqliu
changed the title
kevinjqliu
force-pushed
the
kevinjqliu/open-api
branch
4 times, most recently
from
June 8, 2026 21:27
a0e2845 to
33b9c5c
Compare
kevinjqliu
changed the title
kevinjqliu
changed the title
kevinjqliu
force-pushed
the
kevinjqliu/open-api
branch
from
June 8, 2026 21:31
33b9c5c to
f291147
Compare
kevinjqliu
changed the title
kevinjqliu
force-pushed
the
kevinjqliu/open-api
branch
from
June 27, 2026 20:27
899981e to
1cd4a96
Compare
Contributor
Author
|
After this, I will follow up with more PRs to
|
…rom staged classpath Replaces the iceberg-open-api shadowJar with a Sync task that lays out the REST catalog test-fixtures runtime as a directory of jars plus LICENSE/NOTICE under build/iceberg-rest-server/. The iceberg-rest-fixture Docker image copies that directory and launches the server via java -cp ".../libs/*" org.apache.iceberg.rest.RESTCatalogServer. The publish-iceberg-rest-fixture-docker and cve-scan workflows are updated to run the new installRESTServer task; the open-api CVE scan entry now scans every dependency jar individually.
Document the required cloud bundles and explicit staging lifecycle, reuse archive metadata handling for test fixtures, and keep dependency pruning and the license audit separate from the packaging change. Generated-by: GitHub Copilot Co-authored-by: Copilot App <[email protected]>
kevinjqliu
force-pushed
the
kevinjqliu/open-api
branch
from
November 10, 2026 21:01
d54d785 to
50f0b05
Compare
kevinjqliu
changed the title
Reuse cloud bundle dependency declarations as loose runtime JARs, following the Kafka Connect distribution, rather than staging shaded bundles alongside their unshaded dependencies. Cover unique SDK class definitions and ADLS FileIO initialization to prevent the Azure linkage regression. Generated-by: GitHub Copilot Co-authored-by: Copilot App <[email protected]>
kevinjqliu
marked this pull request as draft
November 10, 2026 22:57
Revert eb20d81 at the user’s request, restoring the prior REST fixture packaging. Generated-by: GitHub Copilot Co-authored-by: Copilot App <[email protected]>
Fokko
self-requested a review
November 11, 2026 08:41
Fokko
reviewed
Nov 11, 2026
Contributor
There was a problem hiding this comment.
Thanks @kevinjqliu! Nice follow-up on the agreed packaging plan 🚀 I checked out the branch and ran the staging task locally: 274 jars under libs/, Guava 33.7.2, LICENSE/NOTICE alongside, and the Kafka Connect-style SDK copying matches the existing pattern in kafka-connect/build.gradle.
kevinjqliu
changed the title
Scope stageRESTFixtureRuntime to staging the runtime JARs under open-api/build/rest-fixture-runtime, drop LICENSE/NOTICE changes for follow-up PRs, simplify cve-scan JAR copying, and add the hadoop-shaded-guava Trivy ignore. Generated-by: GitHub Copilot Co-authored-by: Copilot App <[email protected]>
kevinjqliu
commented
Nov 11, 2026
| # | ||
| # Trivy ignore file for the OpenAPI test-fixtures runtime. | ||
| # | ||
| vulnerabilities: |
Contributor
Author
There was a problem hiding this comment.
to pass cve scan, same as
iceberg/.github/trivyignore/kafka-connect-runtime.trivyignore.yaml
Lines 20 to 25 in 83bf0ff
| unzip ${{ matrix.scan-path }}/*.zip -d /tmp/cve-scan | ||
| else | ||
| cp ${{ matrix.scan-path }}/iceberg-${{ matrix.distribution }}-*.jar /tmp/cve-scan/ | ||
| cp ${{ matrix.scan-path }}/*.jar /tmp/cve-scan/ |
Contributor
Author
There was a problem hiding this comment.
so it matches the jars produced by :iceberg-open-api:stageRESTFixtureRuntime
| # Copy runtime files directly to the target location | ||
| COPY --chown=iceberg:iceberg open-api/build/libs/iceberg-open-api-test-fixtures-runtime-*.jar /usr/lib/iceberg-rest/iceberg-rest-adapter.jar | ||
| # Copy runtime JARs staged by ./gradlew :iceberg-open-api:stageRESTFixtureRuntime | ||
| COPY --chown=iceberg:iceberg open-api/build/rest-fixture-runtime/ /usr/lib/iceberg-rest/libs/ |
Contributor
Author
There was a problem hiding this comment.
this is the glue.
moves the jars built by :iceberg-open-api:stageRESTFixtureRuntime and saved in open-api/build/rest-fixture-runtime/ to /usr/lib/iceberg-rest/libs/ which docker/iceberg-rest-fixture/entrypoint.sh then reference
| group = 'build' | ||
| into layout.buildDirectory.dir('rest-fixture-runtime') | ||
| from configurations.testFixturesRuntimeClasspath | ||
| from tasks.named('testFixturesJar') |
Contributor
Author
There was a problem hiding this comment.
im purposely leaving out pruning deps. will do that in a follow up PR
kevinjqliu
marked this pull request as ready for review
November 11, 2026 19:01
Fokko
approved these changes
Nov 11, 2026
Contributor
Author
|
Thanks @Fokko, I'll follow up with more PRs to prune what goes into the docker image and then the LICENSE/NOTICE files |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Instead of bundling everything into a shaded uber-JAR (
:iceberg-open-api:shadowJar), this PR stagestestFixturesJarand its runtime dependencies as individual JARs via:iceberg-open-api:stageRESTFixtureRuntime, which theiceberg-rest-fixtureDocker image copies into/usr/lib/iceberg-rest/libs/and runs on the classpath (libs/*).For context, Docker publishing was paused over
LICENSE/NOTICEissues (see #18421 and #18877).This is the first step of the plan to fix
iceberg-rest-fixturepackaging and licensing.Follow-up PRs will prune the runtime dependencies (and drop the redundant cloud bundle fat JARs), add
runtime-deps.txt, and reconcileopen-api/LICENSEandopen-api/NOTICE.What goes into the Docker Image
:iceberg-open-api:stageRESTFixtureRuntime's output. Follow up PR will trim this.Currently stages 273 JARs into
open-api/build/rest-fixture-runtime:iceberg-api,iceberg-common,iceberg-core,iceberg-core-tests,iceberg-aws,iceberg-aws-bundle,iceberg-azure,iceberg-azure-bundle,iceberg-gcp,iceberg-gcp-bundle,iceberg-bigquery,iceberg-bundled-guava,iceberg-open-api-test-fixturesFull list of 273 staged JARs
AI Disclosure
gemini-4-argon)