Automate the release guide - Take 2 - Github workflows#2383
Merged
snazy merged 126 commits intoapache:mainfrom Nov 18, 2025
Merged
Automate the release guide - Take 2 - Github workflows#2383snazy merged 126 commits intoapache:mainfrom
snazy merged 126 commits intoapache:mainfrom
Conversation
snazy
reviewed
Nov 8, 2025
Member
snazy
left a comment
snazy
left a comment
There was a problem hiding this comment.
Created a PR against your branch.
It contains:
- the one suggestion
- fixes for GH actions
- proposal for the helm-version update
We have to use hash-references (see Actions Policy. And also only use action+version that are allowed by the policy, the currently approved patterns are here. Otherwise the workflows won't run.
I ran most of the commands in the workflows locally, leaving out the git pushes and svn commits and that worked, except for two things:
- Helm version update didn't work
- Helm GPG signing didn't work for me locally. Would you mind cross-checking?
Otherwise this looks really really good.
I'd appreciate if someone else could take a look at the workflows and scripts just to double-check that there are no oversights.
|
|
||
| exec_process cd helm | ||
| exec_process helm package polaris | ||
| exec_process helm gpg sign polaris-${version_without_rc}.tgz |
Member
There was a problem hiding this comment.
Could you cross-check this one?
It failed for me locally:
$ exec_process helm gpg sign polaris-2.3.0-incubating-SNAPSHOT.tgz
DEBUG: Executing 'helm gpg sign polaris-2.3.0-incubating-SNAPSHOT.tgz'
Signing polaris-2.3.0-incubating-SNAPSHOT.tgz
tar: Pattern matching characters used in file names
tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning
tar: */Chart.yaml: Not found in archive
tar: Exiting with failure status due to previous errors
Error: plugin "gpg" exited with error
The tarball contains these files (abbreviated list):
polaris/Chart.yaml
polaris/values.yaml
...
Contributor
Author
There was a problem hiding this comment.
I wonder if it is linked to a tar version? I do not have any warning on my end.
$ tar tzf polaris-99.98.97-incubating.tgz | head -n 5
polaris/Chart.yaml
polaris/values.yaml
polaris/templates/_helpers.tpl
polaris/templates/configmap.yaml
polaris/templates/deployment.yaml
$ helm gpg sign polaris-99.98.97-incubating.tgz
Signing polaris-99.98.97-incubating.tgz
$ echo $?
0
Member
There was a problem hiding this comment.
I think this could work with helm package --sign and we can get rid of the helm-gpg-plugin, which is unmaintained for 5 years now.
Member
There was a problem hiding this comment.
@jbonofre could you provide the details how you generated generated the gpg key? Then we can do another round of testing simulating the whole roundtrip (key -> secret -> workflow -> get artifacts -> verify) to be sure that it works.
Releasey workflow review
snazy
reviewed
Nov 10, 2025
Member
snazy
left a comment
snazy
left a comment
There was a problem hiding this comment.
Woot! Very close!
The only thing that worries me a bit is the helm gpg sign/tar thing.
Let me see if I can hack a dummy repo/workflow to see whether this works or not in GH.
adutra
previously approved these changes
Nov 10, 2025
github-project-automation
bot
moved this from PRs In Progress
to Ready to merge
in Basic Kanban Board
snazy
reviewed
Nov 13, 2025
Co-authored-by: Robert Stupp <[email protected]>
snazy
previously approved these changes
Nov 13, 2025
Member
snazy
left a comment
snazy
left a comment
There was a problem hiding this comment.
This is really a big improvement!
The change now LGTM, so +1!
We just need to do a final safety cross-check with a dummy generated GPG key built in the same way as the "real" GPG key. But after that we should merge this PR.
Then we can test this (nearly) end-to-end and clean up the release-test artifacts. What we cannot test is the final release publication, because there is no CI for this use case but the good old try-n-error approach with some fingers-crossed.
Contributor
Author
|
@snazy There were a couple of things that needed fixed, like the fact that That last one correctly created a staging repository on Nexus (see screenshot below). It also successfully pushed artifacts to dist dev. I verified that those artifacts were in SVN and deleted them. I will drop the staging repository next. 
The only thing that is not tested is the last release workflow to publish artifacts after a release is successful. We will have to wait for the official 1.3.0 release to test it given its real-life impacts. |
snazy
reviewed
Nov 18, 2025
Member
snazy
left a comment
snazy
left a comment
There was a problem hiding this comment.
Nice work!
The GH WF UI is playing tricks...
Although all jobs depend on the Prerequisite ... job and those are ran in parallel, the UI gives the illusion that those are run sequentially.
Another (dummy) job that depends on the 3 parallel ones could help with that. But I don't insist on doing that in this PR.
This reverts commit bf235e7.
Contributor
Author
|
Run with the manual |
snazy
approved these changes
Nov 18, 2025
Member
snazy
left a comment
snazy
left a comment
There was a problem hiding this comment.
Not much to comment anymore... +1 :)
github-project-automation
bot
moved this from Ready to merge
to Done
in Basic Kanban Board
snazy
added a commit
to snazy/polaris
that referenced
this pull request
Feb 11, 2026
* Add loadEntities batch call and rename listFullEntities (apache#2508) * Add loadEntities batch call and rename listFullEntities * Changed batch call to implement loadResolvedEntities instead * Add loadResolvedEntities by id and entity cache support * Add additional test for loadResolvedEntities by id * Added additional test and updated comments in EntityCache interface * Add additional constructor to ResolvedEntitiesResult * Fixed unused method reference * Removed loadResolvedEntities method with lookup record param * Pulled out toResolvedPolarisEntity method per PR comment * Core: made the ARN role regex more generic (apache#3005) * fix(docs): Generify S3 index page (apache#2997) * Remove the mention of "cloud" since not all possible storage options are provided in "cloud". * Avoid listing specific child pages in the doc test. Rely on Hugo-general index (on the left-hand pane). --------- Co-authored-by: Alexandre Dutra <[email protected]> * fix(deps): update dependency io.prometheus:prometheus-metrics-exporter-servlet-jakarta to v1.4.3 (apache#3009) * fix(deps): update dependency com.google.cloud:google-cloud-storage-bom to v2.60.0 (apache#3011) * fix(deps): update dependency io.opentelemetry:opentelemetry-bom to v1.56.0 (apache#3012) * fix(deps): update dependency com.adobe.testing:s3mock-testcontainers to v4.10.0 (apache#3010) * fix(deps): update dependency org.agrona:agrona to v2.3.2 (apache#3014) * fix(deps): update quarkus platform and group to v3.29.2 (apache#3013) * chore(deps): update dependency pre-commit to v4.4.0 (apache#3015) * fix(deps): update dependency software.amazon.awssdk:bom to v2.38.2 (apache#3019) * Add test for TracingFilter (apache#2847) * NoSQL: Add (micro-ish) benchmarks (apache#3006) A project for JMH based benchmarks against NoSQL persistence. * Helm chart: include configmap checksum in deployment annotations (apache#3023) * fix(deps): update dependency ch.qos.logback:logback-classic to v1.5.21 (apache#3025) * NoSQL: Realms handling (apache#3007) Introduces handling for realms including realm-state management/transition. The `RealmStore` implementation for NoSQL depends on CDI components, coming in a follo-up PR. * Rename AccessConfig and AccessConfigProvider for clarity (apache#2883) * rename AccessConfig for clarity * rename getStorageAccessConfig() and add javadoc * Refactor: improve and clean up Dockerfiles (apache#2957) * Refactor: improve and clean up Dockerfiles * Refactor: improve and clean up Dockerfiles * Refactor: improve and clean up Dockerfiles * Refactor: improve and clean up Dockerfiles * Refactor: improve and clean up Dockerfiles * Refactor: improve and clean up Dockerfiles * Make StorageAccessConfigProvider request-scoped (apache#2974) - add `StorageCredentialsVendor` as request-scoped wrapper around `PolarisCredentialVendor` - make `FileIOFactory` request-scoped - make `TaskFileIOSupplier` request-scoped * Increase javadoc visibility in `nosql/realms` (apache#3029) This is to fix javadoc error: `No public or protected classes found to document` * NoSQL: Add correctness tests (apache#3027) Verifies the correctness of concurrent commits, and big index handling. These tests are intentionally _not_ part of the base-backend test suite for two reasons: 1. These tests do not run against the `Backend` interface but the `Persistence` interface, including commit and index logic. 2. These tests are intended to be runnable against a custom provisioned database cluster, not just tiny-ish test containers. * NoSQL: Add maintenance API, SPI (apache#3028) Maintenance operations include a bunch of tasks that are regularly executed against a backend database. Types of maintenance operations include: * Purging unreferenced objects and references within a catalog * Purging whole catalogs that are marked to be purged * Purging whole realms that are marked to be purged Implementation added in a follow-up PR. * Embrace request-scoped TokenBroker (apache#3024) * Embrace request-scoped TokenBroker `TokenBroker` and `CallContext` are both request-scoped, so instead of passing the former into the latter, we can do this via the `TokenBrokerFactory` and thus simplify the `TokenBroker` interface. * fix(deps): update dependency io.smallrye:jandex to v3.5.2 (apache#3032) * Fix monkey patching (apache#3016) * chore(deps): update quay.io/keycloak/keycloak docker tag to v26.4.5 (apache#3034) * chore(deps): update registry.access.redhat.com/ubi9/openjdk-21-runtime docker tag to v1.23-6.1762870925 (apache#3053) * fix(deps): update dependency org.testcontainers:testcontainers-bom to v2.0.2 (apache#3054) * chore(deps): update postgres docker tag to v18.1 (apache#3055) * Add Polaris Community Meeting 2025-11-13 (apache#3060) * Site: Rename menu "downloads" to "releases" (apache#2928) * Update dependency software.amazon.awssdk:bom to v2.38.7 (apache#3065) * Test-fix: Cleanup OPA test container on stop (apache#3041) Quarkus takes care of reusing a test-resource across tests. The current behavior leaves the container around. Plus some nit-fixes (deprecation + local var) * Update dependency org.apache.commons:commons-lang3 to v3.20.0 (apache#3063) * Build: ensure LICENSE/NOTICE is in all jars, always add pom-files to all jars (apache#3057) There are a some inconsistencies between the different kinds of jars and the included information: * LICENSE/NOTICE files are present in the "main" jar and in the sources jar, but not in the javadoc jar. * The Maven pom.xml and pom.properties files are only present for release builds or when explicitly requested. * "Additional" jar-manifest attributes that are only present in release builds. This change fixes the three mentioned issues: * Always include pom.xml and pom.properties in the built jar files. * Always include the additional jar-manifest attributes, except the Git information, which would otherwise render the Gradle build cache ineffective. * Include pom.xml + pom.properties + license/notice in literally all jar files. The Gradle logic to include the license+notice+pom files has been simplified as well. * Remove unused polarisEventListener field from IcebergCatalogHandler (apache#3045) it was added in c3f5001 but then its only usage was removed in d03c717 * fix(deps): update quarkus platform and group to v3.29.3 (apache#3052) * Site: Add Open Policy Agent (OPA) as External Policy Decision Point (apache#3030) Doc PR following up the introduction of OpaPolarisAuthorizer: apache#2680 * OPA: Tackle deprecation warnings (apache#3042) Instead of suppressing the deprecations, this change updates the code a little bit to remove the mocks (except to create a non-nullable parameter). * Use POJOs for OPA JSON schema construction and publish schema (apache#3031) Co-authored-by: Robert Stupp <[email protected]> * Use CDI for more test setups (apache#3040) this avoids a bunch of redundant manual setup. the important parts are establishing a `RealmContext` by calling `QuarkusMock.installMockForType` and then populating `polarisContext` from the injected `CallContext`. * fix(deps): update dependency com.github.dasniko:testcontainers-keycloak to v4 (apache#3070) * chore(deps): update actions/checkout digest to 93cb6ef (apache#3068) * OPA: Fail fast when OPA bearer token file is unreadable (apache#3062) * fix(deps): update immutables to v2.11.7 (apache#3072) * Skip Hugo Site workflow on forks (apache#3056) Forks usually don't have the "versioned-docs" tag and thus PRs against forks or rebasing the main branch on a fork currently always causes workflow failures. * Fix warnings around TransactionWorkspaceMetaStoreManager (apache#3044) - dont return `null` for interface methods that are `@Nonnull` - fix wrong method name parameters - dont annotate void methods as `@Nonnull` * NoSQL: Add CDI/common+testing + necessary nosql-store implementations (apache#3035) Adds common and test-specific CDI functionality. Requires the NoSQL store implementations `:polaris-persistence-nosql-realms-store-nosql` and `:polaris-nodes-store-nosql`. Those modules have cross-project dependencies for test purposes, hence those are all contained in this PR. CDI for Quarkus will be added in a follow-up. * Automate the release guide - Take 2 - Github workflows (apache#2383) The release automation is simplified to four GitHub workflows that just require the really mandatory user input: the version number. 1. workflow: Trigger the creation of the release branch 2. workflow: Upgrade the release branch with the version and build the the final change-log for that version 3. workflow: Build the RC artifacts from the release branch and push those to the various staging repositories 4. workflow: Eventually release the artifacts. See also the [email announcement](https://lists.apache.org/thread/d0smz07gnr509yj5dc6omo3cvkf1pnh7). --------- Co-authored-by: Robert Stupp <[email protected]> * Update actions/checkout digest to 93cb6ef (apache#3082) * NoSQL: adapt to conflicting changes in main * Last merged commit 8ccddc5 --------- Co-authored-by: Michael Collado <[email protected]> Co-authored-by: cccs-cat001 <[email protected]> Co-authored-by: Dmitri Bourlatchkov <[email protected]> Co-authored-by: Alexandre Dutra <[email protected]> Co-authored-by: Mend Renovate <[email protected]> Co-authored-by: Nuoya Jiang <[email protected]> Co-authored-by: Yong Zheng <[email protected]> Co-authored-by: Christopher Lambert <[email protected]> Co-authored-by: JB Onofré <[email protected]> Co-authored-by: Yufei Gu <[email protected]> Co-authored-by: Sung Yun <[email protected]> Co-authored-by: Pierre Laporte <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is the next iteration of release automation. It builds on top of #2156 and reuses the common bash libraries that were defined.
Differences from the initial PR
DOCKERHUB_USERNAMEandDOCKERHUB_TOKEN- the credentials that can be used to push Docker images to DockerhubGPG_PRIVATE_KEYandGPG_PASSPHRASE- the ASCII armored private key that should be used to sign artifacts and its associated passphrase. The associated public key is assumed to be added to theKEYSfile prior to being entered here.APACHE_USERNAMEandAPACHE_PASSWORD- the credentials that can be used to connect to the ASF SVN server as well as to the ASF Nexus server.Similarities with the initial PR
This PR builds on the same assumptions from the initial PR:
Remaining known-unknowns
I can see that
apache/polarishas aNightly buildgithub workflow that publishes snapshots every night to the Apache Nexus repository. In this workflow's definition, I can find references two Nexus credentials (secrets.NEXUS_USERandsecrets.NEXUS_PW). However I cannot find any such secret defined on https://github.com/apache/polaris/settings/secrets/actions, nor am I sure whether those credentials can be used to interact with the ASF SVN server. Some clarification is needed to ensure proper credentials configuration.Example runs
I have used this PR to simulate the release of Polaris 99.98.97-incubating-rc1 on my own fork. No upload was performed, this is just to prove out that things should be working as expected. You can find links to the following workflow executions:
release/99.98.97-incubating. It was run withdry-run=0so that the release branch was actually created.99.98.97-incubating, update the changelog and push the RC1 tag. It was also run withdry-run=0so that the modifications were actually performed (on my fork only)dry-run=1so that no interaction with any ASF server happened.dry-run=1to check the commands that would be executed, if an actual release was to be performed. We can see the binaries and Helm Charts would be built, signed, checksum'ed and publish to ASF SVN and Nexus repositories. Docker images would be built but not published.Next steps
If this PR is approved, it should be possible to use the
Create Release BranchandUpdate Release Candidateworkflows to respectively cut the release branch and set the RC version as well as create the RC tag.Recommendations for reviewers
releasey/release-process-flowchart.pngflowchart first. It shows how each workflow participates to the overall release process. It should substantially help make sense of the Github workflows for the rest of the review.