[Issue 3209] Surface Intermediate Data as part of Polaris Events by adnanhemani · Pull Request #3456 · apache/polaris

adnanhemani · 2026-01-16T01:58:36Z

Adds a RequestScoped annotation to AttributeMap, which allows us to inject this bean into intermediary business logic and be able to patch data from there back out to the PolarisEvent.

@dimas-b had a great idea that we may be able to retire the Delegator model eventually by setting a "closing" hook on the RequestScoped bean - but for simplicity, I am introducing this smaller change that will solve the GH issue linked first and will investigate that approach later.

Closes #3209.

Checklist

🛡️ Don't disclose security issues! (contact [email protected])
🔗 Clearly explained why the changes are needed, or linked related issues: Fixes Implement sending POJOs not within the Response to Events #3209
🧪 Added/updated tests with good coverage, or manually tested (and explained how)
💡 Added comments for complex logic
🧾 Updated CHANGELOG.md (if needed)
[N/A] 📚 Updated documentation in site/content/in-dev/unreleased (if needed)

evindj

overall looks good to me, I left some questions mostly for my own understanding of the commit and event flow.

...java/org/apache/polaris/service/catalog/iceberg/IcebergRestCatalogEventServiceDelegator.java

...ice/src/test/java/org/apache/polaris/service/catalog/iceberg/CommitTransactionEventTest.java

dimas-b

Thanks, @adnanhemani ! The general direction of this PR LGTM... some comments below.

.../service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java

runtime/service/src/main/java/org/apache/polaris/service/events/EventAttributeMap.java

CHANGELOG.md

runtime/service/src/main/java/org/apache/polaris/service/events/EventAttributeMap.java

.../service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java

adutra · 2026-01-20T16:37:23Z

...java/org/apache/polaris/service/catalog/iceberg/IcebergRestCatalogEventServiceDelegator.java

Shouldn't we change all these new AttributeMap() occurrences to use the injected attribute map instead?

I had a different thought - but would like to hear your thoughts on this as well.

I was using AttributeMap to inject into the deeper parts of the codebase - but then using that to populate the final AttributeMap that gets passed to the Event Listener. The chief concern was to ensure that the After event attributes don't accidentally become strictly a super-set of the Before events attributes - and that cleaner attribute maps going into the event listener may help with filtering concerns that we will need to tackle in order to enable all events going to their eventual sink.

WDYT?

EventAttributeMap is a proper CDI bean now. It should be ok to rely on CDI to create and pass around (inject) one instance of it per request.

If we want to distinguish before/after data, we could do that by code / methods inside EventAttributeMap. As I commented in another thread, I see value in evolving this class to become a rich and typed container of event-related data for each request (more than a simple key-value map).

@dimas-b, if you're okay with it, let's handle this in a separate PR then? I think we may need to iterate on this idea a few times before we get it right - but this PR is attempting to do something very related but slightly orthogonal and IMO enough for a first take at this functionality.

I do not mind incremental work here, but please sync with @adutra in terms of timing.

...java/org/apache/polaris/service/catalog/iceberg/IcebergRestCatalogEventServiceDelegator.java

.../service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java

runtime/service/src/main/java/org/apache/polaris/service/events/EventAttributes.java

adutra

The PR is almost good to go imho, we just need to settle on the LOAD_TABLE_RESPONSES attribute.

.../service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java

runtime/service/src/main/java/org/apache/polaris/service/events/EventAttributes.java

adutra · 2026-01-26T08:45:06Z

runtime/service/src/main/java/org/apache/polaris/service/events/EventAttributes.java

+  public static final AttributeKey<List<TableMetadata>> TABLE_METADATA =
+      new AttributeKey<>("table_metadatas", new TypeToken<List<TableMetadata>>() {});


The field name should reflect the attribute name: TABLE_METADATAS.

There seems to be an inconsistency here: sometimes this attribute seems to be legitimately collecting a list of metadatas, cf. IcebergCatalogHandler. But sometimes, it's collecting a singleton list of a unique table metadata, cf. IcebergCatalogEventServiceDelegator.

I think it would be better to distinguish two attributes: TABLE_METADATA and TABLE_METADATAS. (I understand that the latter may become a sort of "internal" attribute that is never exposed to listeners – we could maybe declare those separately if we are concerned about that.)

Sure, I'm glad to do this. I was trying to make everything fit within TABLE_METADATAS due to the similar comment that Dmitri had pointed out above. But I will make a comment for this for now so we can move past and revisit if there is a better way to do this later.

runtime/service/src/main/java/org/apache/polaris/service/events/EventAttributes.java

...java/org/apache/polaris/service/catalog/iceberg/IcebergRestCatalogEventServiceDelegator.java

dimas-b

Thanks, @adnanhemani ! The PR LGTM 👍

adnanhemani · 2026-01-29T05:26:21Z

@adutra @dimas-b - I had to fix the merge conflicts :( Can you both please re-approve and merge?

HonahX

LGTM!

Thanks @adnanhemani for the work and @evindj @dimas-b @adutra for reviewing! Looks like all the comments are either resolved or planned to resolve incrementally. Merging.

@dimas-b

…che#3456) Adds a RequestScoped annotation to AttributeMap, which allows us to inject this bean into intermediary business logic and be able to patch data from there back out to the PolarisEvent. @dimas-b had a great idea that we may be able to retire the Delegator model eventually by setting a "closing" hook on the RequestScoped bean - but for simplicity, I am introducing this smaller change that will solve the GH issue linked first and will investigate that approach later.

@dimas-b

* Nit: fix shell-check warnings for await-s3.sh (apache#3593) * Nit: add upgrade notes for develocity plugin (apache#3599) * fix(deps): update dependency org.apache.hive:hive-exec to v2.3.10 (apache#3580) * fix(deps): update dependency com.google.cloud:google-cloud-storage-bom to v2.62.1 (apache#3600) * Change quick start example from MinIO to RustFS backend (apache#3586) * chore(deps): update gradle/actions digest to f29f5a9 (apache#3602) * Simplify getting started example for Ceph (apache#3587) * Simplify getting started example for Ceph * Simplify getting started example for Ceph * fix(deps): update dependency io.smallrye.config:smallrye-config-core to v3.16.0 (apache#3604) * chore(deps): update docker.io/mongo docker tag to v8.2.4 (apache#3606) * fix(deps): update dependency net.ltgt.gradle:gradle-errorprone-plugin to v5 (apache#3605) * chore(deps): update plugin com.gradle.develocity to v4.3.2 (apache#3598) * chore(deps): update localstack/localstack docker tag to v4.13 (apache#3609) * Guides: use `sql` code block (apache#3590) Replaces the "anonymous" code blocks with `sql` code blocks, which is easier to "copy + paste" for users. * Spark guide: document the launch command in the Markdown (apache#3592) Replaces the "invisible" launch-docker.sh script with a "visible" `shell` code block, so the commands are immediately visible to users. * Minio guide: fix service dependencies (apache#3594) Use the "long" options for compose service dependencies, and adds the explicit option `restart: no` option for setup tasks, which is necessary to let docker-compose correctly interpret the termination of such tasks. * Ozone guide: Fix service dependencies (apache#3595) Use the "long" options for compose service dependencies, add healthchecks, uses more healthcheck-retries for slow-ish machines, and adds the explicit option `restart: no` option for setup tasks, which is necessary to let docker-compose correctly interpret the termination of such tasks. * Rustfs guide: fix service dependencies (apache#3596) Just adds a necessary service dependency * Telemetry guide: fix setup-polaris service (apache#3597) Fixes the polaris-setup service to use the "working" way/syntax, in alignment to other guides. * Site-workflow: add `workflow_call` trigger (apache#3520) This trigger allows the Hugo Site publication workflow to be called from the `versioned-docs` branch. This change also introduces a concurrency group to prevent concurrent site publications. Contributed to apache#3516 Co-authored-by: Dmitri Bourlatchkov <[email protected]> * [Issue 3209] Surface Intermediate Data as part of Polaris Events (apache#3456) Adds a RequestScoped annotation to AttributeMap, which allows us to inject this bean into intermediary business logic and be able to patch data from there back out to the PolarisEvent. @dimas-b had a great idea that we may be able to retire the Delegator model eventually by setting a "closing" hook on the RequestScoped bean - but for simplicity, I am introducing this smaller change that will solve the GH issue linked first and will investigate that approach later. * Getting-Started: Jaeger upgraded from V1 to V2 (apache#3603) * Jaeger V2 * Jaeger V2 * Update Gradle to 9.3.1 (apache#3615) * Last merged commit 90ba09c --------- Co-authored-by: Mend Renovate <[email protected]> Co-authored-by: Yong Zheng <[email protected]> Co-authored-by: Dmitri Bourlatchkov <[email protected]> Co-authored-by: Adnan Hemani <[email protected]>

github-project-automation bot added this to Basic Kanban Board Jan 16, 2026

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board Jan 16, 2026

adnanhemani marked this pull request as ready for review January 16, 2026 02:18

adnanhemani mentioned this pull request Jan 16, 2026

[DO NOT MERGE] POC for Using Intermediate Data as part of Events #3391

Closed

6 tasks

evindj reviewed Jan 16, 2026

View reviewed changes

dimas-b requested a review from adutra January 16, 2026 19:33

dimas-b reviewed Jan 16, 2026

View reviewed changes

.../service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java Outdated Show resolved Hide resolved

runtime/service/src/main/java/org/apache/polaris/service/events/EventAttributeMap.java Show resolved Hide resolved

CHANGELOG.md Outdated Show resolved Hide resolved

adutra reviewed Jan 20, 2026

View reviewed changes

dimas-b reviewed Jan 21, 2026

View reviewed changes

runtime/service/src/main/java/org/apache/polaris/service/events/EventAttributes.java Outdated Show resolved Hide resolved

adutra reviewed Jan 23, 2026

View reviewed changes

.../service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java Outdated Show resolved Hide resolved

adnanhemani added 7 commits January 23, 2026 14:10

[Issue 3209] Surface Intermediate Data as part of Polaris Events

33a7aec

spotless and CHANGELOG

116e563

Revision based on comments from @adutra and @dimas-b

de9f411

Addressing review comments from @dimas-b

0ee94cc

fix failing test

3100572

fix test failures

f21aa0e

address comments from @adutra and @dimas-b

344a434

adnanhemani force-pushed the ahemani/solve_issue_3209 branch from 478e8df to 344a434 Compare January 23, 2026 22:46

fix build

8d0835d

adutra reviewed Jan 26, 2026

View reviewed changes

address comments from @adutra

3e5b832

adutra previously approved these changes Jan 27, 2026

View reviewed changes

github-project-automation bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Jan 27, 2026

dimas-b previously approved these changes Jan 27, 2026

View reviewed changes

Merge branch 'main' into ahemani/solve_issue_3209

256ba24

adnanhemani dismissed stale reviews from dimas-b and adutra via 256ba24 January 29, 2026 05:25

adutra approved these changes Jan 29, 2026

View reviewed changes

HonahX approved these changes Jan 29, 2026

View reviewed changes

HonahX merged commit baf01a5 into apache:main Jan 29, 2026
15 checks passed

github-project-automation bot moved this from Ready to merge to Done in Basic Kanban Board Jan 29, 2026

adnanhemani deleted the ahemani/solve_issue_3209 branch January 29, 2026 21:46

		public static final AttributeKey<List<TableMetadata>> TABLE_METADATA =
		new AttributeKey<>("table_metadatas", new TypeToken<List<TableMetadata>>() {});

Conversation

adnanhemani commented Jan 16, 2026

Checklist

Uh oh!

evindj left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dimas-b left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

adutra Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

adnanhemani Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

dimas-b Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adnanhemani Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimas-b Jan 24, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

adutra left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

adutra Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

adutra Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

adnanhemani Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

dimas-b left a comment

Choose a reason for hiding this comment

Uh oh!

adnanhemani commented Jan 29, 2026

Uh oh!

HonahX left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

dimas-b Jan 22, 2026 •

edited

Loading

adnanhemani Jan 23, 2026 •

edited

Loading