Skip to content

[Policy Store] Add policyTypeCode to Slice/Index for Future Filtering Support and Update Policy Persistence Method#1628

Merged
HonahX merged 3 commits intoapache:mainfrom
HonahX:honahx_load_all_targets_on_policy_refactor
May 22, 2025
Merged

[Policy Store] Add policyTypeCode to Slice/Index for Future Filtering Support and Update Policy Persistence Method#1628
HonahX merged 3 commits intoapache:mainfrom
HonahX:honahx_load_all_targets_on_policy_refactor

Conversation

@HonahX
Copy link
Contributor

@HonahX HonahX commented May 20, 2025

This PR adds policyTypeCode to both the in-memory tree map store's slice and the SQL table policy_mapping_records index (JDBC has already included this in #1468 ). This change lays the groundwork for future features that require efficient filtering by policy type—such as fetching all entities with a data compaction policy attached.

As part of this update, the signature of loadAllTargetsOnPolicy is also modified to accept policyTypeCode, allowing the method to take advantage of the new index for improved performance.

cc: @flyrain @singhpk234

@github-project-automation github-project-automation bot moved this to PRs In Progress in Basic Kanban Board May 20, 2025
@HonahX HonahX changed the title Add policyTypeCode to slice/index for optimization and future feature [Draft] Add policyTypeCode to slice/index for optimization and future feature May 20, 2025
@HonahX HonahX changed the title [Draft] Add policyTypeCode to slice/index for optimization and future feature [Policy Store] Add policyTypeCode to Slice/Index for Future Filtering Support and Update Policy Persistence Method May 20, 2025
@HonahX HonahX marked this pull request as ready for review May 20, 2025 20:17
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure it's worth bothering with EclipseLink support for new features since EclipseLink is deprecated.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm OK with either way. Adding it to EclipseLink is nice to have.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since policy support is already available in EclipseLink and this PR introduces only a minor change ahead of the 1.0 release, I think it's reasonable to include it. My understanding is that we don’t plan to add any major new features to EclipseLink moving forward.

Copy link
Contributor

@flyrain flyrain May 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a blocker: I feel like these two classes are interchangeable in Polaris. Can we merge them to avoid any type mismatching like this? cc @dennishuo

flyrain
flyrain previously approved these changes May 21, 2025
Copy link
Contributor

@flyrain flyrain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@github-project-automation github-project-automation bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board May 21, 2025
@dimas-b
Copy link
Contributor

dimas-b commented May 22, 2025

Please rebase to get the latest CI changes

@HonahX HonahX force-pushed the honahx_load_all_targets_on_policy_refactor branch from 4395fbb to dc12b13 Compare May 22, 2025 17:05
@HonahX HonahX requested a review from ajantha-bhat as a code owner May 22, 2025 17:05
dimas-b
dimas-b previously approved these changes May 22, 2025
default void deleteAllEntityPolicyMappingRecordsInCurrentTxn(
@Nonnull PolarisCallContext callCtx,
@Nonnull PolarisEntityCore entity,
@Nonnull PolarisBaseEntity entity,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

side note: we should probably review the PolarisEntityCore type hierarchy. PolarisBaseEntity dominates all child types, so it can probably be folded into PolarisEntityCore (or the other way around).

singhpk234
singhpk234 previously approved these changes May 22, 2025
@HonahX HonahX dismissed stale reviews from singhpk234, dimas-b, and flyrain via 437653c May 22, 2025 20:51
@HonahX HonahX merged commit a534193 into apache:main May 22, 2025
6 checks passed
@github-project-automation github-project-automation bot moved this from Ready to merge to Done in Basic Kanban Board May 22, 2025
snazy added a commit to snazy/polaris that referenced this pull request Jun 13, 2025
* fix(nightly-CI): Do not publish snapshots from forks (apache#1635)

Adopt the `Nightly Build` workflow to not (try to) publish every night from forks.

* main: Update dependency io.smallrye.config:smallrye-config-core to v3.13.0 (apache#1637)

* Use echo to print script errors (apache#1648)

* [HOTFIX] QUICKSTART (apache#1646)

The change adds the following to fix Quick start experience : 
[1] ENV variables required by common assets after apache#1522 
[2] New configs required to enable FILE based sources apache#1649

Co-authored-by: singhpk234 <[email protected]>
Co-authored-by: pjanuario <[email protected]>

* main: Update dependency gradle to v8.14.1 (apache#1652)

* main: Update dependency gradle to v8.14.1

* Re-adopt PR to the project's needs

---------

Co-authored-by: Robert Stupp <[email protected]>

* [Policy Store] Add policyTypeCode to Slice/Index for Future Filtering Support and Update Policy Persistence Method (apache#1628)

This PR adds policyTypeCode to the in-memory tree map store's slice and the SQL index on policy_mapping_records (already done in JDBC in apache#1468). This prepares for future features that need to filter efficiently by policy type, like listing all entities with a data compaction policy.

It also updates the loadAllTargetsOnPolicy method to accept policyTypeCode, enabling it to use the new index for better performance.

* fix(test): Do not let some more tests spam `/tmp` (apache#1651)

* fix(test): Do not let some more tests not spam `/tmp`

* `PolarisRestCatalogViewFileIntegrationTest`
* `FileIOExceptionsTest`
* `PolarisRestCatalogViewFileIntegrationTest`

Changes the tests to leverage JUnit's `@TempDir`.

Simplifies `PolarisEclipseLinkMetaStoreManagerTest`

* review: rename the (now) abstract class

* fix(testing): Do not let PolarisOverlappingTableTest spam `/tmp` (apache#1641)

Changes the test to leverage JUnit's `@TempDir`.

* Add CATALOG_MANAGE_METADATA to super privilege set of policy attachment privileges (apache#1643)

* Fix quickstart doc with docker compose (apache#1610)

* main: Update dependency boto3 to v1.38.22 (apache#1657)

* Refactor IcebergCatalog to isolate internal state (apache#1659)

Following up on apache#1694

* Restore `private` scope on internal fields in `IcebergCatalog`

* Use a test-only setter instead of sub-classing to manage injecting
  test FileIO implementations

* Refactor: Use per-request STS credentials (apache#1629)

* Refactor: Use per-request STS credentials

No functional changes.

This is mostly to allow more storage integration
flexibility in downstream build.

This might also be useful for non-AWS storage.

* fix and enforce more errorprone checks (apache#1663)

enforces the following checks:
https://errorprone.info/bugpattern/ObjectsHashCodePrimitive
https://errorprone.info/bugpattern/OptionalMapToOptional
https://errorprone.info/bugpattern/StringCharset
https://errorprone.info/bugpattern/VariableNameSameAsType

* Create a wrapper script to generate python client; regenerate the python client (apache#1347)

As noted in apache#755 and elsewhere, the generated types in client/python are currently out of date. This introduces a script to regenerate them and a gradle task to run that script.

I've also run the script, which necessitated several things to get tests passing:

1. There were small nonfunctional spec changes needed in order to keep the Python client working
2. The CLI and its tests required a few fixes to work with the updated Python client
3. Many of the regtests required fixes to work with the updated Python client

* [Python Client] CI for Python client (Continue PR#1096) (apache#1639)

Adds CI for python client. It does not include caching poetry step for now since we do not have poetry.lock (it is in .gitignore), see relevant discussion in: apache#1102 (comment), apache#1096 (comment), we can add that later

* main: Update actions/setup-python action to v5 (apache#1671)

* main: Update actions/checkout action to v4 (apache#1670)

* main: Update python Docker tag to v3.13 (apache#1669)

* main: Update dependency pytest to ~=7.4.4 (apache#1668)

* main: Update dependency software.amazon.awssdk:bom to v2.31.50 (apache#1677)

* main: Update dependency boto3 to v1.38.23 (apache#1667)

* feat(build): make archive builds reproducible (apache#1664)

See https://docs.gradle.org/current/userguide/working_with_files.html#sec:reproducible_archives

* main: Update dependency io.prometheus:prometheus-metrics-exporter-servlet-jakarta to v1.3.8 (apache#1679)

* NoSQL: adapt to change on oss/main

* INFO: Last merged commit: 6ef8b3e

---------

Co-authored-by: Mend Renovate <[email protected]>
Co-authored-by: ModEtchFill <[email protected]>
Co-authored-by: Prashant Singh <[email protected]>
Co-authored-by: singhpk234 <[email protected]>
Co-authored-by: pjanuario <[email protected]>
Co-authored-by: Honah (Jonas) J. <[email protected]>
Co-authored-by: MonkeyCanCode <[email protected]>
Co-authored-by: Dmitri Bourlatchkov <[email protected]>
Co-authored-by: Dmitri Bourlatchkov <[email protected]>
Co-authored-by: Christopher Lambert <[email protected]>
Co-authored-by: Eric Maynard <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants