[Policy Store] Add policyTypeCode to Slice/Index for Future Filtering Support and Update Policy Persistence Method#1628
Conversation
There was a problem hiding this comment.
I'm not sure it's worth bothering with EclipseLink support for new features since EclipseLink is deprecated.
There was a problem hiding this comment.
I'm OK with either way. Adding it to EclipseLink is nice to have.
There was a problem hiding this comment.
Since policy support is already available in EclipseLink and this PR introduces only a minor change ahead of the 1.0 release, I think it's reasonable to include it. My understanding is that we don’t plan to add any major new features to EclipseLink moving forward.
There was a problem hiding this comment.
Not a blocker: I feel like these two classes are interchangeable in Polaris. Can we merge them to avoid any type mismatching like this? cc @dennishuo
|
Please rebase to get the latest CI changes |
4395fbb to
dc12b13
Compare
| default void deleteAllEntityPolicyMappingRecordsInCurrentTxn( | ||
| @Nonnull PolarisCallContext callCtx, | ||
| @Nonnull PolarisEntityCore entity, | ||
| @Nonnull PolarisBaseEntity entity, |
There was a problem hiding this comment.
side note: we should probably review the PolarisEntityCore type hierarchy. PolarisBaseEntity dominates all child types, so it can probably be folded into PolarisEntityCore (or the other way around).
...c/src/main/java/org/apache/polaris/extension/persistence/relational/jdbc/QueryGenerator.java
Outdated
Show resolved
Hide resolved
* fix(nightly-CI): Do not publish snapshots from forks (apache#1635) Adopt the `Nightly Build` workflow to not (try to) publish every night from forks. * main: Update dependency io.smallrye.config:smallrye-config-core to v3.13.0 (apache#1637) * Use echo to print script errors (apache#1648) * [HOTFIX] QUICKSTART (apache#1646) The change adds the following to fix Quick start experience : [1] ENV variables required by common assets after apache#1522 [2] New configs required to enable FILE based sources apache#1649 Co-authored-by: singhpk234 <[email protected]> Co-authored-by: pjanuario <[email protected]> * main: Update dependency gradle to v8.14.1 (apache#1652) * main: Update dependency gradle to v8.14.1 * Re-adopt PR to the project's needs --------- Co-authored-by: Robert Stupp <[email protected]> * [Policy Store] Add policyTypeCode to Slice/Index for Future Filtering Support and Update Policy Persistence Method (apache#1628) This PR adds policyTypeCode to the in-memory tree map store's slice and the SQL index on policy_mapping_records (already done in JDBC in apache#1468). This prepares for future features that need to filter efficiently by policy type, like listing all entities with a data compaction policy. It also updates the loadAllTargetsOnPolicy method to accept policyTypeCode, enabling it to use the new index for better performance. * fix(test): Do not let some more tests spam `/tmp` (apache#1651) * fix(test): Do not let some more tests not spam `/tmp` * `PolarisRestCatalogViewFileIntegrationTest` * `FileIOExceptionsTest` * `PolarisRestCatalogViewFileIntegrationTest` Changes the tests to leverage JUnit's `@TempDir`. Simplifies `PolarisEclipseLinkMetaStoreManagerTest` * review: rename the (now) abstract class * fix(testing): Do not let PolarisOverlappingTableTest spam `/tmp` (apache#1641) Changes the test to leverage JUnit's `@TempDir`. * Add CATALOG_MANAGE_METADATA to super privilege set of policy attachment privileges (apache#1643) * Fix quickstart doc with docker compose (apache#1610) * main: Update dependency boto3 to v1.38.22 (apache#1657) * Refactor IcebergCatalog to isolate internal state (apache#1659) Following up on apache#1694 * Restore `private` scope on internal fields in `IcebergCatalog` * Use a test-only setter instead of sub-classing to manage injecting test FileIO implementations * Refactor: Use per-request STS credentials (apache#1629) * Refactor: Use per-request STS credentials No functional changes. This is mostly to allow more storage integration flexibility in downstream build. This might also be useful for non-AWS storage. * fix and enforce more errorprone checks (apache#1663) enforces the following checks: https://errorprone.info/bugpattern/ObjectsHashCodePrimitive https://errorprone.info/bugpattern/OptionalMapToOptional https://errorprone.info/bugpattern/StringCharset https://errorprone.info/bugpattern/VariableNameSameAsType * Create a wrapper script to generate python client; regenerate the python client (apache#1347) As noted in apache#755 and elsewhere, the generated types in client/python are currently out of date. This introduces a script to regenerate them and a gradle task to run that script. I've also run the script, which necessitated several things to get tests passing: 1. There were small nonfunctional spec changes needed in order to keep the Python client working 2. The CLI and its tests required a few fixes to work with the updated Python client 3. Many of the regtests required fixes to work with the updated Python client * [Python Client] CI for Python client (Continue PR#1096) (apache#1639) Adds CI for python client. It does not include caching poetry step for now since we do not have poetry.lock (it is in .gitignore), see relevant discussion in: apache#1102 (comment), apache#1096 (comment), we can add that later * main: Update actions/setup-python action to v5 (apache#1671) * main: Update actions/checkout action to v4 (apache#1670) * main: Update python Docker tag to v3.13 (apache#1669) * main: Update dependency pytest to ~=7.4.4 (apache#1668) * main: Update dependency software.amazon.awssdk:bom to v2.31.50 (apache#1677) * main: Update dependency boto3 to v1.38.23 (apache#1667) * feat(build): make archive builds reproducible (apache#1664) See https://docs.gradle.org/current/userguide/working_with_files.html#sec:reproducible_archives * main: Update dependency io.prometheus:prometheus-metrics-exporter-servlet-jakarta to v1.3.8 (apache#1679) * NoSQL: adapt to change on oss/main * INFO: Last merged commit: 6ef8b3e --------- Co-authored-by: Mend Renovate <[email protected]> Co-authored-by: ModEtchFill <[email protected]> Co-authored-by: Prashant Singh <[email protected]> Co-authored-by: singhpk234 <[email protected]> Co-authored-by: pjanuario <[email protected]> Co-authored-by: Honah (Jonas) J. <[email protected]> Co-authored-by: MonkeyCanCode <[email protected]> Co-authored-by: Dmitri Bourlatchkov <[email protected]> Co-authored-by: Dmitri Bourlatchkov <[email protected]> Co-authored-by: Christopher Lambert <[email protected]> Co-authored-by: Eric Maynard <[email protected]>
This PR adds
policyTypeCodeto both the in-memory tree map store's slice and the SQL tablepolicy_mapping_recordsindex (JDBC has already included this in #1468 ). This change lays the groundwork for future features that require efficient filtering by policy type—such as fetching all entities with a data compaction policy attached.As part of this update, the signature of
loadAllTargetsOnPolicyis also modified to acceptpolicyTypeCode, allowing the method to take advantage of the new index for improved performance.cc: @flyrain @singhpk234