Skip to content

[FEATURE] Nessie integration of Iceberg REST - #7043

Merged
snazy merged 28 commits into
mainfrom
feature/nessie-catalog-server
May 28, 2024
Merged

snazy merged 28 commits into
mainfrom
feature/nessie-catalog-server

Conversation

@snazy

@snazy snazy commented Jun 14, 2023 •

Copy link
Copy Markdown
Member

No description provided.

@snazy snazy added this to the 1.0.0 milestone Jun 14, 2023
@snazy snazy changed the title [Nessie Catalog Server] Feature branch [FEATURE] Nessie Catalog Server] Jun 14, 2023
@snazy snazy changed the title [FEATURE] Nessie Catalog Server] [FEATURE] Nessie Catalog Server Jun 14, 2023
@snazy
snazy force-pushed the feature/nessie-catalog-server branch 3 times, most recently from ae85c09 to 5783a01 Compare July 11, 2023 15:44
@snazy
snazy force-pushed the feature/nessie-catalog-server branch 4 times, most recently from 422dd34 to 9b4e0fb Compare July 19, 2023 17:13
@snazy
snazy force-pushed the feature/nessie-catalog-server branch 7 times, most recently from 694c558 to 899ce14 Compare August 3, 2023 08:04
@snazy snazy changed the title [FEATURE] Nessie Catalog Server [FEATURE] TBD May 2, 2024
@snazy snazy closed this May 2, 2024
@snazy
snazy force-pushed the feature/nessie-catalog-server branch from fd576c7 to d696495 Compare May 2, 2024 15:25
@snazy snazy reopened this May 2, 2024
@snazy snazy changed the title [FEATURE] TBD [FEATURE] Nessie integration of Iceberg REST May 13, 2024
@snazy
snazy force-pushed the feature/nessie-catalog-server branch from c34104b to 47de890 Compare May 13, 2024 15:06
@snazy
snazy requested review from adutra and dimas-b May 13, 2024 15:06
@snazy
snazy marked this pull request as ready for review May 13, 2024 15:06
@snazy
snazy force-pushed the feature/nessie-catalog-server branch 2 times, most recently from b8c6df1 to b1b16ff Compare May 17, 2024 07:57
@snazy snazy added pr-integrations NesQuEIT (Iceberg, Spark, Flink, Presto) pr-macos-win Test PR on macOS + Windows labels May 17, 2024
adutra and others added 17 commits May 27, 2024 20:55
Really use a non-default warehouse, configure the warehouse _name_ via the misleadingly named property `CatalogProperties.WAREHOUSE_LOCATION`.

Also eliminate some duplicate code.
and add some unit tests

also remove unused `CatalogServerConfig`
…8539)

The Nessie client used from Nessie Catalog is currently a "simple" one. This change lets it use a client that's configured against the actual config+tree resources, which also perform authZ.
Start collecting all base locations in `NessieTableSnapshot`. This change is a start to collect _all_ (base) table locations, to handle cases when a table's (meta)data files are "spread" across multiple base locations. Ideally (and probably usually), each table has only one base location.
* Make `nessie.catalog.default-warehouse` just the optional name of the default warehouse (prevent duplicate configuration).
* Remove `WarehouseConfig.name()` - the name is implicitly defined via the map key of `CatalogConfig.warehouses()`
The handling of secrets is a bit awkward at the moment. We declare "IDs" on the objects, which are to be looked up via the `SecretsProvider`, which is currently just a `Map<String, String>`.

This change changes the behavior: The (default) behavior to lookup via another map is gone.

Secrets are now generally looked up in the following order:
1. If the value is defined on the "per bucket" config, it is used as is.
2. If a secret for the bucket can be resolved, it is used.
3. If the value is defined on the "default" config, it is used as is.
4. If a secret for the "default" config can be resolved, it is used.
5. The secret is assumed to not exist.

The above logic is implemented in the new `SecretsHelper` class. It calls the `SecretProvider` using a collection of keys to look up, which shall return a map with the existing, resolved secrets.

Follow-up, after #7043 is merged to `main`: refactor secrets handlings in "Nessie Core", where/if necessary.
- Client `client1` has access to everything.
- Client `client2` has *NO* access to the `sales` namespace.
- Client `client3` has *NO* access to the `eng` namespace.

----------
Co-authored-by: Alexandre Dutra <[email protected]>
`CONNECT TO` and `USE` working, `SHOW TABLE/VIEW` displays the table/view metadata.

Also adds the `ON initialReference` clause to `CONNECT TO`.
@snazy
snazy force-pushed the feature/nessie-catalog-server branch from 5e176f9 to 49cf7f7 Compare May 27, 2024 18:55
@snazy

snazy commented May 27, 2024

Copy link
Copy Markdown
Member Author

@dimas-b @adutra Let's give this "small piece" a final cross-check before we finally merge it.

@dimas-b

dimas-b commented May 28, 2024

Copy link
Copy Markdown
Member

I ran a quick smoke test in my local on bd33b94 - it seems to work fine 🎉

@adutra adutra left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving although I think we have at least one issue with base location that we just discussed offline.

@snazy
snazy merged commit e4778c2 into main May 28, 2024
@snazy
snazy deleted the feature/nessie-catalog-server branch May 28, 2024 16:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-docker Smoke test Docker images pr-helm Helm chart testing pr-integrations NesQuEIT (Iceberg, Spark, Flink, Presto) pr-macos-win Test PR on macOS + Windows pr-newer-java

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants