Skip to content

Added support for migrating Legacy Table ACLs from workspace-local to account-level groups#412

Merged
nfx merged 12 commits intomainfrom
feat/hms-grants-for-local-groups
Oct 8, 2023
Merged

Added support for migrating Legacy Table ACLs from workspace-local to account-level groups#412
nfx merged 12 commits intomainfrom
feat/hms-grants-for-local-groups

Conversation

@nfx
Copy link
Copy Markdown
Collaborator

@nfx nfx commented Oct 7, 2023

When workspace-local group is replaced with account-level group, Legacy Table ACLs disappear from Hive Metastore objects. This PR plugs HMS grants with crawler/applier framework on top of $inventory.permissions

… account-level groups

TODO:

- [ ] unit test coverage
- [ ] task documentation
@nfx nfx added enhancement New feature or request migrate/groups Corresponds to Migrate Groups Step of go/uc/upgrade labels Oct 7, 2023
@nfx nfx requested review from a team October 7, 2023 22:25
@codecov
Copy link
Copy Markdown

codecov bot commented Oct 7, 2023

Codecov Report

Merging #412 (146da4e) into main (d3be572) will increase coverage by 0.24%.
The diff coverage is 84.28%.

@@            Coverage Diff             @@
##             main     #412      +/-   ##
==========================================
+ Coverage   83.23%   83.48%   +0.24%     
==========================================
  Files          30       31       +1     
  Lines        2511     2555      +44     
  Branches      442      448       +6     
==========================================
+ Hits         2090     2133      +43     
- Misses        318      319       +1     
  Partials      103      103              
Files Coverage Δ
src/databricks/labs/ucx/runtime.py 52.56% <100.00%> (ø)
src/databricks/labs/ucx/workspace_access/base.py 100.00% <ø> (ø)
src/databricks/labs/ucx/workspace_access/groups.py 90.22% <100.00%> (+0.34%) ⬆️
...rc/databricks/labs/ucx/workspace_access/manager.py 90.00% <85.71%> (+0.58%) ⬆️
.../databricks/labs/ucx/workspace_access/migration.py 50.00% <0.00%> (+2.23%) ⬆️
src/databricks/labs/ucx/workspace_access/tacl.py 96.87% <96.87%> (ø)
src/databricks/labs/ucx/hive_metastore/grants.py 91.30% <63.63%> (-6.76%) ⬇️

... and 1 file with indirect coverage changes

@nfx nfx added the pr/do-not-merge this pull request is not ready to merge label Oct 7, 2023
@nfx nfx removed the pr/do-not-merge this pull request is not ready to merge label Oct 8, 2023
@nfx nfx merged commit d7cc195 into main Oct 8, 2023
@nfx nfx deleted the feat/hms-grants-for-local-groups branch October 8, 2023 14:32
@nfx nfx mentioned this pull request Oct 12, 2023
nfx added a commit that referenced this pull request Oct 12, 2023
# Version changelog

## 0.4.0

* Added exception handling for secret scope not found.
([#418](#418)).
* Added a crawler for creating an inventory of Azure Service Principals
([#326](#326)).
* Added check if account group already exists during failure recovery
([#446](#446)).
* Added checking for index out of range.
([#429](#429)).
* Added hyperlink to UCX releases in the main readme
([#408](#408)).
* Added integration test to check backup groups get deleted
([#387](#387)).
* Added logging of errors during threadpool operations.
([#376](#376)).
* Added recovery mode for workspace-local groups from temporary groups
([#435](#435)).
* Added support for migrating Legacy Table ACLs from workspace-local to
account-level groups
([#412](#412)).
* Added detection for installations of unreleased versions
([#399](#399)).
* Decoupled `PermissionsManager` from `GroupMigrationToolkit`
([#407](#407)).
* Enabled debug logging for every job task run through a file, which is
accessible from both workspace UI and Databricks CLI
([#426](#426)).
* Ensured that table exists, even when crawlers produce zero records
([#373](#373)).
* Extended test suite for HMS->HMS TACL migration
([#439](#439)).
* Fixed handling of secret scope responses
([#431](#431)).
* Fixed `crawl_permissions` task to respect 'workspace_start_path'
config ([#444](#444)).
* Fixed broken logic in `parallel` module and applied hardened error
handling design for parallel code
([#405](#405)).
* Fixed codecov.io reporting
([#403](#403)).
* Fixed integration tests for crawlers
([#379](#379)).
* Improved README.py and logging messages
([#433](#433)).
* Improved cleanup for workspace backup groups by adding more retries on
errors ([#375](#375)).
* Improved dashboard queries to show unsupported storage types.
([#398](#398)).
* Improved documentation for readme notebook
([#257](#257)).
* Improved test coverage for installer
([#371](#371)).
* Introduced deterministic `env_or_skip` fixture for integration tests
([#396](#396)).
* Made HMS & UC fixtures return `CatalogInfo`, `SchemaInfo`, and
`TableInfo` ([#409](#409)).
* Merge `workspace_access.Crawler` and `workspace_access.Applier`
interfaces to `workspace_access.AclSupport`
([#436](#436)).
* Moved examples to docs
([#404](#404)).
* Properly isolated integration testing for workflows on an existing
shared cluster ([#414](#414)).
* Removed thread pool for any IAM Group removals and additions
([#394](#394)).
* Replace plus char with minus in version tag for GCP dev installation
of UCX ([#420](#420)).
* Run integration tests on shared clusters for a faster devloop
([#397](#397)).
* Show difference between serverless and PRO warehouses during
installation ([#385](#385)).
* Split `migrate-groups` workflow into three different stages for
reliability ([#442](#442)).
* Use groups instead of usernames in code owners file
([#389](#389)).
FastLee pushed a commit that referenced this pull request Oct 25, 2023
… account-level groups (#412)

When workspace-local group is replaced with account-level group, Legacy
Table ACLs disappear from Hive Metastore objects. This PR plugs HMS
grants with `crawler`/`applier` framework on top of
`$inventory.permissions`
FastLee pushed a commit that referenced this pull request Oct 25, 2023
# Version changelog

## 0.4.0

* Added exception handling for secret scope not found.
([#418](#418)).
* Added a crawler for creating an inventory of Azure Service Principals
([#326](#326)).
* Added check if account group already exists during failure recovery
([#446](#446)).
* Added checking for index out of range.
([#429](#429)).
* Added hyperlink to UCX releases in the main readme
([#408](#408)).
* Added integration test to check backup groups get deleted
([#387](#387)).
* Added logging of errors during threadpool operations.
([#376](#376)).
* Added recovery mode for workspace-local groups from temporary groups
([#435](#435)).
* Added support for migrating Legacy Table ACLs from workspace-local to
account-level groups
([#412](#412)).
* Added detection for installations of unreleased versions
([#399](#399)).
* Decoupled `PermissionsManager` from `GroupMigrationToolkit`
([#407](#407)).
* Enabled debug logging for every job task run through a file, which is
accessible from both workspace UI and Databricks CLI
([#426](#426)).
* Ensured that table exists, even when crawlers produce zero records
([#373](#373)).
* Extended test suite for HMS->HMS TACL migration
([#439](#439)).
* Fixed handling of secret scope responses
([#431](#431)).
* Fixed `crawl_permissions` task to respect 'workspace_start_path'
config ([#444](#444)).
* Fixed broken logic in `parallel` module and applied hardened error
handling design for parallel code
([#405](#405)).
* Fixed codecov.io reporting
([#403](#403)).
* Fixed integration tests for crawlers
([#379](#379)).
* Improved README.py and logging messages
([#433](#433)).
* Improved cleanup for workspace backup groups by adding more retries on
errors ([#375](#375)).
* Improved dashboard queries to show unsupported storage types.
([#398](#398)).
* Improved documentation for readme notebook
([#257](#257)).
* Improved test coverage for installer
([#371](#371)).
* Introduced deterministic `env_or_skip` fixture for integration tests
([#396](#396)).
* Made HMS & UC fixtures return `CatalogInfo`, `SchemaInfo`, and
`TableInfo` ([#409](#409)).
* Merge `workspace_access.Crawler` and `workspace_access.Applier`
interfaces to `workspace_access.AclSupport`
([#436](#436)).
* Moved examples to docs
([#404](#404)).
* Properly isolated integration testing for workflows on an existing
shared cluster ([#414](#414)).
* Removed thread pool for any IAM Group removals and additions
([#394](#394)).
* Replace plus char with minus in version tag for GCP dev installation
of UCX ([#420](#420)).
* Run integration tests on shared clusters for a faster devloop
([#397](#397)).
* Show difference between serverless and PRO warehouses during
installation ([#385](#385)).
* Split `migrate-groups` workflow into three different stages for
reliability ([#442](#442)).
* Use groups instead of usernames in code owners file
([#389](#389)).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request migrate/groups Corresponds to Migrate Groups Step of go/uc/upgrade

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant