Skip to content

Added support for crawling grants and applying Hive Metastore UDF ACLs#812

Merged
nfx merged 13 commits intomainfrom
feature/functions_crawl_apply
Jan 18, 2024
Merged

Added support for crawling grants and applying Hive Metastore UDF ACLs#812
nfx merged 13 commits intomainfrom
feature/functions_crawl_apply

Conversation

@mwojtyczka
Copy link
Copy Markdown
Contributor

@mwojtyczka mwojtyczka commented Jan 18, 2024 •
edited
Loading

Changes

Added support for crawling grants and applying Hive Metastore UDF ACLs.

Resolves #808

Functionality

  • added new crawler for udfs
  • added handling of udfs to the existing TableAclSupport
  • added new column to inventory permissions table for udfs

Tests

  • tested manually
  • added unit tests
  • added integration tests

@mwojtyczka mwojtyczka requested review from a team, nfx and william-conti January 18, 2024 14:27
@codecov
Copy link
Copy Markdown

codecov bot commented Jan 18, 2024 •
edited
Loading

Codecov Report

Attention: 7 lines in your changes are missing coverage. Please review.

Comparison is base (af80620) 84.07% compared to head (1969124) 84.20%.

Files Patch % Lines
src/databricks/labs/ucx/hive_metastore/udfs.py 91.66% 2 Missing and 3 partials ⚠️
src/databricks/labs/ucx/runtime.py 33.33% 2 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main     #812      +/-   ##
==========================================
+ Coverage   84.07%   84.20%   +0.13%     
==========================================
  Files          39       40       +1     
  Lines        4872     4952      +80     
  Branches      913      924      +11     
==========================================
+ Hits         4096     4170      +74     
- Misses        564      567       +3     
- Partials      212      215       +3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

nfx
nfx approved these changes Jan 18, 2024
View reviewed changes
Copy link
Copy Markdown
Collaborator

@nfx nfx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall lgtm, couple of nits though

@mwojtyczka @nfx
Update src/databricks/labs/ucx/hive_metastore/udfs.py
5a31799 
review feedback implementation

Co-authored-by: Serge Smertin <[email protected]>
@mwojtyczka mwojtyczka changed the title [FEATURE]: Crawl and Apply grants for Hive Metastore UDFs Crawl Hive Metastore UDFs and apply Table Acl grants for groups migration Jan 18, 2024
@mwojtyczka mwojtyczka changed the title Crawl Hive Metastore UDFs and apply Table Acl grants for groups migration Crawl Hive Metastore UDFs and apply Table Acl grants Jan 18, 2024
@mwojtyczka mwojtyczka changed the title Crawl Hive Metastore UDFs and apply Table Acl grants Crawl and apply Table Acl grants for Hive Metastore UDFs Jan 18, 2024
@mwojtyczka mwojtyczka changed the title Crawl and apply Table Acl grants for Hive Metastore UDFs Added support for crawling Hive Metastore Udf grants and applying ACLs Jan 18, 2024
@mwojtyczka mwojtyczka requested a review from nfx January 18, 2024 21:43
@mwojtyczka mwojtyczka changed the title Added support for crawling Hive Metastore Udf grants and applying ACLs Added support for crawling Hive Metastore UDF grants and applying ACLs Jan 18, 2024
@mwojtyczka mwojtyczka changed the title Added support for crawling Hive Metastore UDF grants and applying ACLs Added support for crawling grants and applying Hive Metastore UDF ACLs Jan 18, 2024
nfx
nfx approved these changes Jan 18, 2024
View reviewed changes
Copy link
Copy Markdown
Collaborator

@nfx nfx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@nfx nfx merged commit c9ee832 into main Jan 18, 2024
@nfx nfx deleted the feature/functions_crawl_apply branch January 18, 2024 21:53
nfx added a commit that referenced this pull request Jan 19, 2024
@nfx
Release v0.10.0
10ea0b1 
* Added `databricks labs ucx validate-groups-membership` command to validate groups to see if they have same membership across acount and workspace level ([#772](#772)).
* Added baseline for getting Azure Resource Role Assignments ([#764](#764)).
* Added issue and pull request templates ([#791](#791)).
* Added linked issues to PR template ([#793](#793)).
* Added optional `debug_truncate_bytes` parameter to the config and extend the default log truncation limit ([#782](#782)).
* Added support for crawling grants and applying Hive Metastore UDF ACLs ([#812](#812)).
* Changed Python requirement from 3.10.6 to 3.10 ([#805](#805)).
* Extend error handling of delta issues in crawlers and hive metastore ([#795](#795)).
* Fixed `databricks labs ucx repair-run` command to execute correctly ([#801](#801)).
* Fixed handling of `DELTASHARING` table format ([#802](#802)).
* Fixed listing of workflows via CLI ([#811](#811)).
* Fixed logger import path for DEBUG notebook ([#792](#792)).
* Fixed move table command to delete table/view regardless if permissions are present, skipping corrupted tables when crawling table size and making existing tests more stable ([#777](#777)).
* Fixed the issue of `databricks labs ucx installations` and `databricks labs ucx manual-workspace-info` ([#814](#814)).
* Increase the unit test coverage for cli.py ([#800](#800)).
* Mount Point crawler lists /Volume with four variations which is confusing ([#779](#779)).
* Updated README.md to remove mention of deprecated install.sh ([#781](#781)).
* Updated `bug` issue template ([#797](#797)).
* Fixed writing log readme in multiprocess safe way ([#794](#794)).
@nfx nfx mentioned this pull request Jan 19, 2024
Merged
nfx added a commit that referenced this pull request Jan 19, 2024
@nfx
Release v0.10.0 (#819)
38256dc 
* Added `databricks labs ucx validate-groups-membership` command to
validate groups to see if they have same membership across acount and
workspace level
([#772](#772)).
* Added baseline for getting Azure Resource Role Assignments
([#764](#764)).
* Added issue and pull request templates
([#791](#791)).
* Added linked issues to PR template
([#793](#793)).
* Added optional `debug_truncate_bytes` parameter to the config and
extend the default log truncation limit
([#782](#782)).
* Added support for crawling grants and applying Hive Metastore UDF ACLs
([#812](#812)).
* Changed Python requirement from 3.10.6 to 3.10
([#805](#805)).
* Extend error handling of delta issues in crawlers and hive metastore
([#795](#795)).
* Fixed `databricks labs ucx repair-run` command to execute correctly
([#801](#801)).
* Fixed handling of `DELTASHARING` table format
([#802](#802)).
* Fixed listing of workflows via CLI
([#811](#811)).
* Fixed logger import path for DEBUG notebook
([#792](#792)).
* Fixed move table command to delete table/view regardless if
permissions are present, skipping corrupted tables when crawling table
size and making existing tests more stable
([#777](#777)).
* Fixed the issue of `databricks labs ucx installations` and `databricks
labs ucx manual-workspace-info`
([#814](#814)).
* Increase the unit test coverage for cli.py
([#800](#800)).
* Mount Point crawler lists /Volume with four variations which is
confusing ([#779](#779)).
* Updated README.md to remove mention of deprecated install.sh
([#781](#781)).
* Updated `bug` issue template
([#797](#797)).
* Fixed writing log readme in multiprocess safe way
([#794](#794)).
dmoore247 pushed a commit that referenced this pull request Mar 23, 2024
@nfx @dmoore247
Release v0.10.0 (#819)
d30eb4d 
* Added `databricks labs ucx validate-groups-membership` command to
validate groups to see if they have same membership across acount and
workspace level
([#772](#772)).
* Added baseline for getting Azure Resource Role Assignments
([#764](#764)).
* Added issue and pull request templates
([#791](#791)).
* Added linked issues to PR template
([#793](#793)).
* Added optional `debug_truncate_bytes` parameter to the config and
extend the default log truncation limit
([#782](#782)).
* Added support for crawling grants and applying Hive Metastore UDF ACLs
([#812](#812)).
* Changed Python requirement from 3.10.6 to 3.10
([#805](#805)).
* Extend error handling of delta issues in crawlers and hive metastore
([#795](#795)).
* Fixed `databricks labs ucx repair-run` command to execute correctly
([#801](#801)).
* Fixed handling of `DELTASHARING` table format
([#802](#802)).
* Fixed listing of workflows via CLI
([#811](#811)).
* Fixed logger import path for DEBUG notebook
([#792](#792)).
* Fixed move table command to delete table/view regardless if
permissions are present, skipping corrupted tables when crawling table
size and making existing tests more stable
([#777](#777)).
* Fixed the issue of `databricks labs ucx installations` and `databricks
labs ucx manual-workspace-info`
([#814](#814)).
* Increase the unit test coverage for cli.py
([#800](#800)).
* Mount Point crawler lists /Volume with four variations which is
confusing ([#779](#779)).
* Updated README.md to remove mention of deprecated install.sh
([#781](#781)).
* Updated `bug` issue template
([#797](#797)).
* Fixed writing log readme in multiprocess safe way
([#794](#794)).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nfx nfx nfx approved these changes

@william-conti william-conti Awaiting requested review from william-conti

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[FEATURE]: Crawl and Apply grants for SQL UDFs

2 participants

@mwojtyczka @nfx