Added CLI Command `databricks labs ucx save-uc-compatible-roles` by FastLee · Pull Request #863 · databrickslabs/ucx

FastLee · 2024-01-31T02:18:51Z

Changes

CLI command to scan roles that are set with trust relationships with the UC master roles and the S3 buckets they have access to.
Genererates a CSV file.

The CSV File has the following format:

iam_role_arn,resource_type,privilege,resource_path
arn:aws:iam::12345:rolerole1,s3,READ_FILES,s3://bucket1
arn:aws:iam::12345:role/role1,s3,READ_FILES,s3a://bucket1
arn:aws:iam::12345:role/role1,s3,READ_FILES,s3://bucket2
arn:aws:iam::12345:role/role1,s3,READ_FILES,s3a://bucket2

The command relies on AWS CLI Command and require the user to setup and configure it.
Requires a working setup of AWS CLI.
AWS CLI
The command saves a CSV to the UCX installation folder with the mapping.

The user has to be authenticated with AWS and the have the permissions to browse the resources and iam services.
More information can be found here:
https://docs.aws.amazon.com/IAM/latest/UserGuide/access_permissions-required.html

Linked issues

closes #861

Functionality

added relevant user documentation
added new CLI command
modified existing command: databricks labs ucx ...
added a new workflow
modified existing workflow: ...
added a new table
modified existing table: ...

Tests

manually tested
added unit tests
added integration tests
verified on staging environment (screenshot attached)

codecov · 2024-01-31T02:20:37Z

Codecov Report

Attention: 5 lines in your changes are missing coverage. Please review.

Comparison is base (e11494c) 86.48% compared to head (86d42b5) 86.54%.

Files	Patch %	Lines
src/databricks/labs/ucx/assessment/aws.py	94.82%	2 Missing and 1 partial ⚠️
src/databricks/labs/ucx/cli.py	84.61%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #863      +/-   ##
==========================================
+ Coverage   86.48%   86.54%   +0.06%     
==========================================
  Files          41       41              
  Lines        5016     5077      +61     
  Branches      907      925      +18     
==========================================
+ Hits         4338     4394      +56     
- Misses        469      473       +4     
- Partials      209      210       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

src/databricks/labs/ucx/assessment/aws.py

nfx

Lgtm

william-conti

Some informations should be hidden

src/databricks/labs/ucx/assessment/aws.py

src/databricks/labs/ucx/cli.py

* Added CLI Command `databricks labs ucx save-uc-compatible-roles` ([#863](#863)). * Added dashboard widget with table count by storage and format ([#852](#852)). * Added verification of group permissions ([#841](#841)). * Checking pipeline cluster config and cluster policy in 'crawl_pipelines' task ([#864](#864)). * Created cluster policy (ucx-policy) to be used by all UCX compute. This may require customers to reinstall UCX. ([#853](#853)). * Skip scanning objects that were removed on platform side since the last scan time, so that integration tests are less flaky ([#922](#922)). * Updated assessment documentation ([#873](#873)). Dependency updates: * Updated databricks-sdk requirement from ~=0.18.0 to ~=0.19.0 ([#930](#930)).

FastLee had a problem deploying to account-admin January 31, 2024 02:18 — with GitHub Actions Failure

FastLee temporarily deployed to account-admin January 31, 2024 14:31 — with GitHub Actions Inactive

FastLee had a problem deploying to account-admin January 31, 2024 15:49 — with GitHub Actions Failure

FastLee had a problem deploying to account-admin January 31, 2024 17:04 — with GitHub Actions Failure

FastLee had a problem deploying to account-admin January 31, 2024 17:08 — with GitHub Actions Failure

FastLee temporarily deployed to account-admin January 31, 2024 17:33 — with GitHub Actions Inactive

FastLee changed the title ~~New CLI Command databricks labs ucx uc-compatible-roles~~ New CLI Command databricks labs ucx save-uc-compatible-roles Jan 31, 2024

nfx requested changes Jan 31, 2024

View reviewed changes

src/databricks/labs/ucx/assessment/aws.py Outdated Show resolved Hide resolved

src/databricks/labs/ucx/assessment/aws.py Outdated Show resolved Hide resolved

FastLee had a problem deploying to account-admin January 31, 2024 18:25 — with GitHub Actions Failure

FastLee had a problem deploying to account-admin January 31, 2024 19:24 — with GitHub Actions Failure

FastLee had a problem deploying to account-admin January 31, 2024 19:28 — with GitHub Actions Failure

FastLee had a problem deploying to account-admin January 31, 2024 19:38 — with GitHub Actions Failure

FastLee changed the title ~~New CLI Command databricks labs ucx save-uc-compatible-roles~~ Added CLI Command databricks labs ucx save-uc-compatible-roles Jan 31, 2024

FastLee had a problem deploying to account-admin January 31, 2024 19:47 — with GitHub Actions Failure

FastLee requested a review from nfx January 31, 2024 19:49

FastLee had a problem deploying to account-admin January 31, 2024 20:07 — with GitHub Actions Failure

FastLee had a problem deploying to account-admin February 1, 2024 02:24 — with GitHub Actions Failure

FastLee force-pushed the feature/uc-compatible-roles-861 branch from d918053 to 6743c1c Compare February 1, 2024 02:24

FastLee had a problem deploying to account-admin February 1, 2024 02:25 — with GitHub Actions Failure

FastLee had a problem deploying to account-admin February 1, 2024 02:34 — with GitHub Actions Failure

FastLee marked this pull request as ready for review February 1, 2024 17:52

FastLee requested review from a team and nkvuong February 1, 2024 17:52

FastLee force-pushed the feature/uc-compatible-roles-861 branch from eafaf6e to 5d86948 Compare February 1, 2024 21:08

FastLee had a problem deploying to account-admin February 1, 2024 21:08 — with GitHub Actions Failure

FastLee temporarily deployed to account-admin February 1, 2024 22:23 — with GitHub Actions Inactive

FastLee force-pushed the feature/uc-compatible-roles-861 branch from ec22358 to 2790b22 Compare February 1, 2024 22:47

FastLee had a problem deploying to account-admin February 1, 2024 22:47 — with GitHub Actions Failure

Initial Commit

c7593ee

FastLee had a problem deploying to account-admin February 1, 2024 23:22 — with GitHub Actions Failure

FastLee enabled auto-merge February 1, 2024 23:25

nfx approved these changes Feb 1, 2024

View reviewed changes

william-conti suggested changes Feb 2, 2024

View reviewed changes

src/databricks/labs/ucx/assessment/aws.py Show resolved Hide resolved

src/databricks/labs/ucx/assessment/aws.py Show resolved Hide resolved

src/databricks/labs/ucx/cli.py Show resolved Hide resolved

FastLee requested a review from william-conti February 2, 2024 11:24

william-conti approved these changes Feb 2, 2024

View reviewed changes

FastLee had a problem deploying to account-admin February 2, 2024 14:02 — with GitHub Actions Failure

FastLee had a problem deploying to account-admin February 2, 2024 14:36 — with GitHub Actions Failure

FastLee had a problem deploying to account-admin February 2, 2024 15:06 — with GitHub Actions Failure

FastLee had a problem deploying to account-admin February 2, 2024 16:03 — with GitHub Actions Failure

FastLee had a problem deploying to account-admin February 2, 2024 17:35 — with GitHub Actions Failure

FastLee had a problem deploying to account-admin February 2, 2024 17:51 — with GitHub Actions Failure

FastLee had a problem deploying to account-admin February 3, 2024 02:24 — with GitHub Actions Failure

FastLee had a problem deploying to account-admin February 3, 2024 11:04 — with GitHub Actions Failure

FastLee had a problem deploying to account-admin February 3, 2024 11:36 — with GitHub Actions Failure

FastLee had a problem deploying to account-admin February 3, 2024 12:23 — with GitHub Actions Failure

FastLee had a problem deploying to account-admin February 3, 2024 17:06 — with GitHub Actions Failure

FastLee had a problem deploying to account-admin February 4, 2024 15:29 — with GitHub Actions Failure

FastLee had a problem deploying to account-admin February 4, 2024 17:10 — with GitHub Actions Failure

FastLee temporarily deployed to account-admin February 4, 2024 22:11 — with GitHub Actions Inactive

FastLee added this pull request to the merge queue Feb 4, 2024

nfx disabled auto-merge February 4, 2024 22:59

nfx removed this pull request from the merge queue due to a manual request Feb 4, 2024

nfx merged commit c99708c into main Feb 4, 2024

nfx deleted the feature/uc-compatible-roles-861 branch February 4, 2024 23:00

nfx mentioned this pull request Feb 9, 2024

Release v0.12.0 #931

Merged

dmoore247 pushed a commit that referenced this pull request Mar 23, 2024

Added CLI Command databricks labs ucx save-uc-compatible-roles (#863)

25c3886

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added CLI Command `databricks labs ucx save-uc-compatible-roles`#863

Added CLI Command `databricks labs ucx save-uc-compatible-roles`#863
nfx merged 16 commits intomainfrom
feature/uc-compatible-roles-861

FastLee commented Jan 31, 2024 •

edited

Loading

Uh oh!

codecov bot commented Jan 31, 2024 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

nfx left a comment

Uh oh!

william-conti left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

FastLee commented Jan 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Linked issues

Functionality

Tests

Uh oh!

codecov bot commented Jan 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

nfx left a comment

Choose a reason for hiding this comment

Uh oh!

william-conti left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

FastLee commented Jan 31, 2024 •

edited

Loading

codecov bot commented Jan 31, 2024 •

edited

Loading