Added `databricks labs ucx save-aws-iam-profiles` command to scan instance profiles identify S3 access and save a CSV with permissions by FastLee · Pull Request #817 · databrickslabs/ucx

FastLee · 2024-01-19T16:28:47Z

Changes

CLI command to scan service principals and link to all the S3 buckets they have access to.
Genererates a CSV file.

The CSV File has the following format:

instance_profile_arn,resource_type,privilege,resource_path,iam_role_arn
arn:aws:iam::12345:instance-profile/role1,s3,READ_FILES,s3://bucket1,arn:aws:iam::12345:role/role1
arn:aws:iam::12345:instance-profile/role1,s3,READ_FILES,s3a://bucket1,arn:aws:iam::12345:role/role1
arn:aws:iam::12345:instance-profile/role1,s3,READ_FILES,s3://bucket2,arn:aws:iam::12345:role/role1
arn:aws:iam::12345:instance-profile/role1,s3,READ_FILES,s3a://bucket2,arn:aws:iam::12345:role/role1

The command relies on AWS CLI Command and require the user to setup and configure it.
Requires a working setup of AWS CLI.
AWS CLI
The command saves a CSV to the UCX installation folder with the mapping.

The user has to be authenticated with AWS and the have the permissions to browse the resources and iam services.
More information can be found here:
https://docs.aws.amazon.com/IAM/latest/UserGuide/access_permissions-required.html

Linked issues

closes #338

Functionality

added relevant user documentation
added new CLI command
modified existing command: databricks labs ucx ...
added a new workflow
modified existing workflow: ...
added a new table
modified existing table: ...

Tests

manually tested
added unit tests
added integration tests
verified on staging environment (screenshot attached)

codecov · 2024-01-19T17:58:50Z

Codecov Report

Attention: 31 lines in your changes are missing coverage. Please review.

Comparison is base (e36db5f) 85.39% compared to head (5cfdd8a) 85.39%.

Files	Patch %	Lines
src/databricks/labs/ucx/assessment/aws.py	89.83%	9 Missing and 10 partials ⚠️
src/databricks/labs/ucx/cli.py	57.14%	11 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##             main     #817    +/-   ##
========================================
  Coverage   85.39%   85.39%            
========================================
  Files          40       41     +1     
  Lines        5031     5246   +215     
  Branches      921      965    +44     
========================================
+ Hits         4296     4480   +184     
- Misses        523      543    +20     
- Partials      212      223    +11

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

src/databricks/labs/ucx/assessment/aws.py

labs.yml

src/databricks/labs/ucx/assessment/aws.py

src/databricks/labs/ucx/cli.py

tests/integration/assessment/test_aws.py

src/databricks/labs/ucx/cli.py

src/databricks/labs/ucx/assessment/aws.py

src/databricks/labs/ucx/cli.py

tests/unit/assessment/test_aws.py

tests/unit/test_cli.py

All changes were implemented

* Added `databricks labs ucx alias` command to create a view of tables from one schema/catalog in another schema/catalog ([#837](#837)). * Added `databricks labs ucx save-aws-iam-profiles` command to scan instance profiles identify AWS S3 access and save a CSV with permissions ([#817](#817)). * Added total view counts in the assessment dashboard ([#834](#834)). * Cleaned up `assess_jobs` and `assess_clusters` tasks in the `assessment` workflow to improve testing and reduce redundancy.([#825](#825)). * Added documentation for the assessment report ([#806](#806)). * Fixed escaping for SQL object names ([#836](#836)). Dependency updates: * Updated databricks-sdk requirement from ~=0.17.0 to ~=0.18.0 ([#832](#832)).

…tance profiles identify S3 access and save a CSV with permissions (#817) ## Changes CLI command to scan service principals and link to all the S3 buckets they have access to. Genererates a CSV file. The CSV File has the following format: ``` instance_profile_arn,resource_type,privilege,resource_path,iam_role_arn arn:aws:iam::12345:instance-profile/role1,s3,READ_FILES,s3://bucket1,arn:aws:iam::12345:role/role1 arn:aws:iam::12345:instance-profile/role1,s3,READ_FILES,s3a://bucket1,arn:aws:iam::12345:role/role1 arn:aws:iam::12345:instance-profile/role1,s3,READ_FILES,s3://bucket2,arn:aws:iam::12345:role/role1 arn:aws:iam::12345:instance-profile/role1,s3,READ_FILES,s3a://bucket2,arn:aws:iam::12345:role/role1 ``` The command relies on AWS CLI Command and require the user to setup and configure it. Requires a working setup of AWS CLI. [AWS CLI](https://aws.amazon.com/cli/) The command saves a CSV to the UCX installation folder with the mapping. The user has to be authenticated with AWS and the have the permissions to browse the resources and iam services. More information can be found here: https://docs.aws.amazon.com/IAM/latest/UserGuide/access_permissions-required.html ### Linked issues closes #338 ### Functionality - [ ] added relevant user documentation - [x] added new CLI command - [ ] modified existing command: `databricks labs ucx ...` - [ ] added a new workflow - [ ] modified existing workflow: `...` - [ ] added a new table - [ ] modified existing table: `...` ### Tests  - [x] manually tested - [x] added unit tests - [ ] added integration tests - [ ] verified on staging environment (screenshot attached) --------- Co-authored-by: Serge Smertin <[email protected]>

* Added `databricks labs ucx alias` command to create a view of tables from one schema/catalog in another schema/catalog ([#837](#837)). * Added `databricks labs ucx save-aws-iam-profiles` command to scan instance profiles identify AWS S3 access and save a CSV with permissions ([#817](#817)). * Added total view counts in the assessment dashboard ([#834](#834)). * Cleaned up `assess_jobs` and `assess_clusters` tasks in the `assessment` workflow to improve testing and reduce redundancy.([#825](#825)). * Added documentation for the assessment report ([#806](#806)). * Fixed escaping for SQL object names ([#836](#836)). Dependency updates: * Updated databricks-sdk requirement from ~=0.17.0 to ~=0.18.0 ([#832](#832)).

FastLee had a problem deploying to account-admin January 19, 2024 16:28 — with GitHub Actions Failure

FastLee had a problem deploying to account-admin January 19, 2024 17:57 — with GitHub Actions Failure

FastLee had a problem deploying to account-admin January 19, 2024 18:07 — with GitHub Actions Failure

FastLee had a problem deploying to account-admin January 19, 2024 18:09 — with GitHub Actions Failure

nfx requested changes Jan 19, 2024

View reviewed changes

FastLee had a problem deploying to account-admin January 19, 2024 19:43 — with GitHub Actions Failure

FastLee had a problem deploying to account-admin January 19, 2024 20:08 — with GitHub Actions Failure

FastLee had a problem deploying to account-admin January 19, 2024 20:14 — with GitHub Actions Failure

FastLee had a problem deploying to account-admin January 19, 2024 20:41 — with GitHub Actions Failure

FastLee had a problem deploying to account-admin January 19, 2024 20:48 — with GitHub Actions Failure

FastLee requested a review from nfx January 19, 2024 20:50

nfx requested changes Jan 20, 2024

View reviewed changes

FastLee had a problem deploying to account-admin January 20, 2024 06:02 — with GitHub Actions Failure

nfx marked this pull request as ready for review January 20, 2024 11:13

nfx requested review from a team and tnguyen-db January 20, 2024 11:13

FastLee had a problem deploying to account-admin January 20, 2024 16:17 — with GitHub Actions Failure

FastLee had a problem deploying to account-admin January 20, 2024 16:25 — with GitHub Actions Failure

dmoore247 previously requested changes Jan 20, 2024

View reviewed changes

FastLee force-pushed the feature/aws-instance-profiles branch from 73216f5 to cd41add Compare January 20, 2024 16:40

FastLee had a problem deploying to account-admin January 20, 2024 16:40 — with GitHub Actions Failure

FastLee had a problem deploying to account-admin January 20, 2024 21:36 — with GitHub Actions Failure

FastLee had a problem deploying to account-admin January 20, 2024 21:45 — with GitHub Actions Failure

FastLee requested a review from dmoore247 January 20, 2024 21:50

FastLee temporarily deployed to account-admin January 20, 2024 22:29 — with GitHub Actions Inactive

FastLee had a problem deploying to account-admin January 20, 2024 22:39 — with GitHub Actions Failure

FastLee requested a review from nfx January 20, 2024 22:42

nfx requested changes Jan 20, 2024

View reviewed changes

FastLee had a problem deploying to account-admin January 20, 2024 22:54 — with GitHub Actions Failure

FastLee added 6 commits January 22, 2024 15:50

Addressed Lint Comment

d9c694b

Addressed Lint Comment

f32f76a

Addressed Lint Comment

cf21bf0

Addressed Lint Comment

9e07717

Addressed Lint Comment

5a02c67

Addressed Lint Comment

df5864b

FastLee force-pushed the feature/aws-instance-profiles branch from a941fb9 to df5864b Compare January 22, 2024 21:49

FastLee temporarily deployed to account-admin January 22, 2024 21:49 — with GitHub Actions Inactive

Fixed unit test

a526d10

FastLee temporarily deployed to account-admin January 22, 2024 22:00 — with GitHub Actions Inactive

Unit test

6fff103

FastLee temporarily deployed to account-admin January 23, 2024 02:46 — with GitHub Actions Inactive

Addressed FMT Issues

e04e752

FastLee had a problem deploying to account-admin January 23, 2024 14:22 — with GitHub Actions Failure

Addressed FMT Issues

5cfdd8a

FastLee had a problem deploying to account-admin January 23, 2024 14:25 — with GitHub Actions Failure

nfx approved these changes Jan 23, 2024

View reviewed changes

FastLee had a problem deploying to account-admin January 23, 2024 15:03 — with GitHub Actions Failure

FastLee enabled auto-merge January 23, 2024 15:51

FastLee temporarily deployed to account-admin January 23, 2024 16:06 — with GitHub Actions Inactive

nfx disabled auto-merge January 23, 2024 16:48

nfx merged commit d4abc27 into main Jan 23, 2024

nfx deleted the feature/aws-instance-profiles branch January 23, 2024 16:48

nfx mentioned this pull request Jan 26, 2024

Release v0.11.0 #848

Merged

nfx mentioned this pull request Jan 30, 2024

[FEATURE]: databricks labs ucx uc-compatible-roles for AWS #861

Closed

1 task

Conversation

FastLee commented Jan 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Linked issues

Functionality

Tests

Uh oh!

codecov bot commented Jan 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

FastLee commented Jan 19, 2024 •

edited

Loading

codecov bot commented Jan 19, 2024 •

edited

Loading