Fixed `make_dbfs_data_copy` fixture on AWS by nkvuong · Pull Request #1403 · databrickslabs/ucx

nkvuong · 2024-04-15T13:01:10Z

Changes

Switch to Command Execution API on AWS

Linked issues

Resolves #1378

Tests

manually tested
added integration tests
verified on staging environment (screenshot attached)

github-actions · 2024-04-15T13:21:45Z

❌ 140/141 passed, 3 flaky, 1 failed, 21 skipped, 1h42m16s total

❌ test_delete_ws_groups_should_delete_renamed_and_reflected_groups_only: Failed: DID NOT RAISE (47.248s)

Failed: DID NOT RAISE <class 'databricks.sdk.errors.platform.NotFound'>
21:53 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sgnzb: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sgnzb
21:53 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sgnzb', metastore_id=None, name='ucx_sgnzb', owner=None, properties=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
[gw1] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
21:53 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sgnzb: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sgnzb
21:53 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sgnzb', metastore_id=None, name='ucx_sgnzb', owner=None, properties=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
21:53 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='7942407560622568', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
21:53 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx_lYjv: https://DATABRICKS_HOST#setting/accounts/groups/416594192388451
21:53 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx_lYjv', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='416594192388451', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/7942407560622568', type=None, value='7942407560622568')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
21:53 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx_lYjv: https://accounts.CLOUD_ENVdatabricks.net/users/groups/736993316667608/members
21:53 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx_lYjv', entitlements=[], external_id=None, groups=[], id='736993316667608', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/7942407560622568', type=None, value='7942407560622568')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
21:53 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=Group) with id,displayName,externalId,meta...
21:53 INFO [databricks.labs.ucx.workspace_access.groups] Found 61 Group
21:53 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName,meta,externalId,members,roles,entitlements...
21:53 INFO [databricks.labs.ucx.workspace_access.groups] Found 8 WorkspaceGroup
21:53 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_sgnzb.groups] fetching groups inventory
21:53 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_sgnzb.groups] crawling new batch for groups
21:53 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName,meta,externalId,members,roles,entitlements...
21:53 INFO [databricks.labs.ucx.workspace_access.groups] Found 6 WorkspaceGroup
21:53 INFO [databricks.labs.ucx.workspace_access.groups] Listing account groups with id,displayName,externalId...
21:53 INFO [databricks.labs.ucx.workspace_access.groups] Found 500 account groups
21:53 INFO [databricks.labs.ucx.workspace_access.groups] Group listing provided, a subset of all groups will be migrated
21:53 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_sgnzb.groups] found 1 new records for groups
21:53 INFO [databricks.labs.ucx.workspace_access.groups] Renaming: ucx_lYjv -> ucx-temp-ucx_lYjv
21:53 INFO [databricks.labs.ucx.workspace_access.groups] Listing account groups with id,displayName,externalId...
21:53 INFO [databricks.labs.ucx.workspace_access.groups] Found 500 account groups
21:53 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=Group) with id,displayName,externalId,meta...
21:53 INFO [databricks.labs.ucx.workspace_access.groups] Found 61 Group
21:53 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_sgnzb.groups] fetching groups inventory
21:53 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName,meta,externalId,members,roles,entitlements...
21:53 INFO [databricks.labs.ucx.workspace_access.groups] Found 9 WorkspaceGroup
21:53 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=Group) with id,displayName,externalId,meta...
21:53 INFO [databricks.labs.ucx.workspace_access.groups] Found 61 Group
21:53 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_sgnzb.groups] fetching groups inventory
21:54 INFO [databricks.labs.ucx.workspace_access.groups] Deleting the workspace-level group ucx-temp-ucx_lYjv with id 416594192388451
21:54 INFO [databricks.labs.ucx.workspace_access.groups] Workspace-level group ucx-temp-ucx_lYjv with id 416594192388451 was deleted
21:53 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sgnzb: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sgnzb
21:53 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sgnzb', metastore_id=None, name='ucx_sgnzb', owner=None, properties=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
21:53 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='7942407560622568', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
21:53 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx_lYjv: https://DATABRICKS_HOST#setting/accounts/groups/416594192388451
21:53 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx_lYjv', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='416594192388451', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/7942407560622568', type=None, value='7942407560622568')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
21:53 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx_lYjv: https://accounts.CLOUD_ENVdatabricks.net/users/groups/736993316667608/members
21:53 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx_lYjv', entitlements=[], external_id=None, groups=[], id='736993316667608', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/7942407560622568', type=None, value='7942407560622568')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
21:53 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=Group) with id,displayName,externalId,meta...
21:53 INFO [databricks.labs.ucx.workspace_access.groups] Found 61 Group
21:53 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName,meta,externalId,members,roles,entitlements...
21:53 INFO [databricks.labs.ucx.workspace_access.groups] Found 8 WorkspaceGroup
21:53 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_sgnzb.groups] fetching groups inventory
21:53 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_sgnzb.groups] crawling new batch for groups
21:53 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName,meta,externalId,members,roles,entitlements...
21:53 INFO [databricks.labs.ucx.workspace_access.groups] Found 6 WorkspaceGroup
21:53 INFO [databricks.labs.ucx.workspace_access.groups] Listing account groups with id,displayName,externalId...
21:53 INFO [databricks.labs.ucx.workspace_access.groups] Found 500 account groups
21:53 INFO [databricks.labs.ucx.workspace_access.groups] Group listing provided, a subset of all groups will be migrated
21:53 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_sgnzb.groups] found 1 new records for groups
21:53 INFO [databricks.labs.ucx.workspace_access.groups] Renaming: ucx_lYjv -> ucx-temp-ucx_lYjv
21:53 INFO [databricks.labs.ucx.workspace_access.groups] Listing account groups with id,displayName,externalId...
21:53 INFO [databricks.labs.ucx.workspace_access.groups] Found 500 account groups
21:53 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=Group) with id,displayName,externalId,meta...
21:53 INFO [databricks.labs.ucx.workspace_access.groups] Found 61 Group
21:53 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_sgnzb.groups] fetching groups inventory
21:53 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName,meta,externalId,members,roles,entitlements...
21:53 INFO [databricks.labs.ucx.workspace_access.groups] Found 9 WorkspaceGroup
21:53 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=Group) with id,displayName,externalId,meta...
21:53 INFO [databricks.labs.ucx.workspace_access.groups] Found 61 Group
21:53 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_sgnzb.groups] fetching groups inventory
21:54 INFO [databricks.labs.ucx.workspace_access.groups] Deleting the workspace-level group ucx-temp-ucx_lYjv with id 416594192388451
21:54 INFO [databricks.labs.ucx.workspace_access.groups] Workspace-level group ucx-temp-ucx_lYjv with id 416594192388451 was deleted
21:54 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 schema fixtures
21:54 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sgnzb', metastore_id=None, name='ucx_sgnzb', owner=None, properties=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
21:54 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace user fixtures
21:54 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='7942407560622568', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
21:54 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 account group fixtures
21:54 DEBUG [databricks.labs.ucx.mixins.fixtures] removing account group fixture: Group(display_name='ucx_lYjv', entitlements=[], external_id=None, groups=[], id='736993316667608', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/7942407560622568', type=None, value='7942407560622568')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
21:54 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace group fixtures
21:54 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace group fixture: Group(display_name='ucx_lYjv', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='416594192388451', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/7942407560622568', type=None, value='7942407560622568')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
21:54 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while workspace group Group(display_name='ucx_lYjv', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='416594192388451', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/7942407560622568', type=None, value='7942407560622568')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>]) teardown: None Group with id 416594192388451 not found.
[gw1] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python

Flaky tests:

🤪 test_migrate_view (2m5.408s)
🤪 test_running_real_assessment_job (7m55.367s)
🤪 test_running_real_validate_groups_permissions_job (2m31.693s)

_{Running from acceptance #2329}

nfx · 2024-04-15T21:23:58Z

src/databricks/labs/ucx/mixins/fixtures.py

+        cluster = make_cluster(
+            single_node=True,
+            instance_pool_id=env_or_skip("TEST_INSTANCE_POOL_ID"),
+            aws_attributes=compute.AwsAttributes(instance_profile_arn=env_or_skip("TEST_WILDCARD_INSTANCE_PROFILE")),
+        ).result()
+        cmd_exec = CommandExecutor(ws.clusters, ws.command_execution, lambda: cluster.cluster_id)


Suggested change

cluster = make_cluster(

single_node=True,

instance_pool_id=env_or_skip("TEST_INSTANCE_POOL_ID"),

aws_attributes=compute.AwsAttributes(instance_profile_arn=env_or_skip("TEST_WILDCARD_INSTANCE_PROFILE")),

).result()

cmd_exec = CommandExecutor(ws.clusters, ws.command_execution, lambda: cluster.cluster_id)

cmd_exec = CommandExecutor(ws.clusters, ws.command_execution, lambda: env_or_skip("TEST_DEFAULT_CLUSTER_ID"))

Why do we have to wait for nodes to free up if we have a running cluster?...

nfx

Lgtm

nkvuong requested review from a team and dleiva04 April 15, 2024 13:01

nkvuong had a problem deploying to account-admin April 15, 2024 13:01 — with GitHub Actions Failure

nkvuong temporarily deployed to account-admin April 15, 2024 13:21 — with GitHub Actions Inactive

nkvuong mentioned this pull request Apr 15, 2024

Add support for migrating Table ACL for interactive cluster in AWS using Instance Profile #1285

Merged

11 tasks

nkvuong requested review from nfx and removed request for dleiva04 April 15, 2024 17:03

nfx requested changes Apr 15, 2024

View reviewed changes

nkvuong added 3 commits April 15, 2024 22:43

fix make_dbfs_data_copy fixture

a228112

make fmt

e23148c

feedback

b9f8d71

nkvuong had a problem deploying to account-admin April 15, 2024 21:46 — with GitHub Actions Failure

nkvuong added 3 commits April 15, 2024 22:54

fix make_dbfs_data_copy fixture

8e609bd

make fmt

2c66379

feedback

504ab33

nkvuong force-pushed the fix/make_dbfs_data_copy branch from 59194c1 to 504ab33 Compare April 15, 2024 21:55

nfx approved these changes Apr 16, 2024

View reviewed changes

nfx merged commit 4418950 into main Apr 16, 2024

nfx deleted the fix/make_dbfs_data_copy branch April 16, 2024 07:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixed `make_dbfs_data_copy` fixture on AWS#1403

Fixed `make_dbfs_data_copy` fixture on AWS#1403
nfx merged 6 commits intomainfrom
fix/make_dbfs_data_copy

nkvuong commented Apr 15, 2024

Uh oh!

github-actions bot commented Apr 15, 2024 •

edited

Loading

Uh oh!

nfx Apr 15, 2024

Uh oh!

nfx left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

nkvuong commented Apr 15, 2024

Changes

Linked issues

Tests

Uh oh!

github-actions bot commented Apr 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nfx Apr 15, 2024

Choose a reason for hiding this comment

Uh oh!

nfx left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions bot commented Apr 15, 2024 •

edited

Loading