Fixed `test_migrate_view` timeout by nkvuong · Pull Request #1385 · databrickslabs/ucx

nkvuong · 2024-04-12T14:42:39Z

Changes

test_migrate_view timed out because migration_status_refresher table was not created

Tests

manually tested
verified on staging environment (screenshot attached)

Signed-off-by: Vuong <[email protected]>

nfx

lgtm

github-actions · 2024-04-12T15:07:52Z

❌ 139/140 passed, 2 flaky, 1 failed, 20 skipped, 1h51m4s total

❌ test_delete_ws_groups_should_delete_renamed_and_reflected_groups_only: Failed: DID NOT RAISE (41.212s)

Failed: DID NOT RAISE <class 'databricks.sdk.errors.platform.NotFound'>
14:49 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_srdqm: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_srdqm
14:49 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_srdqm', metastore_id=None, name='ucx_srdqm', owner=None, properties=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
[gw4] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
14:49 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_srdqm: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_srdqm
14:49 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_srdqm', metastore_id=None, name='ucx_srdqm', owner=None, properties=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
14:49 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='5786057723352366', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
14:49 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx_Nu6L: https://DATABRICKS_HOST#setting/accounts/groups/1038758588179280
14:49 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx_Nu6L', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='1038758588179280', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/5786057723352366', type=None, value='5786057723352366')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
14:49 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx_Nu6L: https://accounts.CLOUD_ENVdatabricks.net/users/groups/16788938632621/members
14:49 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx_Nu6L', entitlements=[], external_id=None, groups=[], id='16788938632621', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/5786057723352366', type=None, value='5786057723352366')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=Group) with id,displayName,externalId,meta...
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Found 57 Group
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName,meta,externalId,members,roles,entitlements...
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Found 3 WorkspaceGroup
14:49 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_srdqm.groups] fetching groups inventory
14:49 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_srdqm.groups] crawling new batch for groups
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName,meta,externalId,members,roles,entitlements...
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Found 4 WorkspaceGroup
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Listing account groups with id,displayName,externalId...
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Found 431 account groups
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Group listing provided, a subset of all groups will be migrated
14:49 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_srdqm.groups] found 1 new records for groups
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Renaming: ucx_Nu6L -> ucx-temp-ucx_Nu6L
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Listing account groups with id,displayName,externalId...
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Found 431 account groups
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=Group) with id,displayName,externalId,meta...
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Found 57 Group
14:49 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_srdqm.groups] fetching groups inventory
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName,meta,externalId,members,roles,entitlements...
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Found 4 WorkspaceGroup
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=Group) with id,displayName,externalId,meta...
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Found 59 Group
14:49 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_srdqm.groups] fetching groups inventory
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Deleting the workspace-level group ucx-temp-ucx_Nu6L with id 1038758588179280
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Workspace-level group ucx-temp-ucx_Nu6L with id 1038758588179280 was deleted
14:49 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_srdqm: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_srdqm
14:49 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_srdqm', metastore_id=None, name='ucx_srdqm', owner=None, properties=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
14:49 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='5786057723352366', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
14:49 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx_Nu6L: https://DATABRICKS_HOST#setting/accounts/groups/1038758588179280
14:49 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx_Nu6L', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='1038758588179280', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/5786057723352366', type=None, value='5786057723352366')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
14:49 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx_Nu6L: https://accounts.CLOUD_ENVdatabricks.net/users/groups/16788938632621/members
14:49 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx_Nu6L', entitlements=[], external_id=None, groups=[], id='16788938632621', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/5786057723352366', type=None, value='5786057723352366')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=Group) with id,displayName,externalId,meta...
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Found 57 Group
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName,meta,externalId,members,roles,entitlements...
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Found 3 WorkspaceGroup
14:49 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_srdqm.groups] fetching groups inventory
14:49 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_srdqm.groups] crawling new batch for groups
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName,meta,externalId,members,roles,entitlements...
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Found 4 WorkspaceGroup
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Listing account groups with id,displayName,externalId...
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Found 431 account groups
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Group listing provided, a subset of all groups will be migrated
14:49 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_srdqm.groups] found 1 new records for groups
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Renaming: ucx_Nu6L -> ucx-temp-ucx_Nu6L
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Listing account groups with id,displayName,externalId...
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Found 431 account groups
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=Group) with id,displayName,externalId,meta...
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Found 57 Group
14:49 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_srdqm.groups] fetching groups inventory
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName,meta,externalId,members,roles,entitlements...
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Found 4 WorkspaceGroup
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=Group) with id,displayName,externalId,meta...
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Found 59 Group
14:49 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_srdqm.groups] fetching groups inventory
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Deleting the workspace-level group ucx-temp-ucx_Nu6L with id 1038758588179280
14:49 INFO [databricks.labs.ucx.workspace_access.groups] Workspace-level group ucx-temp-ucx_Nu6L with id 1038758588179280 was deleted
14:50 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 schema fixtures
14:50 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_srdqm', metastore_id=None, name='ucx_srdqm', owner=None, properties=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
14:50 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace user fixtures
14:50 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='5786057723352366', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
14:50 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 account group fixtures
14:50 DEBUG [databricks.labs.ucx.mixins.fixtures] removing account group fixture: Group(display_name='ucx_Nu6L', entitlements=[], external_id=None, groups=[], id='16788938632621', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/5786057723352366', type=None, value='5786057723352366')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
14:50 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace group fixtures
14:50 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace group fixture: Group(display_name='ucx_Nu6L', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='1038758588179280', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/5786057723352366', type=None, value='5786057723352366')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
14:50 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while workspace group Group(display_name='ucx_Nu6L', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='1038758588179280', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/5786057723352366', type=None, value='5786057723352366')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>]) teardown: None Group with id 1038758588179280 not found.
[gw4] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python

Flaky tests:

🤪 test_migrate_managed_tables_with_acl (2m27.203s)
🤪 test_replace_workspace_groups_with_account_groups (3m26.205s)

_{Running from acceptance #2265}

fix test_migrate_view timeout

9e5f938

Signed-off-by: Vuong <[email protected]>

nkvuong requested review from a team and dleiva04 April 12, 2024 14:42

nkvuong had a problem deploying to account-admin April 12, 2024 14:42 — with GitHub Actions Failure

nfx approved these changes Apr 12, 2024

View reviewed changes

nfx changed the title ~~fixed test_migrate_view timeout~~ Fixed test_migrate_view timeout Apr 12, 2024

nfx merged commit 679af39 into main Apr 12, 2024

nfx deleted the fix/test_migrate_view branch April 12, 2024 14:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixed `test_migrate_view` timeout#1385

Fixed `test_migrate_view` timeout#1385
nfx merged 1 commit intomainfrom
fix/test_migrate_view

nkvuong commented Apr 12, 2024

Uh oh!

nfx left a comment

Uh oh!

github-actions bot commented Apr 12, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

nkvuong commented Apr 12, 2024

Changes

Tests

Uh oh!

nfx left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 12, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants