Skip to content

Test failure: test_table_migration_job_refreshes_migration_status[hiveserde-migrate-external-tables-ctas] #1615

@github-actions

Description

@github-actions
❌ test_table_migration_job_refreshes_migration_status[hiveserde-migrate-external-tables-ctas]: AssertionError: No migration status found found for TableType.EXTERNAL hive_metastore.hiveserde_in_place_6ybya.parquet_serde_6ybya (14m43.3s)
AssertionError: No migration status found found for TableType.EXTERNAL hive_metastore.hiveserde_in_place_6ybya.parquet_serde_6ybya
assert 0 == 1
 +  where 0 = len([])
07:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/ukAo
07:03 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.hiveserde_in_place_6ybya: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_6ybya
07:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.hiveserde_in_place_6ybya', metastore_id=None, name='hiveserde_in_place_6ybya', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_storage_dir fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_6ybya
07:03 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_6ybya.parquet_serde_6ybya: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_6ybya/parquet_serde_6ybya
07:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_6ybya.parquet_serde_6ybya', metastore_id=None, name='parquet_serde_6ybya', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='hiveserde_in_place_6ybya', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_6ybya/parquet_serde_6ybya', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:03 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_6ybya.orc_serde_6ybya: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_6ybya/orc_serde_6ybya
07:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_6ybya.orc_serde_6ybya', metastore_id=None, name='orc_serde_6ybya', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='hiveserde_in_place_6ybya', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_6ybya/orc_serde_6ybya', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:03 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_6ybya.avro_serde_6ybya: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_6ybya/avro_serde_6ybya
07:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_6ybya.avro_serde_6ybya', metastore_id=None, name='avro_serde_6ybya', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='hiveserde_in_place_6ybya', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_6ybya/avro_serde_6ybya', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=<CatalogType.MANAGED_CATALOG: 'MANAGED_CATALOG'>, comment='', connection_name=None, created_at=1714719805246, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_clqed', isolation_mode=<IsolationMode.OPEN: 'OPEN'>, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_clqed', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=<CatalogInfoSecurableKind.CATALOG_STANDARD: 'CATALOG_STANDARD'>, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1714719805246, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d')
07:03 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_clqed.hiveserde_in_place_6ybya: https://DATABRICKS_HOST/explore/data/ucx_clqed/hiveserde_in_place_6ybya
07:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_clqed', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_clqed.hiveserde_in_place_6ybya', metastore_id=None, name='hiveserde_in_place_6ybya', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:03 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sd2m1: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sd2m1
07:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sd2m1', metastore_id=None, name='ucx_sd2m1', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
[gw1] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
07:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/ukAo
07:03 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.hiveserde_in_place_6ybya: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_6ybya
07:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.hiveserde_in_place_6ybya', metastore_id=None, name='hiveserde_in_place_6ybya', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_storage_dir fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_6ybya
07:03 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_6ybya.parquet_serde_6ybya: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_6ybya/parquet_serde_6ybya
07:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_6ybya.parquet_serde_6ybya', metastore_id=None, name='parquet_serde_6ybya', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='hiveserde_in_place_6ybya', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_6ybya/parquet_serde_6ybya', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:03 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_6ybya.orc_serde_6ybya: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_6ybya/orc_serde_6ybya
07:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_6ybya.orc_serde_6ybya', metastore_id=None, name='orc_serde_6ybya', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='hiveserde_in_place_6ybya', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_6ybya/orc_serde_6ybya', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:03 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_6ybya.avro_serde_6ybya: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_6ybya/avro_serde_6ybya
07:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_6ybya.avro_serde_6ybya', metastore_id=None, name='avro_serde_6ybya', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='hiveserde_in_place_6ybya', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_6ybya/avro_serde_6ybya', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=<CatalogType.MANAGED_CATALOG: 'MANAGED_CATALOG'>, comment='', connection_name=None, created_at=1714719805246, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_clqed', isolation_mode=<IsolationMode.OPEN: 'OPEN'>, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_clqed', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=<CatalogInfoSecurableKind.CATALOG_STANDARD: 'CATALOG_STANDARD'>, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1714719805246, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d')
07:03 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_clqed.hiveserde_in_place_6ybya: https://DATABRICKS_HOST/explore/data/ucx_clqed/hiveserde_in_place_6ybya
07:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_clqed', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_clqed.hiveserde_in_place_6ybya', metastore_id=None, name='hiveserde_in_place_6ybya', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:03 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sd2m1: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sd2m1
07:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sd2m1', metastore_id=None, name='ucx_sd2m1', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:03 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.JlmR/config.yml) doesn't exist.
07:03 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
07:03 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
07:03 INFO [databricks.labs.ucx.install] Fetching installations...
07:03 WARNING [databricks.labs.ucx.install] Existing installation at /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.JlmR is corrupted. Skipping...
07:03 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
07:03 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:11 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:11 INFO [databricks.labs.ucx.install] Installing UCX v0.22.1+2720240503071158
07:11 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:12 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:12 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:12 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
07:12 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:12 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:12 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:12 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:12 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:12 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:12 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:12 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.JlmR/README for the next steps.
07:12 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-external-tables-ctas job: https://DATABRICKS_HOST#job/48395959256828
07:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/ukAo
07:03 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.hiveserde_in_place_6ybya: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_6ybya
07:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.hiveserde_in_place_6ybya', metastore_id=None, name='hiveserde_in_place_6ybya', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_storage_dir fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_6ybya
07:03 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_6ybya.parquet_serde_6ybya: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_6ybya/parquet_serde_6ybya
07:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_6ybya.parquet_serde_6ybya', metastore_id=None, name='parquet_serde_6ybya', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='hiveserde_in_place_6ybya', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_6ybya/parquet_serde_6ybya', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:03 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_6ybya.orc_serde_6ybya: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_6ybya/orc_serde_6ybya
07:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_6ybya.orc_serde_6ybya', metastore_id=None, name='orc_serde_6ybya', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='hiveserde_in_place_6ybya', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_6ybya/orc_serde_6ybya', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:03 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_6ybya.avro_serde_6ybya: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_6ybya/avro_serde_6ybya
07:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_6ybya.avro_serde_6ybya', metastore_id=None, name='avro_serde_6ybya', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='hiveserde_in_place_6ybya', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_6ybya/avro_serde_6ybya', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=<CatalogType.MANAGED_CATALOG: 'MANAGED_CATALOG'>, comment='', connection_name=None, created_at=1714719805246, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_clqed', isolation_mode=<IsolationMode.OPEN: 'OPEN'>, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_clqed', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=<CatalogInfoSecurableKind.CATALOG_STANDARD: 'CATALOG_STANDARD'>, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1714719805246, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d')
07:03 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_clqed.hiveserde_in_place_6ybya: https://DATABRICKS_HOST/explore/data/ucx_clqed/hiveserde_in_place_6ybya
07:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_clqed', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_clqed.hiveserde_in_place_6ybya', metastore_id=None, name='hiveserde_in_place_6ybya', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:03 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sd2m1: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sd2m1
07:03 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sd2m1', metastore_id=None, name='ucx_sd2m1', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:03 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.JlmR/config.yml) doesn't exist.
07:03 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
07:03 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
07:03 INFO [databricks.labs.ucx.install] Fetching installations...
07:03 WARNING [databricks.labs.ucx.install] Existing installation at /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.JlmR is corrupted. Skipping...
07:03 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
07:03 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:11 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:11 INFO [databricks.labs.ucx.install] Installing UCX v0.22.1+2720240503071158
07:11 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:12 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:12 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:12 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
07:12 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:12 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:12 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:12 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:12 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:12 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:12 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:12 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.JlmR/README for the next steps.
07:12 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-external-tables-ctas job: https://DATABRICKS_HOST#job/48395959256828
07:17 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 make_storage_dir fixtures
07:17 DEBUG [databricks.labs.ucx.mixins.fixtures] removing make_storage_dir fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_6ybya
07:17 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 make_dbfs_data_copy fixtures
07:17 DEBUG [databricks.labs.ucx.mixins.fixtures] removing make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/ukAo
07:17 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 cluster fixtures
07:17 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 catalog fixtures
07:17 DEBUG [databricks.labs.ucx.mixins.fixtures] removing catalog fixture: CatalogInfo(browse_only=False, catalog_type=<CatalogType.MANAGED_CATALOG: 'MANAGED_CATALOG'>, comment='', connection_name=None, created_at=1714719805246, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_clqed', isolation_mode=<IsolationMode.OPEN: 'OPEN'>, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_clqed', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=<CatalogInfoSecurableKind.CATALOG_STANDARD: 'CATALOG_STANDARD'>, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1714719805246, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d')
07:17 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while catalog CatalogInfo(browse_only=False, catalog_type=<CatalogType.MANAGED_CATALOG: 'MANAGED_CATALOG'>, comment='', connection_name=None, created_at=1714719805246, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_clqed', isolation_mode=<IsolationMode.OPEN: 'OPEN'>, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_clqed', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=<CatalogInfoSecurableKind.CATALOG_STANDARD: 'CATALOG_STANDARD'>, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1714719805246, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d') teardown: Catalog 'ucx_clqed' does not exist.
07:17 INFO [databricks.labs.ucx.install] Deleting UCX v0.22.1+2720240503071740 from https://DATABRICKS_HOST
07:17 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_sd2m1
07:17 INFO [databricks.labs.ucx.install] Deleting jobs
07:17 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=862643181163144.
07:17 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=649914824330201.
07:17 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=250417275660785.
07:17 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=48395959256828.
07:17 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=47638626206043.
07:17 INFO [databricks.labs.ucx.install] Deleting failing job_id=128916440497905.
07:17 INFO [databricks.labs.ucx.install] Deleting assessment job_id=19206548252845.
07:17 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=59217357936323.
07:17 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=262686142010312.
07:17 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=18721649091754.
07:17 INFO [databricks.labs.ucx.install] Deleting cluster policy
07:17 ERROR [databricks.labs.ucx.install] UCX Policy already deleted
07:17 INFO [databricks.labs.ucx.install] Deleting secret scope
07:17 INFO [databricks.labs.ucx.install] UnInstalling UCX complete
07:17 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace user fixtures
07:17 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 account group fixtures
07:17 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace group fixtures
07:17 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
07:17 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 table fixtures
07:17 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_6ybya.parquet_serde_6ybya', metastore_id=None, name='parquet_serde_6ybya', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='hiveserde_in_place_6ybya', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_6ybya/parquet_serde_6ybya', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:17 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_6ybya.orc_serde_6ybya', metastore_id=None, name='orc_serde_6ybya', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='hiveserde_in_place_6ybya', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_6ybya/orc_serde_6ybya', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:17 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_6ybya.avro_serde_6ybya', metastore_id=None, name='avro_serde_6ybya', owner=None, pipeline_id=None, properties=None, row_filter=None, schema_name='hiveserde_in_place_6ybya', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_6ybya/avro_serde_6ybya', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:17 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 schema fixtures
07:17 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.hiveserde_in_place_6ybya', metastore_id=None, name='hiveserde_in_place_6ybya', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:17 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_clqed', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_clqed.hiveserde_in_place_6ybya', metastore_id=None, name='hiveserde_in_place_6ybya', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:17 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sd2m1', metastore_id=None, name='ucx_sd2m1', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
[gw1] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python

Running from nightly #49

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions