Skip to content

[FEATURE]: Connect _migrate_table_create_ctas to migrate-tables workflow #332

Closed
#1621
Closed
[FEATURE]: Connect _migrate_table_create_ctas to migrate-tables workflow#332
#1621
Assignees
HariGS-DB
Labels
feat/workflowtriggered as a Databricks Job managed by UCXmigrate/managedgo/uc/upgrade Upgrade Managed Tables and Jobs
@nfx

Description

@nfx
nfx
opened on Sep 29, 2023

most of the code is done in:

Diff:

ucx/src/databricks/labs/ucx/hive_metastore/table_migrate.py

Lines 286 to 299 in 9cb9dbf

def _migrate_table_create_ctas(self, src_table: Table, rule: Rule, grants: list[Grant] | None = None):
table_migrate_sql = self._get_create_ctas_sql(src_table, rule)
logger.debug(f"Migrating table (Create Like) {src_table.key} to using SQL query: {table_migrate_sql}")
self._backend.execute(table_migrate_sql)
self._backend.execute(src_table.sql_alter_to(rule.as_uc_table_key))
self._backend.execute(src_table.sql_alter_from(rule.as_uc_table_key, self._ws.get_workspace_id()))
return self._migrate_acl(src_table, rule, grants)
def _get_create_ctas_sql(self, src_table: Table, rule: Rule) -> str:
create_sql = (
f"CREATE TABLE IF NOT EXISTS {escape_sql_identifier(rule.as_uc_table_key)} "
f"AS SELECT * FROM {src_table.safe_sql_key}"
)
return create_sql

Tasks:

  • Copy this table using CTAS
  • Copy the Metadata with CREATE TABLE ... LIKE ...
  • Skip tables as needed based on size threshold or an exception list
  • Update the source table with upgraded_to property
  • Link it to the workflow as task

Metadata

Metadata

Assignees

Labels

feat/workflowtriggered as a Databricks Job managed by UCXmigrate/managedgo/uc/upgrade Upgrade Managed Tables and Jobs

Type

No type

Projects

UCX

Status

No status

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions