Skip to content

Allow skipping DBFS Root Delta Table if size exceeds preconfigured threshold #334

Open
Feature
Open
Allow skipping DBFS Root Delta Table if size exceeds preconfigured threshold#334
Feature
Labels
feat/workflowtriggered as a Databricks Job managed by UCXmigrate/managedgo/uc/upgrade Upgrade Managed Tables and Jobs
@nfx

Description

@nfx
nfx
opened on Sep 29, 2023

Based on

ucx/src/databricks/labs/ucx/hive_metastore/tables.py

Lines 101 to 105 in db580a9

def sql_migrate_dbfs(self, target_table_key):
if not self.is_delta:
msg = f"{self.key} is not DELTA: {self.table_format}"
raise ValueError(msg)
return f"CREATE TABLE IF NOT EXISTS {target_table_key} DEEP CLONE {self.key};"

and

ucx/src/databricks/labs/ucx/hive_metastore/table_migrate.py

Lines 277 to 284 in ec1801c

def _migrate_dbfs_root_table(self, src_table: Table, rule: Rule, grants: list[Grant] | None = None):
target_table_key = rule.as_uc_table_key
table_migrate_sql = src_table.sql_migrate_dbfs(target_table_key)
logger.debug(f"Migrating managed table {src_table.key} to using SQL query: {table_migrate_sql}")
self._backend.execute(table_migrate_sql)
self._backend.execute(src_table.sql_alter_to(rule.as_uc_table_key))
self._backend.execute(src_table.sql_alter_from(rule.as_uc_table_key, self._ws.get_workspace_id()))
return self._migrate_acl(src_table, rule, grants)

Tasks remaining:

  • Allow skipping DBFS Root Delta Table if size exceeds preconfigured threshold

Metadata

Metadata

Assignees

No one assigned

    Labels

    feat/workflowtriggered as a Databricks Job managed by UCXmigrate/managedgo/uc/upgrade Upgrade Managed Tables and Jobs

    Type

    Feature

    Projects

    UCX

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions