[BUG]: Failed to evaluate hive_metastore.default.tableName table size #1913
Description
Is there an existing issue for this?
- I have searched the existing issues
Current Behavior
Hello Team,
I was able to enable UCX on one of our workspace and we are seeing below error in estimate_table_size_for_migration task while running assessment.
Error Details:
01:37:49 WARN [d.l.u.hive_metastore.table_size] Failed to evaluate hive_metastore.default.audit_recon_test_sdn table size. Table not found.
01:37:49 WARN [d.l.u.hive_metastore.table_size] Failed to evaluate hive_metastore.default.auto22 table size. Table not found.
01:47:19 ERROR [d.l.u.hive_metastore.table_size] Failed to evaluate hive_metastore.default.lapdata5 table size: : Traceback (most recent call last):
File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/databricks/labs/ucx/hive_metastore/table_size.py", line 77, in _safe_get_table_size
return self._spark._jsparkSession.table(table_full_name).queryExecution().analyzed().stats().sizeInBytes()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/databricks/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1355, in call
return_value = get_return_value(
^^^^^^^^^^^^^^^^^
File "/databricks/spark/python/pyspark/errors/exceptions/captured.py", line 254, in deco
raise converted from None
pyspark.errors.exceptions.captured.AnalysisException: [DELTA_INVALID_FORMAT] Incompatible format detected.
A transaction log for Delta was found at dbfs:/com/lap_tool/data/_delta_log,
but you are trying to read from dbfs:/com/lap_tool/data using format("text"). You must use
'format("delta")' when reading and writing to a delta table.
To learn more about Delta, see https://docs.databricks.com/delta/index.html
Could you please help me.
Expected Behavior
Assessment workflow should complete with out errors.
Steps To Reproduce
No response
Cloud
AWS
Operating System
Windows
Version
latest via Databricks CLI
Relevant log output
01:37:47 WARN [d.l.u.hive_metastore.table_size] Failed to evaluate hive_metastore.default.assessment_input table size. Table not found.
01:37:47 WARN [d.l.u.hive_metastore.table_size] Failed to evaluate hive_metastore.default.ata_weights_test table size. Table not found.
01:37:48 WARN [d.l.u.hive_metastore.table_size] Failed to evaluate hive_metastore.default.att_hold table size. Table not found.
01:37:49 WARN [d.l.u.hive_metastore.table_size] Failed to evaluate hive_metastore.default.audit_recon_test_sdn table size. Table not found.
01:37:49 WARN [d.l.u.hive_metastore.table_size] Failed to evaluate hive_metastore.default.auto22 table size. Table not found.
01:47:19 ERROR [d.l.u.hive_metastore.table_size] Failed to evaluate hive_metastore.default.lapdata5 table size: : Traceback (most recent call last):
File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/databricks/labs/ucx/hive_metastore/table_size.py", line 77, in _safe_get_table_size
return self._spark._jsparkSession.table(table_full_name).queryExecution().analyzed().stats().sizeInBytes()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/databricks/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1355, in __call__
return_value = get_return_value(
^^^^^^^^^^^^^^^^^
File "/databricks/spark/python/pyspark/errors/exceptions/captured.py", line 254, in deco
raise converted from None
pyspark.errors.exceptions.captured.AnalysisException: [DELTA_INVALID_FORMAT] Incompatible format detected.
A transaction log for Delta was found at `dbfs:/com/lap_tool/data/_delta_log`,
but you are trying to read from `dbfs:/com/lap_tool/data` using format("text"). You must use
'format("delta")' when reading and writing to a delta table.
To learn more about Delta, see https://docs.databricks.com/delta/index.html