* Added troubleshooting guide for self-signed SSL cert related error ([#2346](#2346)). In this release, we have added a troubleshooting guide to the README file to address a specific error that may occur when connecting from a local machine to a Databricks Account and Workspace using a web proxy and self-signed SSL certificate. This error, SSLCertVerificationError, can prevent UCX from connecting to the Account and Workspace. To resolve this issue, users can now set the `REQUESTS_CA_BUNDLE` and `CURL_CA_BUNDLE` environment variables to force the requests library to set `verify=False`, and set the `SSL_CERT_DIR` env var pointing to the proxy CA cert for the urllib3 library. This guide will help users understand and resolve this error, making it easier to connect to Databricks Accounts and Workspaces using a web proxy and self-signed SSL certificate.
* Code Compatibility Dashboard: Fix broken links ([#2347](#2347)). In this release, we have addressed and resolved two issues in the Code Compatibility Dashboard of the UCX Migration (Main) project, enhancing its overall usability. Previously, the Markdown panel contained a broken link to the workflow due to an incorrect anchor, and the links in the table widget to the workflow and task definitions did not render correctly. These problems have been rectified, and the dashboard has been manually tested and verified in a staging environment. Additionally, we have updated the `invisibleColumns` section in the SQL file by changing the `fieldName` attribute to 'name', which will now display the `workflow_id` as a link. Before and after screenshots have been provided for visual reference. The corresponding workflow is now referred to as "Jobs Static Code Analysis Workflow".
* Filter out missing import problems for imports within a try-except clause with ImportError ([#2332](#2332)). This release introduces changes to handle missing import problems within a try-except clause that catches ImportError. A new method, `_filter_import_problem_in_try_except`, has been added to filter out import-not-found issues when they occur in such a clause, preventing unnecessary build failures. The `_register_import` method now returns an Iterable[DependencyProblem] instead of yielding problems directly. Supporting classes and methods, including Dependency, DependencyGraph, and DependencyProblem from the databricks.labs.ucx.source_code.graph module, as well as FileLoader and PythonCodeAnalyzer from the databricks.labs.ucx.source_code.notebooks.cells module, have been added. The ImportSource.extract_from_tree method has been updated to accept a DependencyProblem object as an argument. Additionally, a new test case has been included for the scenario where a failing import in a try-except clause goes unreported. Issue [#1705](#1705) has been resolved, and unit tests have been added to ensure proper functionality.
* Fixed `report-account-compatibility` cli command docstring ([#2340](#2340)). In this release, we have updated the `report-account-compatibility` CLI command's docstring to accurately reflect its functionality, addressing a previous issue where it inadvertently duplicated the `sync-workspace-info` command's description. This command now provides a clear and concise explanation of its purpose: "Report compatibility of all workspaces available in the account." Upon execution, it generates a readiness report for the account, specifically focusing on workspaces where ucx is installed. This enhancement improves the clarity of the CLI's functionality for software engineers, enabling them to understand and effectively utilize the `report-account-compatibility` command.
* Fixed broken table migration workflow links in README ([#2286](#2286)). In this release, we have made significant improvements to the README file of our open-source library, including fixing broken links and adding a mermaid flowchart to demonstrate the table migration workflows. The table migration workflow has been renamed to the table migration process, which includes migrating Delta tables, non-Delta tables, external tables, and views. Two optional workflows have been added for migrating HiveSerDe tables in place and for migrating external tables using CTAS. Additionally, the commands related to table migration have been updated, with the table migration workflow being renamed to the table migration process. These changes are aimed at providing a more comprehensive understanding of the table migration process and enhancing the overall user experience.
* Fixed dashboard queries fail when default catalog is not `hive_metastore` ([#2278](#2278)). In this release, we have addressed an issue where dashboard queries fail when the default catalog is not set to `hive_metastore`. This has been achieved by modifying the existing `databricks labs ucx install` command to always include the `hive_metastore` namespace in dashboard queries. Additionally, the code has been updated to add the `hive_metastore` namespace to the `DashboardMetadata` object used in creating a dashboard from SQL queries in a folder, ensuring queries are executed in the correct database. The commit also includes modifications to the `test_install.py` unit test file to ensure the installation process correctly handles specific configurations related to the `ucx` namespace for managing data storage and retrieval. The changes have been manually tested and verified on a staging environment.
* Improve group migration error reporting ([#2344](#2344)). This PR introduces enhancements to the group migration dashboard, focusing on improved error reporting and a more informative user experience. The documentation widgets have been fine-tuned, and the failed-migration widget now provides formatted failure information with a link to the failed job run. The dashboard will display only failures from the latest workflow run, complete with logs. A new link to the job list has been added in the [workflows](/jobs) section of the documentation to assist users in identifying and troubleshooting issues. Additionally, the SQL query for retrieving group migration failure information has been refactored, improving readability and extracting relevant data using regular expressions. The changes have been tested and verified on the staging environment, providing clearer and more actionable insights during group migrations. The PR is related to previous work in [#2333](#2333) and [#1914](#1914), with updates to the UCX Migration (Groups) dashboard, but no new methods have been added.
* Improve type checking in cli command ([#2335](#2335)). This release introduces enhanced type checking in the command line interface (CLI) of our open-source library, specifically in the `lint_local_code` function of the `cli.py` file. By utilizing a newly developed local code linter object, the function now performs more rigorous and accurate type checking for potential issues in the local code. While the functionality remains consistent, this improvement is expected to prevent similar occurrences like issue [#2221](#2221), ensuring more robust and reliable code. This change underscores our commitment to delivering a high-quality, efficient, and developer-friendly library.
* Lint dependencies in context ([#2236](#2236)). The `InheritedContext` class has been introduced to gather code fragments from parent files or notebooks during linting of child files or notebooks, addressing issues [#2155](#2155), [#2156](#2156), and [#2221](#2221). This new feature includes the addition of the `InheritedContext` class, with methods for building instances from a route of dependencies, appending other `InheritedContext` instances, and finalizing them for use with linting. The `DependencyGraph` class has been updated to support the new functionality, and various classes, methods, and functions for handling the linter context have been added or updated. Unit, functional, and integration tests have been added to ensure the correct functioning of the changes, which improve the linting functionality by allowing it to consider the broader context of the codebase.
* Make ucx pylsp plugin configurable ([#2280](#2280)). This commit introduces the ability to configure the ucx pylsp plugin with cluster information, which can be provided either in a file or by a client and is managed by the pylsp infrastructure. The Spark Connect linter is now only applied to UC Shared clusters, as Single-User clusters run in Spark Classic mode. A new entry point `pylsp_ucx` has been added to the pylsp configuration file. The changes affect the pylsp plugin configuration and the application of the Spark Connect linter. Unit tests and manual testing have been conducted, but integration tests and verification on a staging environment are not included in this release.
* New dashboard: group migration, showing groups that failed to migrate ([#2333](#2333)). In this release, we have developed a new dashboard for monitoring group migration in the UCX Migration (Groups) workspace. This dashboard includes a widget displaying messages related to groups that failed to migrate during the `migrate-groups-experimental` workflow, aiding users in identifying and addressing migration issues. The group migration process consists of several steps, including renaming workspace groups, provisioning account-level groups, and replicating permissions. The release features new methods for displaying and monitoring migration-related messages, as well as links to documentation and workflows for assessing, validating, and removing workspace-level groups post-migration. The new dashboard is currently not connected to the existing system, but it has undergone manual testing and verification on the staging environment. The changes include the addition of a new SQL query file to implement the logic for fetching group migration failures and a new Markdown file displaying the Group Migration Failures section.
* Support spaces in run cmd args ([#2330](#2330)). The recent commit resolves an issue where the system had trouble handling spaces in command-line arguments when running subprocesses. The previous implementation only accepted a full command line, which it would split on spaces, causing problems when the command line contained arguments with spaces. The new implementation supports argument lists, which are passed `as is` to `Popen`, allowing for proper handling of command lines with spaces. This change is incorporated in the `run_command` function of the `utils.py` file and the `_install_pip` method of the `PythonLibraryResolver` class. The `shlex.join()` function has been replaced with direct string formatting for increased flexibility. The feature is intended for use with the `PythonLibraryResolver` class and is co-authored by Eric Vergnaud and Andrew Snare. Integration tests have been enabled to ensure the proper functioning of the updated code.
* Updated error messages for SparkConnect linter ([#2348](#2348)). The SparkConnect linter's error messages have been updated to improve clarity and precision. The term `UC Shared clusters` has been replaced with `Unity Catalog clusters in Shared access mode` throughout the codebase, affecting messages related to various unsupported functionalities or practices on these clusters. These changes include warnings about direct Spark log level setting, accessing the Spark Driver JVM or its logger, using `sc`, and employing RDD APIs. This revision enhances user experience by providing more accurate and descriptive error messages, enabling them to better understand and address the issues in their code. The functionality of the linter remains unchanged.
* Updated sqlglot requirement from <25.8,>=25.5.0 to >=25.5.0,<25.9 ([#2279](#2279)). In this update, we have updated the required version range of the `sqlglot` dependency in the 'pyproject.toml' file from 'sqlglot>=25.5.0,<25.8' to 'sqlglot>=25.5.0,<25.9'. This change allows the project to utilize any version of `sqlglot` that is greater than or equal to 25.5.0 and less than 25.9, including the latest version. The update also includes a changelog for the updated version range, sourced from 'sqlglot's official changelog. This changelog includes various bug fixes and new features for several dialects such as BigQuery, DuckDB, and tSQL. Additionally, the parser has undergone some refactors and improvements. The commits section lists the individual commits included in this update.

Dependency updates:

 * Updated sqlglot requirement from <25.8,>=25.5.0 to >=25.5.0,<25.9 ([#2279](#2279)).

* Added troubleshooting guide for self-signed SSL cert related error
([#2346](#2346)). In this
release, we have added a troubleshooting guide to the README file to
address a specific error that may occur when connecting from a local
machine to a Databricks Account and Workspace using a web proxy and
self-signed SSL certificate. This error, SSLCertVerificationError, can
prevent UCX from connecting to the Account and Workspace. To resolve
this issue, users can now set the `REQUESTS_CA_BUNDLE` and
`CURL_CA_BUNDLE` environment variables to force the requests library to
set `verify=False`, and set the `SSL_CERT_DIR` env var pointing to the
proxy CA cert for the urllib3 library. This guide will help users
understand and resolve this error, making it easier to connect to
Databricks Accounts and Workspaces using a web proxy and self-signed SSL
certificate.
* Code Compatibility Dashboard: Fix broken links
([#2347](#2347)). In this
release, we have addressed and resolved two issues in the Code
Compatibility Dashboard of the UCX Migration (Main) project, enhancing
its overall usability. Previously, the Markdown panel contained a broken
link to the workflow due to an incorrect anchor, and the links in the
table widget to the workflow and task definitions did not render
correctly. These problems have been rectified, and the dashboard has
been manually tested and verified in a staging environment.
Additionally, we have updated the `invisibleColumns` section in the SQL
file by changing the `fieldName` attribute to 'name', which will now
display the `workflow_id` as a link. Before and after screenshots have
been provided for visual reference. The corresponding workflow is now
referred to as "Jobs Static Code Analysis Workflow".
* Filter out missing import problems for imports within a try-except
clause with ImportError
([#2332](#2332)). This
release introduces changes to handle missing import problems within a
try-except clause that catches ImportError. A new method,
`_filter_import_problem_in_try_except`, has been added to filter out
import-not-found issues when they occur in such a clause, preventing
unnecessary build failures. The `_register_import` method now returns an
Iterable[DependencyProblem] instead of yielding problems directly.
Supporting classes and methods, including Dependency, DependencyGraph,
and DependencyProblem from the databricks.labs.ucx.source_code.graph
module, as well as FileLoader and PythonCodeAnalyzer from the
databricks.labs.ucx.source_code.notebooks.cells module, have been added.
The ImportSource.extract_from_tree method has been updated to accept a
DependencyProblem object as an argument. Additionally, a new test case
has been included for the scenario where a failing import in a
try-except clause goes unreported. Issue
[#1705](#1705) has been
resolved, and unit tests have been added to ensure proper functionality.
* Fixed `report-account-compatibility` cli command docstring
([#2340](#2340)). In this
release, we have updated the `report-account-compatibility` CLI
command's docstring to accurately reflect its functionality, addressing
a previous issue where it inadvertently duplicated the
`sync-workspace-info` command's description. This command now provides a
clear and concise explanation of its purpose: "Report compatibility of
all workspaces available in the account." Upon execution, it generates a
readiness report for the account, specifically focusing on workspaces
where ucx is installed. This enhancement improves the clarity of the
CLI's functionality for software engineers, enabling them to understand
and effectively utilize the `report-account-compatibility` command.
* Fixed broken table migration workflow links in README
([#2286](#2286)). In this
release, we have made significant improvements to the README file of our
open-source library, including fixing broken links and adding a mermaid
flowchart to demonstrate the table migration workflows. The table
migration workflow has been renamed to the table migration process,
which includes migrating Delta tables, non-Delta tables, external
tables, and views. Two optional workflows have been added for migrating
HiveSerDe tables in place and for migrating external tables using CTAS.
Additionally, the commands related to table migration have been updated,
with the table migration workflow being renamed to the table migration
process. These changes are aimed at providing a more comprehensive
understanding of the table migration process and enhancing the overall
user experience.
* Fixed dashboard queries fail when default catalog is not
`hive_metastore`
([#2278](#2278)). In this
release, we have addressed an issue where dashboard queries fail when
the default catalog is not set to `hive_metastore`. This has been
achieved by modifying the existing `databricks labs ucx install` command
to always include the `hive_metastore` namespace in dashboard queries.
Additionally, the code has been updated to add the `hive_metastore`
namespace to the `DashboardMetadata` object used in creating a dashboard
from SQL queries in a folder, ensuring queries are executed in the
correct database. The commit also includes modifications to the
`test_install.py` unit test file to ensure the installation process
correctly handles specific configurations related to the `ucx` namespace
for managing data storage and retrieval. The changes have been manually
tested and verified on a staging environment.
* Improve group migration error reporting
([#2344](#2344)). This PR
introduces enhancements to the group migration dashboard, focusing on
improved error reporting and a more informative user experience. The
documentation widgets have been fine-tuned, and the failed-migration
widget now provides formatted failure information with a link to the
failed job run. The dashboard will display only failures from the latest
workflow run, complete with logs. A new link to the job list has been
added in the [workflows](/jobs) section of the documentation to assist
users in identifying and troubleshooting issues. Additionally, the SQL
query for retrieving group migration failure information has been
refactored, improving readability and extracting relevant data using
regular expressions. The changes have been tested and verified on the
staging environment, providing clearer and more actionable insights
during group migrations. The PR is related to previous work in
[#2333](#2333) and
[#1914](#1914), with updates
to the UCX Migration (Groups) dashboard, but no new methods have been
added.
* Improve type checking in cli command
([#2335](#2335)). This
release introduces enhanced type checking in the command line interface
(CLI) of our open-source library, specifically in the `lint_local_code`
function of the `cli.py` file. By utilizing a newly developed local code
linter object, the function now performs more rigorous and accurate type
checking for potential issues in the local code. While the functionality
remains consistent, this improvement is expected to prevent similar
occurrences like issue
[#2221](#2221), ensuring
more robust and reliable code. This change underscores our commitment to
delivering a high-quality, efficient, and developer-friendly library.
* Lint dependencies in context
([#2236](#2236)). The
`InheritedContext` class has been introduced to gather code fragments
from parent files or notebooks during linting of child files or
notebooks, addressing issues
[#2155](#2155),
[#2156](#2156), and
[#2221](#2221). This new
feature includes the addition of the `InheritedContext` class, with
methods for building instances from a route of dependencies, appending
other `InheritedContext` instances, and finalizing them for use with
linting. The `DependencyGraph` class has been updated to support the new
functionality, and various classes, methods, and functions for handling
the linter context have been added or updated. Unit, functional, and
integration tests have been added to ensure the correct functioning of
the changes, which improve the linting functionality by allowing it to
consider the broader context of the codebase.
* Make ucx pylsp plugin configurable
([#2280](#2280)). This
commit introduces the ability to configure the ucx pylsp plugin with
cluster information, which can be provided either in a file or by a
client and is managed by the pylsp infrastructure. The Spark Connect
linter is now only applied to UC Shared clusters, as Single-User
clusters run in Spark Classic mode. A new entry point `pylsp_ucx` has
been added to the pylsp configuration file. The changes affect the pylsp
plugin configuration and the application of the Spark Connect linter.
Unit tests and manual testing have been conducted, but integration tests
and verification on a staging environment are not included in this
release.
* New dashboard: group migration, showing groups that failed to migrate
([#2333](#2333)). In this
release, we have developed a new dashboard for monitoring group
migration in the UCX Migration (Groups) workspace. This dashboard
includes a widget displaying messages related to groups that failed to
migrate during the `migrate-groups-experimental` workflow, aiding users
in identifying and addressing migration issues. The group migration
process consists of several steps, including renaming workspace groups,
provisioning account-level groups, and replicating permissions. The
release features new methods for displaying and monitoring
migration-related messages, as well as links to documentation and
workflows for assessing, validating, and removing workspace-level groups
post-migration. The new dashboard is currently not connected to the
existing system, but it has undergone manual testing and verification on
the staging environment. The changes include the addition of a new SQL
query file to implement the logic for fetching group migration failures
and a new Markdown file displaying the Group Migration Failures section.
* Support spaces in run cmd args
([#2330](#2330)). The recent
commit resolves an issue where the system had trouble handling spaces in
command-line arguments when running subprocesses. The previous
implementation only accepted a full command line, which it would split
on spaces, causing problems when the command line contained arguments
with spaces. The new implementation supports argument lists, which are
passed `as is` to `Popen`, allowing for proper handling of command lines
with spaces. This change is incorporated in the `run_command` function
of the `utils.py` file and the `_install_pip` method of the
`PythonLibraryResolver` class. The `shlex.join()` function has been
replaced with direct string formatting for increased flexibility. The
feature is intended for use with the `PythonLibraryResolver` class and
is co-authored by Eric Vergnaud and Andrew Snare. Integration tests have
been enabled to ensure the proper functioning of the updated code.
* Updated error messages for SparkConnect linter
([#2348](#2348)). The
SparkConnect linter's error messages have been updated to improve
clarity and precision. The term `UC Shared clusters` has been replaced
with `Unity Catalog clusters in Shared access mode` throughout the
codebase, affecting messages related to various unsupported
functionalities or practices on these clusters. These changes include
warnings about direct Spark log level setting, accessing the Spark
Driver JVM or its logger, using `sc`, and employing RDD APIs. This
revision enhances user experience by providing more accurate and
descriptive error messages, enabling them to better understand and
address the issues in their code. The functionality of the linter
remains unchanged.
* Updated sqlglot requirement from <25.8,>=25.5.0 to >=25.5.0,<25.9
([#2279](#2279)). In this
update, we have updated the required version range of the `sqlglot`
dependency in the 'pyproject.toml' file from 'sqlglot>=25.5.0,<25.8' to
'sqlglot>=25.5.0,<25.9'. This change allows the project to utilize any
version of `sqlglot` that is greater than or equal to 25.5.0 and less
than 25.9, including the latest version. The update also includes a
changelog for the updated version range, sourced from 'sqlglot's
official changelog. This changelog includes various bug fixes and new
features for several dialects such as BigQuery, DuckDB, and tSQL.
Additionally, the parser has undergone some refactors and improvements.
The commits section lists the individual commits included in this
update.

Dependency updates:

* Updated sqlglot requirement from <25.8,>=25.5.0 to >=25.5.0,<25.9
([#2279](#2279)).