* Added AWS IAM roles support to `databricks labs ucx migrate-credentials` command ([#973](#973)). This commit adds AWS Identity and Access Management (IAM) roles support to the `databricks labs ucx migrate-credentials` command, resolving issue [#862](#862) and being related to pull request [#874](#874). It includes the addition of a `load` function to `AWSResourcePermissions` to return identified instance profiles and the creation of an `IamRoleMigration` class under `aws/credentials.py` to migrate identified AWS instance profiles. Additionally, user documentation and a new CLI command `databricks labs ucx migrate-credentials` have been added, and the changes have been thoroughly tested with manual, unit, and integration tests. The functionality additions include new methods such as `add_uc_role_policy` and `update_uc_trust_role`, among others, designed to facilitate the migration process for AWS IAM roles.
* Added `create-catalogs-schemas` command to prepare destination catalogs and schemas before table migration ([#1028](#1028)). The Databricks Labs Unity Catalog (UCX) tool has been updated with a new `create-catalogs-schemas` command to facilitate the creation of destination catalogs and schemas prior to table migration. This command should be executed after the `create-table-mapping` command and is designed to prepare the workspace for migrating tables to UC. Additionally, a new `CatalogSchema` class has been added to the `hive_metastore` package to manage the creation of catalogs and schemas in the Hive metastore. This new functionality simplifies the process of preparing the destination Hive metastore for table migration, reducing the likelihood of user errors and ensuring that the metastore is properly configured. Unit tests have been added to the `tests/unit/hive_metastore` directory to verify the behavior of the `CatalogSchema` class and the new `create-catalogs-schemas` command. This command is intended for use in contexts where GCP is not supported.
* Added automated upgrade option to set up cluster policy ([#1024](#1024)). This commit introduces an automated upgrade option for setting up a cluster policy for older versions of UCX, separating the cluster creation policy from install.py to installer.policy.py and adding an upgrade script for older UCX versions. A new class, `ClusterPolicyInstaller`, is added to the `policy.py` file in the `installer` package to manage the creation and update of a Databricks cluster policy for Unity Catalog Migration. This class handles creating a new cluster policy with specific configurations, extracting external Hive Metastore configurations, and updating job policies. Additionally, the commit includes refactoring, removal of library references, and a new script, v0.15.0_added_cluster_policy.py, which contains the upgrade function. The changes are tested through manual and automated testing with unit tests and integration tests. This feature is intended for software engineers working with the project.
* Added crawling for init scripts on local files to assessment workflow ([#960](#960)). This commit introduces the ability to crawl init scripts stored on local files and S3 as part of the assessment workflow, resolving issue [#9](#9)
* Added database filter for the `assessment` workflow ([#989](#989)). In this release, we have added a new configuration option, `include_databases`, to the assessment workflow which allows users to specify a list of databases to include for migration, rather than crawling all the databases in the Hive Metastore. This feature is implemented in the `TablesCrawler`, `UdfsCrawler`, `GrantsCrawler` classes and the associated functions such as `_all_databases`, `getIncludeDatabases`, `_select_databases`. These changes aim to improve efficiency and reduce unnecessary crawling, and are accompanied by modifications to existing functionality, as well as the addition of unit and integration tests. The changes have been manually tested and verified on a staging environment.
* Estimate migration effort based on assessment database ([#1008](#1008)). In this release, a new functionality has been added to estimate the migration effort for each asset in the assessment database. The estimation is presented in days and is displayed on a new estimates dashboard with a summary widget for a global estimate per object type, along with assumptions and scope for each object type. A new `query` parameter has been added to the `SimpleQuery` class to support this feature. Additional changes include the update of the `_install_viz` and `_install_query` methods, the inclusion of the `data_source_id` in the query metadata, and the addition of tests to ensure the proper functioning of the new feature. A new fixture, `mock_installation_with_jobs`, has been added to support testing of the assessment estimates dashboard.
* Explicitly write to `hive_metastore` from `crawl_tables` task ([#1021](#1021)). In this release, we have improved the clarity and specificity of our handling of the `hive_metastore` in the `crawl_tables` task. Previously, the `df.write.saveAsTable` method was used without explicitly specifying the `hive_metastore` database, which could result in ambiguity. To address this issue, we have updated the `saveAsTable` method to include the `hive_metastore` database, ensuring that tables are written to the correct location in the Hive metastore. These changes are confined to the `src/databricks/labs/ucx/hive_metastore/tables.scala` file and affect the `crawl_tables` task. While no new methods have been added, the existing `saveAsTable` method has been modified to enhance the accuracy and predictability of our interaction with the Hive metastore.
* Improved documentation for `databricks labs ucx move` command ([#1025](#1025)). The `databricks labs ucx move` command has been updated with new improvements to its documentation, providing enhanced clarity and ease of use for developers and administrators. This command facilitates the movement of UC tables/table(s) from one schema to another, either in the same or different catalog, during the table upgrade process. A significant enhancement is the preservation of the source table's permissions when moving to a new schema or catalog, maintaining the original table's access controls, simplifying the management of table permissions, and streamlining the migration process. These improvements aim to facilitate a more efficient table migration experience, ensuring that developers and administrators can effectively manage their UC tables while maintaining the desired level of access control and security.
* Updated databricks-sdk requirement from ~=0.20.0 to ~=0.21.0 ([#1030](#1030)). In this update, the `databricks-sdk` package requirement has been updated to version `~=0.21.0` from `~=0.20.0`. This new version addresses several bugs and provides enhancements, including the fix for the `get_workspace_client` method in GCP, the use of the `all-apis` scope with the external browser, and an attempt to initialize all Databricks globals. Moreover, the API's settings nesting approach has changed, which may cause compatibility issues with previous versions. Several new services and dataclasses have been added to the API, and documentation and examples have been updated accordingly. There are no updates to the `databricks-labs-blueprint` and `PyYAML` dependencies in this commit.

* Added AWS IAM roles support to `databricks labs ucx
migrate-credentials` command
([#973](#973)). This commit
adds AWS Identity and Access Management (IAM) roles support to the
`databricks labs ucx migrate-credentials` command, resolving issue
[#862](#862) and being
related to pull request
[#874](#874). It includes
the addition of a `load` function to `AWSResourcePermissions` to return
identified instance profiles and the creation of an `IamRoleMigration`
class under `aws/credentials.py` to migrate identified AWS instance
profiles. Additionally, user documentation and a new CLI command
`databricks labs ucx migrate-credentials` have been added, and the
changes have been thoroughly tested with manual, unit, and integration
tests. The functionality additions include new methods such as
`add_uc_role_policy` and `update_uc_trust_role`, among others, designed
to facilitate the migration process for AWS IAM roles.
* Added `create-catalogs-schemas` command to prepare destination
catalogs and schemas before table migration
([#1028](#1028)). The
Databricks Labs Unity Catalog (UCX) tool has been updated with a new
`create-catalogs-schemas` command to facilitate the creation of
destination catalogs and schemas prior to table migration. This command
should be executed after the `create-table-mapping` command and is
designed to prepare the workspace for migrating tables to UC.
Additionally, a new `CatalogSchema` class has been added to the
`hive_metastore` package to manage the creation of catalogs and schemas
in the Hive metastore. This new functionality simplifies the process of
preparing the destination Hive metastore for table migration, reducing
the likelihood of user errors and ensuring that the metastore is
properly configured. Unit tests have been added to the
`tests/unit/hive_metastore` directory to verify the behavior of the
`CatalogSchema` class and the new `create-catalogs-schemas` command.
This command is intended for use in contexts where GCP is not supported.
* Added automated upgrade option to set up cluster policy
([#1024](#1024)). This
commit introduces an automated upgrade option for setting up a cluster
policy for older versions of UCX, separating the cluster creation policy
from install.py to installer.policy.py and adding an upgrade script for
older UCX versions. A new class, `ClusterPolicyInstaller`, is added to
the `policy.py` file in the `installer` package to manage the creation
and update of a Databricks cluster policy for Unity Catalog Migration.
This class handles creating a new cluster policy with specific
configurations, extracting external Hive Metastore configurations, and
updating job policies. Additionally, the commit includes refactoring,
removal of library references, and a new script,
v0.15.0_added_cluster_policy.py, which contains the upgrade function.
The changes are tested through manual and automated testing with unit
tests and integration tests. This feature is intended for software
engineers working with the project.
* Added crawling for init scripts on local files to assessment workflow
([#960](#960)). This commit
introduces the ability to crawl init scripts stored on local files and
S3 as part of the assessment workflow, resolving issue
[#9](#9)
* Added database filter for the `assessment` workflow
([#989](#989)). In this
release, we have added a new configuration option, `include_databases`,
to the assessment workflow which allows users to specify a list of
databases to include for migration, rather than crawling all the
databases in the Hive Metastore. This feature is implemented in the
`TablesCrawler`, `UdfsCrawler`, `GrantsCrawler` classes and the
associated functions such as `_all_databases`, `getIncludeDatabases`,
`_select_databases`. These changes aim to improve efficiency and reduce
unnecessary crawling, and are accompanied by modifications to existing
functionality, as well as the addition of unit and integration tests.
The changes have been manually tested and verified on a staging
environment.
* Estimate migration effort based on assessment database
([#1008](#1008)). In this
release, a new functionality has been added to estimate the migration
effort for each asset in the assessment database. The estimation is
presented in days and is displayed on a new estimates dashboard with a
summary widget for a global estimate per object type, along with
assumptions and scope for each object type. A new `query` parameter has
been added to the `SimpleQuery` class to support this feature.
Additional changes include the update of the `_install_viz` and
`_install_query` methods, the inclusion of the `data_source_id` in the
query metadata, and the addition of tests to ensure the proper
functioning of the new feature. A new fixture,
`mock_installation_with_jobs`, has been added to support testing of the
assessment estimates dashboard.
* Explicitly write to `hive_metastore` from `crawl_tables` task
([#1021](#1021)). In this
release, we have improved the clarity and specificity of our handling of
the `hive_metastore` in the `crawl_tables` task. Previously, the
`df.write.saveAsTable` method was used without explicitly specifying the
`hive_metastore` database, which could result in ambiguity. To address
this issue, we have updated the `saveAsTable` method to include the
`hive_metastore` database, ensuring that tables are written to the
correct location in the Hive metastore. These changes are confined to
the `src/databricks/labs/ucx/hive_metastore/tables.scala` file and
affect the `crawl_tables` task. While no new methods have been added,
the existing `saveAsTable` method has been modified to enhance the
accuracy and predictability of our interaction with the Hive metastore.
* Improved documentation for `databricks labs ucx move` command
([#1025](#1025)). The
`databricks labs ucx move` command has been updated with new
improvements to its documentation, providing enhanced clarity and ease
of use for developers and administrators. This command facilitates the
movement of UC tables/table(s) from one schema to another, either in the
same or different catalog, during the table upgrade process. A
significant enhancement is the preservation of the source table's
permissions when moving to a new schema or catalog, maintaining the
original table's access controls, simplifying the management of table
permissions, and streamlining the migration process. These improvements
aim to facilitate a more efficient table migration experience, ensuring
that developers and administrators can effectively manage their UC
tables while maintaining the desired level of access control and
security.
* Updated databricks-sdk requirement from ~=0.20.0 to ~=0.21.0
([#1030](#1030)). In this
update, the `databricks-sdk` package requirement has been updated to
version `~=0.21.0` from `~=0.20.0`. This new version addresses several
bugs and provides enhancements, including the fix for the
`get_workspace_client` method in GCP, the use of the `all-apis` scope
with the external browser, and an attempt to initialize all Databricks
globals. Moreover, the API's settings nesting approach has changed,
which may cause compatibility issues with previous versions. Several new
services and dataclasses have been added to the API, and documentation
and examples have been updated accordingly. There are no updates to the
`databricks-labs-blueprint` and `PyYAML` dependencies in this commit.

…#960)

## Changes
Scanning Init Script for local file and S3

### Linked issues
#954 

Resolves #954 

### Functionality 

- [ ] added relevant user documentation
- [ ] added new CLI command
- [ ] modified existing command: `databricks labs ucx ...`
- [ ] added a new workflow
- [ ] modified existing workflow: `...`
- [ ] added a new table
- [ ] modified existing table: `...`

### Tests
<!-- How is this tested? Please see the checklist below and also
describe any other relevant tests -->

- [ ] manually tested
- [x] added unit tests
- [ ] added integration tests
- [ ] verified on staging environment (screenshot attached)

* Added AWS IAM roles support to `databricks labs ucx
migrate-credentials` command
([#973](#973)). This commit
adds AWS Identity and Access Management (IAM) roles support to the
`databricks labs ucx migrate-credentials` command, resolving issue
[#862](#862) and being
related to pull request
[#874](#874). It includes
the addition of a `load` function to `AWSResourcePermissions` to return
identified instance profiles and the creation of an `IamRoleMigration`
class under `aws/credentials.py` to migrate identified AWS instance
profiles. Additionally, user documentation and a new CLI command
`databricks labs ucx migrate-credentials` have been added, and the
changes have been thoroughly tested with manual, unit, and integration
tests. The functionality additions include new methods such as
`add_uc_role_policy` and `update_uc_trust_role`, among others, designed
to facilitate the migration process for AWS IAM roles.
* Added `create-catalogs-schemas` command to prepare destination
catalogs and schemas before table migration
([#1028](#1028)). The
Databricks Labs Unity Catalog (UCX) tool has been updated with a new
`create-catalogs-schemas` command to facilitate the creation of
destination catalogs and schemas prior to table migration. This command
should be executed after the `create-table-mapping` command and is
designed to prepare the workspace for migrating tables to UC.
Additionally, a new `CatalogSchema` class has been added to the
`hive_metastore` package to manage the creation of catalogs and schemas
in the Hive metastore. This new functionality simplifies the process of
preparing the destination Hive metastore for table migration, reducing
the likelihood of user errors and ensuring that the metastore is
properly configured. Unit tests have been added to the
`tests/unit/hive_metastore` directory to verify the behavior of the
`CatalogSchema` class and the new `create-catalogs-schemas` command.
This command is intended for use in contexts where GCP is not supported.
* Added automated upgrade option to set up cluster policy
([#1024](#1024)). This
commit introduces an automated upgrade option for setting up a cluster
policy for older versions of UCX, separating the cluster creation policy
from install.py to installer.policy.py and adding an upgrade script for
older UCX versions. A new class, `ClusterPolicyInstaller`, is added to
the `policy.py` file in the `installer` package to manage the creation
and update of a Databricks cluster policy for Unity Catalog Migration.
This class handles creating a new cluster policy with specific
configurations, extracting external Hive Metastore configurations, and
updating job policies. Additionally, the commit includes refactoring,
removal of library references, and a new script,
v0.15.0_added_cluster_policy.py, which contains the upgrade function.
The changes are tested through manual and automated testing with unit
tests and integration tests. This feature is intended for software
engineers working with the project.
* Added crawling for init scripts on local files to assessment workflow
([#960](#960)). This commit
introduces the ability to crawl init scripts stored on local files and
S3 as part of the assessment workflow, resolving issue
[#9](#9)
* Added database filter for the `assessment` workflow
([#989](#989)). In this
release, we have added a new configuration option, `include_databases`,
to the assessment workflow which allows users to specify a list of
databases to include for migration, rather than crawling all the
databases in the Hive Metastore. This feature is implemented in the
`TablesCrawler`, `UdfsCrawler`, `GrantsCrawler` classes and the
associated functions such as `_all_databases`, `getIncludeDatabases`,
`_select_databases`. These changes aim to improve efficiency and reduce
unnecessary crawling, and are accompanied by modifications to existing
functionality, as well as the addition of unit and integration tests.
The changes have been manually tested and verified on a staging
environment.
* Estimate migration effort based on assessment database
([#1008](#1008)). In this
release, a new functionality has been added to estimate the migration
effort for each asset in the assessment database. The estimation is
presented in days and is displayed on a new estimates dashboard with a
summary widget for a global estimate per object type, along with
assumptions and scope for each object type. A new `query` parameter has
been added to the `SimpleQuery` class to support this feature.
Additional changes include the update of the `_install_viz` and
`_install_query` methods, the inclusion of the `data_source_id` in the
query metadata, and the addition of tests to ensure the proper
functioning of the new feature. A new fixture,
`mock_installation_with_jobs`, has been added to support testing of the
assessment estimates dashboard.
* Explicitly write to `hive_metastore` from `crawl_tables` task
([#1021](#1021)). In this
release, we have improved the clarity and specificity of our handling of
the `hive_metastore` in the `crawl_tables` task. Previously, the
`df.write.saveAsTable` method was used without explicitly specifying the
`hive_metastore` database, which could result in ambiguity. To address
this issue, we have updated the `saveAsTable` method to include the
`hive_metastore` database, ensuring that tables are written to the
correct location in the Hive metastore. These changes are confined to
the `src/databricks/labs/ucx/hive_metastore/tables.scala` file and
affect the `crawl_tables` task. While no new methods have been added,
the existing `saveAsTable` method has been modified to enhance the
accuracy and predictability of our interaction with the Hive metastore.
* Improved documentation for `databricks labs ucx move` command
([#1025](#1025)). The
`databricks labs ucx move` command has been updated with new
improvements to its documentation, providing enhanced clarity and ease
of use for developers and administrators. This command facilitates the
movement of UC tables/table(s) from one schema to another, either in the
same or different catalog, during the table upgrade process. A
significant enhancement is the preservation of the source table's
permissions when moving to a new schema or catalog, maintaining the
original table's access controls, simplifying the management of table
permissions, and streamlining the migration process. These improvements
aim to facilitate a more efficient table migration experience, ensuring
that developers and administrators can effectively manage their UC
tables while maintaining the desired level of access control and
security.
* Updated databricks-sdk requirement from ~=0.20.0 to ~=0.21.0
([#1030](#1030)). In this
update, the `databricks-sdk` package requirement has been updated to
version `~=0.21.0` from `~=0.20.0`. This new version addresses several
bugs and provides enhancements, including the fix for the
`get_workspace_client` method in GCP, the use of the `all-apis` scope
with the external browser, and an attempt to initialize all Databricks
globals. Moreover, the API's settings nesting approach has changed,
which may cause compatibility issues with previous versions. Several new
services and dataclasses have been added to the API, and documentation
and examples have been updated accordingly. There are no updates to the
`databricks-labs-blueprint` and `PyYAML` dependencies in this commit.