Skip to content

Remove StaticTablesCrawler in favor of created database tracking #1392

Merged
nfx merged 18 commits intomainfrom
fix/static-configs
Apr 16, 2024
Merged

Remove StaticTablesCrawler in favor of created database tracking #1392
nfx merged 18 commits intomainfrom
fix/static-configs

Conversation

@nfx
Copy link
Copy Markdown
Collaborator

@nfx nfx commented Apr 12, 2024

This PR aims at speeding up real integration testing jobs and making them less flaky.

nfx and others added 7 commits April 16, 2024 11:01
@nkvuong nkvuong force-pushed the fix/static-configs branch from 6c5d391 to 2dd6f51 Compare April 16, 2024 10:06
@nkvuong nkvuong marked this pull request as ready for review April 16, 2024 13:40
@nkvuong nkvuong requested review from a team and pritishpai April 16, 2024 13:40
nfx
nfx commented Apr 16, 2024
View reviewed changes
tests/integration/conftest.py
Comment on lines +336 to +359
def with_dummy_grants_and_tacls(self):
self.sql_backend.save_table(
f"{self.inventory_database}.groups",
[
MigratedGroup(
"group_id",
"test_group_ws",
"test_group_ac",
"tmp",
)
],
MigratedGroup,
)
self.sql_backend.save_table(
f"{self.inventory_database}.grants",
[
Grant(
"test_user",
"SELECT",
database="test_database",
table="test_table",
)
],
Grant,
Copy link
Copy Markdown
Collaborator Author

@nfx nfx Apr 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nkvuong it's not clear what with_dummy_grants_and_tacls is doing

tests/integration/conftest.py
include_databases=self.created_databases,
)

def save_tables(self):
Copy link
Copy Markdown
Collaborator Author

@nfx nfx Apr 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's clear why do we need save_mount, but why do we need save_tables?..

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 16, 2024 •
edited
Loading

❌ 136/140 passed, 2 flaky, 4 failed, 20 skipped, 1h5m6s total

❌ test_running_real_validate_groups_permissions_job: [gw9] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python (0s) 
[gw9] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
file /home/runner/work/ucx/ucx/tests/integration/test_runtime.py, line 61
  @retried(on=[NotFound, InvalidParameterValue], timeout=timedelta(minutes=5))
  def test_running_real_validate_groups_permissions_job(installation_ctx, make_query, make_query_permissions):
E       fixture 'installation_ctx' not found
>       available fixtures: acc, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, class_mocker, cov, debug_env, debug_env_name, doctest_namespace, env_or_skip, fresh_wheel_file, inventory_schema, make_acc_group, make_alert_permissions, make_authorization_permissions, make_catalog, make_cluster, make_cluster_permissions, make_cluster_policy, make_cluster_policy_permissions, make_dashboard_permissions, make_dbfs_data_copy, make_directory, make_directory_permissions, make_experiment, make_experiment_permissions, make_feature_table, make_feature_table_permissions, make_group, make_group_pair, make_instance_pool, make_instance_pool_permissions, make_job, make_job_permissions, make_model, make_notebook, make_notebook_permissions, make_pipeline, make_pipeline_permissions, make_query, make_query_permissions, make_random, make_registered_model_permissions, make_repo, make_repo_permissions, make_schema, make_secret_scope, make_secret_scope_acl, make_serving_endpoint, make_serving_endpoint_permissions, make_storage_credential, make_table, make_ucx_group, make_udf, make_user, make_warehouse, make_warehouse_permissions, make_workspace_file_path_permissions, make_workspace_file_permissions, migrated_group, mocker, module_mocker, monkeypatch, no_cover, package_mocker, product_info, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, runtime_ctx, session_mocker, sql_backend, sql_exec, sql_fetch_all, testrun_uid, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, worker_id, ws, wsfs_wheel
>       use 'pytest --fixtures [testpath]' for help on them.

/home/runner/work/ucx/ucx/tests/integration/test_runtime.py:61
[gw9] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
❌ test_running_real_validate_groups_permissions_job_fails: [gw9] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python (3ms) 
[gw9] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
file /home/runner/work/ucx/ucx/tests/integration/test_runtime.py, line 80
  @retried(on=[NotFound], timeout=timedelta(minutes=5))
  def test_running_real_validate_groups_permissions_job_fails(
E       fixture 'installation_ctx' not found
>       available fixtures: acc, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, class_mocker, cov, debug_env, debug_env_name, doctest_namespace, env_or_skip, fresh_wheel_file, inventory_schema, make_acc_group, make_alert_permissions, make_authorization_permissions, make_catalog, make_cluster, make_cluster_permissions, make_cluster_policy, make_cluster_policy_permissions, make_dashboard_permissions, make_dbfs_data_copy, make_directory, make_directory_permissions, make_experiment, make_experiment_permissions, make_feature_table, make_feature_table_permissions, make_group, make_group_pair, make_instance_pool, make_instance_pool_permissions, make_job, make_job_permissions, make_model, make_notebook, make_notebook_permissions, make_pipeline, make_pipeline_permissions, make_query, make_query_permissions, make_random, make_registered_model_permissions, make_repo, make_repo_permissions, make_schema, make_secret_scope, make_secret_scope_acl, make_serving_endpoint, make_serving_endpoint_permissions, make_storage_credential, make_table, make_ucx_group, make_udf, make_user, make_warehouse, make_warehouse_permissions, make_workspace_file_path_permissions, make_workspace_file_permissions, migrated_group, mocker, module_mocker, monkeypatch, no_cover, package_mocker, product_info, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, runtime_ctx, session_mocker, sql_backend, sql_exec, sql_fetch_all, testrun_uid, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, worker_id, ws, wsfs_wheel
>       use 'pytest --fixtures [testpath]' for help on them.

/home/runner/work/ucx/ucx/tests/integration/test_runtime.py:80
[gw9] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
❌ test_running_real_assessment_job: [gw3] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python (3ms) 
[gw3] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
file /home/runner/work/ucx/ucx/tests/integration/test_runtime.py, line 18
  @retried(on=[NotFound, InvalidParameterValue], timeout=timedelta(minutes=5))
  def test_running_real_assessment_job(ws, installation_ctx, make_cluster_policy, make_cluster_policy_permissions):
E       fixture 'installation_ctx' not found
>       available fixtures: acc, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, class_mocker, cov, debug_env, debug_env_name, doctest_namespace, env_or_skip, fresh_wheel_file, inventory_schema, make_acc_group, make_alert_permissions, make_authorization_permissions, make_catalog, make_cluster, make_cluster_permissions, make_cluster_policy, make_cluster_policy_permissions, make_dashboard_permissions, make_dbfs_data_copy, make_directory, make_directory_permissions, make_experiment, make_experiment_permissions, make_feature_table, make_feature_table_permissions, make_group, make_group_pair, make_instance_pool, make_instance_pool_permissions, make_job, make_job_permissions, make_model, make_notebook, make_notebook_permissions, make_pipeline, make_pipeline_permissions, make_query, make_query_permissions, make_random, make_registered_model_permissions, make_repo, make_repo_permissions, make_schema, make_secret_scope, make_secret_scope_acl, make_serving_endpoint, make_serving_endpoint_permissions, make_storage_credential, make_table, make_ucx_group, make_udf, make_user, make_warehouse, make_warehouse_permissions, make_workspace_file_path_permissions, make_workspace_file_permissions, migrated_group, mocker, module_mocker, monkeypatch, no_cover, package_mocker, product_info, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, runtime_ctx, session_mocker, sql_backend, sql_exec, sql_fetch_all, testrun_uid, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, worker_id, ws, wsfs_wheel
>       use 'pytest --fixtures [testpath]' for help on them.

/home/runner/work/ucx/ucx/tests/integration/test_runtime.py:18
[gw3] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
❌ test_running_real_migrate_groups_job: [gw3] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python (2ms) 
[gw3] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
file /home/runner/work/ucx/ucx/tests/integration/test_runtime.py, line 38
  @retried(on=[NotFound, InvalidParameterValue], timeout=timedelta(minutes=5))
  def test_running_real_migrate_groups_job(ws, installation_ctx, make_cluster_policy, make_cluster_policy_permissions):
E       fixture 'installation_ctx' not found
>       available fixtures: acc, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, class_mocker, cov, debug_env, debug_env_name, doctest_namespace, env_or_skip, fresh_wheel_file, inventory_schema, make_acc_group, make_alert_permissions, make_authorization_permissions, make_catalog, make_cluster, make_cluster_permissions, make_cluster_policy, make_cluster_policy_permissions, make_dashboard_permissions, make_dbfs_data_copy, make_directory, make_directory_permissions, make_experiment, make_experiment_permissions, make_feature_table, make_feature_table_permissions, make_group, make_group_pair, make_instance_pool, make_instance_pool_permissions, make_job, make_job_permissions, make_model, make_notebook, make_notebook_permissions, make_pipeline, make_pipeline_permissions, make_query, make_query_permissions, make_random, make_registered_model_permissions, make_repo, make_repo_permissions, make_schema, make_secret_scope, make_secret_scope_acl, make_serving_endpoint, make_serving_endpoint_permissions, make_storage_credential, make_table, make_ucx_group, make_udf, make_user, make_warehouse, make_warehouse_permissions, make_workspace_file_path_permissions, make_workspace_file_permissions, migrated_group, mocker, module_mocker, monkeypatch, no_cover, package_mocker, product_info, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, runtime_ctx, session_mocker, sql_backend, sql_exec, sql_fetch_all, testrun_uid, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, worker_id, ws, wsfs_wheel
>       use 'pytest --fixtures [testpath]' for help on them.

/home/runner/work/ucx/ucx/tests/integration/test_runtime.py:38
[gw3] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python

Flaky tests:

  • 🤪 test_migrate_managed_tables_with_principal_acl_CLOUD_ENV (2m52.714s)
  • 🤪 test_permission_for_files_anonymous_func (52.556s)

Running from acceptance #2359

@nfx nfx merged commit 7204c8e into main Apr 16, 2024
@nfx nfx deleted the fix/static-configs branch April 16, 2024 18:26
ericvergnaud added a commit to ericvergnaud/ucx that referenced this pull request Apr 18, 2024
@ericvergnaud
Merge branch 'main' into specialized-dependency-loaders
43c2ccb 
* main:
  Give all access connectors `Storage Blob Data Contributor` role (databrickslabs#1425)
  Addressed a bug with AWS UC Role Update. Adding unit tests. (databrickslabs#1429)
  Added integration tests with external HMS & Glue (databrickslabs#1408)
  Modified update existing role to amend the AssumeRole statement rather than rewriting it. (databrickslabs#1423)
  Extend service principal migration with option to create access connectors with managed identity for each storage account (databrickslabs#1417)
  A notebook linter to detect DBFS references within notebook cells (databrickslabs#1393)
  Remove `StaticTablesCrawler` in favor of created database tracking  (databrickslabs#1392)
  Cleaned up integration test suite (databrickslabs#1422)

# Conflicts:
#	tests/unit/source_code/test_notebook_linter.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pritishpai pritishpai Awaiting requested review from pritishpai

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@nfx @nkvuong