Update existing and add new SQL queries to the assessment step dashboard#269
Merged
Update existing and add new SQL queries to the assessment step dashboard#269
Conversation
Contributor
Author
|
|
Codecov Report
@@ Coverage Diff @@
## main #269 +/- ##
==========================================
+ Coverage 83.37% 83.40% +0.02%
==========================================
Files 29 29
Lines 1979 1982 +3
Branches 337 337
==========================================
+ Hits 1650 1653 +3
Misses 261 261
Partials 68 68
|
Contributor
Author
|
|
nfx
reviewed
Sep 22, 2023
Collaborator
nfx
left a comment
nfx
left a comment
There was a problem hiding this comment.
You're also adding more queries?
src/databricks/labs/ucx/assessment/queries/database_summary.sql
Outdated
Show resolved
Hide resolved
src/databricks/labs/ucx/assessment/queries/database_summary.sql
Outdated
Show resolved
Hide resolved
src/databricks/labs/ucx/assessment/queries/database_summary.sql
Outdated
Show resolved
Hide resolved
FastLee
added
enhancement
FastLee
approved these changes
Sep 22, 2023
nfx
requested changes
Sep 22, 2023
src/databricks/labs/ucx/assessment/queries/database_summary.sql
Outdated
Show resolved
Hide resolved
src/databricks/labs/ucx/assessment/queries/database_summary.sql
Outdated
Show resolved
Hide resolved
|
|
||
|
|
||
| def test_creating_widgets(ws: WorkspaceClient): | ||
| def test_creating_widgets(ws: WorkspaceClient, make_warehouse, make_schema): |
Collaborator
There was a problem hiding this comment.
You can use the preprovisioned warehouse through environment variable, no need to create new warehouse to test a dashboard.
| creator, | ||
| CASE WHEN success=1 THEN "Compatible" ELSE "Incompatible" END AS compatible, | ||
| failures | ||
| FROM $inventory.clusters No newline at end of file |
Collaborator
There was a problem hiding this comment.
Add NOT STARTSWITH(cluster_name, "job-") and also change the etl code
Contributor
Author
There was a problem hiding this comment.
As a WHERE clause? And what ETL code?
Contributor
Author
There was a problem hiding this comment.
I am confused where job- is used... looking at our E2 env:
Collaborator
|
@larsgeorge-db please also make the PR title very verbose - release scripts create a changelog line from the PR title. |
larsgeorge-db
changed the title
larsgeorge-db
changed the title
nfx
reviewed
Sep 24, 2023
| CASE | ||
| WHEN STARTSWITH(location, "/dbfs/") | ||
| AND NOT STARTSWITH(location, "/dbfs/mnt") THEN "DBFS ROOT" | ||
| WHEN STARTSWITH(location, "/dbfs/") |
Collaborator
There was a problem hiding this comment.
First startswith is redundant
Contributor
Author
There was a problem hiding this comment.
See comment below.
Collaborator
There was a problem hiding this comment.
STARTSWITH(location, "/dbfs/") AND STARTSWITH(location, "/dbfs/mnt") is redundant by boolean logic.
src/databricks/labs/ucx/assessment/queries/database_summary.sql
Outdated
Show resolved
Hide resolved
src/databricks/labs/ucx/assessment/queries/database_summary.sql
Outdated
Show resolved
Hide resolved
src/databricks/labs/ucx/assessment/queries/database_summary.sql
Outdated
Show resolved
Hide resolved
Contributor
Author
|
Fixes #280
|
larsgeorge-db
force-pushed
the
feature/issue_268_improve_assessment_dashboard
branch
from
9240bac to
9488221
Compare
nfx
reviewed
Sep 25, 2023
Collaborator
nfx
left a comment
nfx
left a comment
There was a problem hiding this comment.
last few comments remaining
| CASE | ||
| WHEN STARTSWITH(location, "/dbfs/") | ||
| AND NOT STARTSWITH(location, "/dbfs/mnt") THEN "DBFS ROOT" | ||
| WHEN STARTSWITH(location, "/dbfs/") |
Collaborator
There was a problem hiding this comment.
STARTSWITH(location, "/dbfs/") AND STARTSWITH(location, "/dbfs/mnt") is redundant by boolean logic.
| creator, | ||
| CASE WHEN success=1 THEN "Compatible" ELSE "Incompatible" END AS compatible, | ||
| failures | ||
| FROM $inventory.clusters No newline at end of file |
src/databricks/labs/ucx/assessment/queries/database_summary.sql
Outdated
Show resolved
Hide resolved
Contributor
Author
|
|
nfx
approved these changes
Sep 25, 2023
nfx
added a commit
that referenced
this pull request
Sep 29, 2023
# Version changelog ## 0.2.0 * Added retrieving for all account-level groups with matching names to workspace-level groups in case no explicit configuration ([#277](#277)). * Added crawler for Azure Service principals used for direct storage access ([#305](#305)). * Added more SQL queries to the assessment step dashboard ([#269](#269)). * Added filtering out for job clusters in the clusters crawler ([#298](#298)). * Added recording errors from `crawl_tables` step in `$inventory.table_failures` table and display counter on the dashboard ([#300](#300)). * Added comprehensive introduction user manual ([#273](#273)). * Added interactive tutorial for local group migration readme ([#291](#291)). * Added tutorial links to the landing page of documentation ([#290](#290)). * Added (internal) support for account-level configuration and multi-cloud workspace list ([#264](#264)). * Improved order of tasks in the README notebook ([#286](#286)). * Improved installation script to run in a Windows Git Bash terminal ([#282](#282)). * Improved installation script by setting log level to uppercase by default ([#271](#271)). * Improved installation finish messages within installer script ([#267](#267)). * Improved automation for `MANAGED` table migration and continued building tables migration component ([#295](#295)). * Fixed debug notebook code with refactored package structure ([#250](#250)) ([#265](#265)). * Fixed replacement of custom configured database to replicate in the report for external locations ([#296](#296)). * Removed redundant `notebooks` top-level folder ([#263](#263)). * Split checking for test failures and linting errors into independent GitHub Actions checks ([#287](#287)). * Verify query metadata for assessment dashboards during unit tests ([#294](#294)).
Merged
nfx
added a commit
that referenced
this pull request
Sep 29, 2023
# Version changelog ## 0.2.0 * Added retrieving for all account-level groups with matching names to workspace-level groups in case no explicit configuration ([#277](#277)). * Added crawler for Azure Service principals used for direct storage access ([#305](#305)). * Added more SQL queries to the assessment step dashboard ([#269](#269)). * Added filtering out for job clusters in the clusters crawler ([#298](#298)). * Added recording errors from `crawl_tables` step in `$inventory.table_failures` table and display counter on the dashboard ([#300](#300)). * Added comprehensive introduction user manual ([#273](#273)). * Added interactive tutorial for local group migration readme ([#291](#291)). * Added tutorial links to the landing page of documentation ([#290](#290)). * Added (internal) support for account-level configuration and multi-cloud workspace list ([#264](#264)). * Improved order of tasks in the README notebook ([#286](#286)). * Improved installation script to run in a Windows Git Bash terminal ([#282](#282)). * Improved installation script by setting log level to uppercase by default ([#271](#271)). * Improved installation finish messages within installer script ([#267](#267)). * Improved automation for `MANAGED` table migration and continued building tables migration component ([#295](#295)). * Fixed debug notebook code with refactored package structure ([#250](#250)) ([#265](#265)). * Fixed replacement of custom configured database to replicate in the report for external locations ([#296](#296)). * Removed redundant `notebooks` top-level folder ([#263](#263)). * Split checking for test failures and linting errors into independent GitHub Actions checks ([#287](#287)). * Verify query metadata for assessment dashboards during unit tests ([#294](#294)).
FastLee
pushed a commit
that referenced
this pull request
Sep 29, 2023
# Version changelog ## 0.2.0 * Added retrieving for all account-level groups with matching names to workspace-level groups in case no explicit configuration ([#277](#277)). * Added crawler for Azure Service principals used for direct storage access ([#305](#305)). * Added more SQL queries to the assessment step dashboard ([#269](#269)). * Added filtering out for job clusters in the clusters crawler ([#298](#298)). * Added recording errors from `crawl_tables` step in `$inventory.table_failures` table and display counter on the dashboard ([#300](#300)). * Added comprehensive introduction user manual ([#273](#273)). * Added interactive tutorial for local group migration readme ([#291](#291)). * Added tutorial links to the landing page of documentation ([#290](#290)). * Added (internal) support for account-level configuration and multi-cloud workspace list ([#264](#264)). * Improved order of tasks in the README notebook ([#286](#286)). * Improved installation script to run in a Windows Git Bash terminal ([#282](#282)). * Improved installation script by setting log level to uppercase by default ([#271](#271)). * Improved installation finish messages within installer script ([#267](#267)). * Improved automation for `MANAGED` table migration and continued building tables migration component ([#295](#295)). * Fixed debug notebook code with refactored package structure ([#250](#250)) ([#265](#265)). * Fixed replacement of custom configured database to replicate in the report for external locations ([#296](#296)). * Removed redundant `notebooks` top-level folder ([#263](#263)). * Split checking for test failures and linting errors into independent GitHub Actions checks ([#287](#287)). * Verify query metadata for assessment dashboards during unit tests ([#294](#294)).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #268