Improved automation for MANAGED table migration and continued building tables migration component#295
Merged
Improved automation for MANAGED table migration and continued building tables migration component#295
MANAGED table migration and continued building tables migration component#295Conversation
Codecov Report
@@ Coverage Diff @@
## main #295 +/- ##
==========================================
+ Coverage 83.27% 83.47% +0.19%
==========================================
Files 30 30
Lines 2146 2184 +38
Branches 366 370 +4
==========================================
+ Hits 1787 1823 +36
- Misses 279 281 +2
Partials 80 80
|
nfx
requested changes
Sep 25, 2023
Collaborator
nfx
left a comment
nfx
left a comment
There was a problem hiding this comment.
Can you create a new workflow in the runtime.py and schedule this as a task? Please also add the integration tests after running the jobs.
See tests/integration/test_installation.py
Contributor
Author
|
What do you mean by creating a task ? Does it need to create all the UC tables by fetching the tables in the assessment + executing all the SQL code returned by |
Collaborator
|
@william-conti yes |
Contributor
Author
|
Then the scope of this ticket is way larger because we need to define the catalog output for the tables / views that needs to be migrated. AFAIK we do not let customer choose where his tables are going to be stored ... We can start by specifying a default catalog as the target, and then the customer can run ALTER TABLES to map these tables to his own data model that he will choose ... |
Collaborator
|
Hm. Can you at least add the integration test to verify that sync has completed successfully? :) other prs would add the task then |
Contributor
Author
|
Won't be able to Integration SYNC with external tables as we need to have access to S3 to create external tables ... |
nfx
requested changes
Sep 26, 2023
william-conti
force-pushed
the
add_sync_command
branch
from
0dd59ce to
459dc71
Compare
nfx
requested changes
Sep 27, 2023
william-conti
force-pushed
the
add_sync_command
branch
from
284d3fb to
c298694
Compare
nfx
requested changes
Sep 27, 2023
nfx
approved these changes
Sep 28, 2023
nfx
changed the title
MANAGED table migration and continued building tables migration component
nfx
approved these changes
Sep 28, 2023
nfx
added a commit
that referenced
this pull request
Sep 29, 2023
# Version changelog ## 0.2.0 * Added retrieving for all account-level groups with matching names to workspace-level groups in case no explicit configuration ([#277](#277)). * Added crawler for Azure Service principals used for direct storage access ([#305](#305)). * Added more SQL queries to the assessment step dashboard ([#269](#269)). * Added filtering out for job clusters in the clusters crawler ([#298](#298)). * Added recording errors from `crawl_tables` step in `$inventory.table_failures` table and display counter on the dashboard ([#300](#300)). * Added comprehensive introduction user manual ([#273](#273)). * Added interactive tutorial for local group migration readme ([#291](#291)). * Added tutorial links to the landing page of documentation ([#290](#290)). * Added (internal) support for account-level configuration and multi-cloud workspace list ([#264](#264)). * Improved order of tasks in the README notebook ([#286](#286)). * Improved installation script to run in a Windows Git Bash terminal ([#282](#282)). * Improved installation script by setting log level to uppercase by default ([#271](#271)). * Improved installation finish messages within installer script ([#267](#267)). * Improved automation for `MANAGED` table migration and continued building tables migration component ([#295](#295)). * Fixed debug notebook code with refactored package structure ([#250](#250)) ([#265](#265)). * Fixed replacement of custom configured database to replicate in the report for external locations ([#296](#296)). * Removed redundant `notebooks` top-level folder ([#263](#263)). * Split checking for test failures and linting errors into independent GitHub Actions checks ([#287](#287)). * Verify query metadata for assessment dashboards during unit tests ([#294](#294)).
Merged
nfx
added a commit
that referenced
this pull request
Sep 29, 2023
# Version changelog ## 0.2.0 * Added retrieving for all account-level groups with matching names to workspace-level groups in case no explicit configuration ([#277](#277)). * Added crawler for Azure Service principals used for direct storage access ([#305](#305)). * Added more SQL queries to the assessment step dashboard ([#269](#269)). * Added filtering out for job clusters in the clusters crawler ([#298](#298)). * Added recording errors from `crawl_tables` step in `$inventory.table_failures` table and display counter on the dashboard ([#300](#300)). * Added comprehensive introduction user manual ([#273](#273)). * Added interactive tutorial for local group migration readme ([#291](#291)). * Added tutorial links to the landing page of documentation ([#290](#290)). * Added (internal) support for account-level configuration and multi-cloud workspace list ([#264](#264)). * Improved order of tasks in the README notebook ([#286](#286)). * Improved installation script to run in a Windows Git Bash terminal ([#282](#282)). * Improved installation script by setting log level to uppercase by default ([#271](#271)). * Improved installation finish messages within installer script ([#267](#267)). * Improved automation for `MANAGED` table migration and continued building tables migration component ([#295](#295)). * Fixed debug notebook code with refactored package structure ([#250](#250)) ([#265](#265)). * Fixed replacement of custom configured database to replicate in the report for external locations ([#296](#296)). * Removed redundant `notebooks` top-level folder ([#263](#263)). * Split checking for test failures and linting errors into independent GitHub Actions checks ([#287](#287)). * Verify query metadata for assessment dashboards during unit tests ([#294](#294)).
FastLee
pushed a commit
that referenced
this pull request
Sep 29, 2023
…ing tables migration component (#295) Fixes #106 --------- Co-authored-by: Serge Smertin <[email protected]>
FastLee
pushed a commit
that referenced
this pull request
Sep 29, 2023
# Version changelog ## 0.2.0 * Added retrieving for all account-level groups with matching names to workspace-level groups in case no explicit configuration ([#277](#277)). * Added crawler for Azure Service principals used for direct storage access ([#305](#305)). * Added more SQL queries to the assessment step dashboard ([#269](#269)). * Added filtering out for job clusters in the clusters crawler ([#298](#298)). * Added recording errors from `crawl_tables` step in `$inventory.table_failures` table and display counter on the dashboard ([#300](#300)). * Added comprehensive introduction user manual ([#273](#273)). * Added interactive tutorial for local group migration readme ([#291](#291)). * Added tutorial links to the landing page of documentation ([#290](#290)). * Added (internal) support for account-level configuration and multi-cloud workspace list ([#264](#264)). * Improved order of tasks in the README notebook ([#286](#286)). * Improved installation script to run in a Windows Git Bash terminal ([#282](#282)). * Improved installation script by setting log level to uppercase by default ([#271](#271)). * Improved installation finish messages within installer script ([#267](#267)). * Improved automation for `MANAGED` table migration and continued building tables migration component ([#295](#295)). * Fixed debug notebook code with refactored package structure ([#250](#250)) ([#265](#265)). * Fixed replacement of custom configured database to replicate in the report for external locations ([#296](#296)). * Removed redundant `notebooks` top-level folder ([#263](#263)). * Split checking for test failures and linting errors into independent GitHub Actions checks ([#287](#287)). * Verify query metadata for assessment dashboards during unit tests ([#294](#294)).
Contributor
|
This issue is for "MANAGED" tables (Step 6 in go/uc/upgrade) SYNC is only for EXTERNAL tables (Step 5 in go/uc/upgrade) We should keep development and workflows separate for these |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #106