Skip to content

[DOC] (RADICAL) Improve Job documentation structure#3511

Merged
alexott merged 3 commits intodatabricks:mainfrom
landlord-matt:patch-4
May 2, 2024
Merged

[DOC] (RADICAL) Improve Job documentation structure#3511
alexott merged 3 commits intodatabricks:mainfrom
landlord-matt:patch-4

Conversation

@landlord-matt
Copy link
Copy Markdown
Contributor

@landlord-matt landlord-matt commented Apr 25, 2024

Changes

It seems the number of parameters has outgrown the current structure of the job documentation. This PR ensures that

  • all documented fields are listed in the first index section
  • the parameter hierarchy of the task parameters is reflected and the task parameters are grouped

Tests

  • make test run locally
  • relevant change in docs/ folder
  • covered with integration tests in internal/acceptance
  • relevant acceptance tests are passing
  • using Go SDK

@landlord-matt landlord-matt requested review from a team as code owners April 25, 2024 14:43
@landlord-matt landlord-matt requested review from hectorcast-db and removed request for a team April 25, 2024 14:43
@landlord-matt
Copy link
Copy Markdown
Contributor Author

Any thoughts on any of these two suggestions (repurposing the top of the argument reference as index and grouping the task parameters)? I guess it is not trivial to structure given that some of the arguments can be placed on both job and task level.

Perhaps main commiter @alexott?

@nkvuong nkvuong requested a review from alexott May 1, 2024 18:40
@alexott alexott enabled auto-merge May 2, 2024 11:02
@alexott alexott disabled auto-merge May 2, 2024 12:11
@alexott alexott enabled auto-merge May 2, 2024 12:11
@alexott alexott added this pull request to the merge queue May 2, 2024
@codecov-commenter
Copy link
Copy Markdown

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.70%. Comparing base (5d255ad) to head (381bdfc).
Report is 1 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #3511      +/-   ##
==========================================
- Coverage   82.71%   82.70%   -0.02%     
==========================================
  Files         183      183              
  Lines       18827    18827              
==========================================
- Hits        15573    15571       -2     
- Misses       2339     2340       +1     
- Partials      915      916       +1     

see 1 file with indirect coverage changes

Merged via the queue into databricks:main with commit fb762de May 2, 2024
@alexott
Copy link
Copy Markdown
Contributor

alexott commented May 2, 2024

Thank you @landlord-matt !

mgyucht added a commit that referenced this pull request May 3, 2024
### New Features and Improvements
* Allow changes to `display_name` for `databricks_service_principal` ([#3523](#3523)).
* Modify `databricks_repo` to support Git folders in the workspace ([#3447](#3447)).
* Retry requests in the Databricks Go SDK until context timeout ([#3537](#3537)).
* Set ForceSendFields for boolean/integer values explicitly set to false/0 ([#3385](#3385)).

### Bug fixes
* Added `ConflictsWith` to legacy tasks in `databricks_job` to avoid errors when job parameters are used ([#3500](#3500)).
* Fixed spn secret permanent diff ([#3525](#3525)).
* Introduced `common.NoClientData` helper function and use it in `databricks_aws_crossaccount_policy` data source to allow its use with workspace- or account-level providers ([#3508](#3508)).
* Set `id` attribute of `databricks_metastore` data source ([#3503](#3503)).

### Documentation Changes
* Added documentation for compute options for tasks in `databricks_job` resource ([#3504](#3504)).
* Added warning about depends_on order ([#3489](#3489)).
* Fixed various doc typos ([#3512](#3512)).
* Fixed `databricks_current_metastore` data source names in docs examples ([#3515](#3515)).
* Improved Job documentation structure ([#3511](#3511)).
* Removed confusing mention of cloud types for `databricks_clusters` and `databricks_sql_warehouses` data sources ([#3518](#3518)).

### Exporter
* Added UC support for DLT pipelines ([#3494](#3494)).
* Ignored instance pools with the empty name ([#3522](#3522)).

### Internal Changes
* Added DATABRICKS_USER_AGENT_EXTRA ([#3520](#3520)).
* Adjusted list of supported platforms in `godownloader-databricks-provider.sh` ([#3527](#3527)).
* Switched to Go SDK for `databricks_directory` resource and data source ([#3509](#3509)).
* Fixed codecov for repository ([#3530](#3530)).

Dependency updates:

 * Bump databricks-sdk-go to v0.40.0 ([#3534](#3534)).
@mgyucht mgyucht mentioned this pull request May 3, 2024
github-merge-queue bot pushed a commit that referenced this pull request May 3, 2024
* Release v1.42.0

### New Features and Improvements
* Allow changes to `display_name` for `databricks_service_principal` ([#3523](#3523)).
* Modify `databricks_repo` to support Git folders in the workspace ([#3447](#3447)).
* Retry requests in the Databricks Go SDK until context timeout ([#3537](#3537)).
* Set ForceSendFields for boolean/integer values explicitly set to false/0 ([#3385](#3385)).

### Bug fixes
* Added `ConflictsWith` to legacy tasks in `databricks_job` to avoid errors when job parameters are used ([#3500](#3500)).
* Fixed spn secret permanent diff ([#3525](#3525)).
* Introduced `common.NoClientData` helper function and use it in `databricks_aws_crossaccount_policy` data source to allow its use with workspace- or account-level providers ([#3508](#3508)).
* Set `id` attribute of `databricks_metastore` data source ([#3503](#3503)).

### Documentation Changes
* Added documentation for compute options for tasks in `databricks_job` resource ([#3504](#3504)).
* Added warning about depends_on order ([#3489](#3489)).
* Fixed various doc typos ([#3512](#3512)).
* Fixed `databricks_current_metastore` data source names in docs examples ([#3515](#3515)).
* Improved Job documentation structure ([#3511](#3511)).
* Removed confusing mention of cloud types for `databricks_clusters` and `databricks_sql_warehouses` data sources ([#3518](#3518)).

### Exporter
* Added UC support for DLT pipelines ([#3494](#3494)).
* Ignored instance pools with the empty name ([#3522](#3522)).

### Internal Changes
* Added DATABRICKS_USER_AGENT_EXTRA ([#3520](#3520)).
* Adjusted list of supported platforms in `godownloader-databricks-provider.sh` ([#3527](#3527)).
* Switched to Go SDK for `databricks_directory` resource and data source ([#3509](#3509)).
* Fixed codecov for repository ([#3530](#3530)).

Dependency updates:

 * Bump databricks-sdk-go to v0.40.0 ([#3534](#3534)).

* Changelog
@landlord-matt landlord-matt deleted the patch-4 branch May 7, 2024 07:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants