Skip to content

Commit da6874a

Browse files
denikIsaac
denik
and
Isaac
authored
direct: allow clearing custom_max_retention_hours (catalog and schema) (#6792)
Removing `custom_max_retention_hours` from a catalog or schema left the old value in place — the field is `omitempty`, so it was dropped from the update request and every later plan re-proposed the change. Force-send it on catalog and schema updates. Verified against a real AWS workspace with raw API calls: `{"custom_max_retention_hours": 0}` clears it (create with 240h → PATCH 0 → GET returns 0) for both catalogs and schemas, whereas an empty PATCH is otherwise rejected with "Nothing to update". **Catalogs** are direct-only (no terraform converter — `validate_direct_only_resources` rejects them in terraform mode), so there is no cross-engine payload to diverge from. Flips `catalog_optional_fields.custom_max_retention_hours` to `CAN_REMOVE`. **Schemas** run through terraform, but the terraform `databricks_schema` resource cannot set this field at all (it silently drops it), so a schema config that uses it would drift under the `migrate` invariant. Handled by: - carrying the field on `schema.yml` (for the direct field-removal check) and excluding `schema.yml` from `migrate`, and - adding a terraform-compatible `schema_tf_compat.yml` that keeps `migrate` coverage for schemas. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <[email protected]>
1 parent 7956221 commit da6874a

17 files changed

Lines changed: 53 additions & 17 deletions

File tree

‎.nextchanges/bundles/clear-catalog-custom-max-retention.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
* direct: Allow clearing a catalog's or schema's `custom_max_retention_hours` by removing it from configuration. ([#6792](https://github.com/databricks/cli/pull/6792))

‎acceptance/bundle/invariant/configs/catalog_optional_fields.yml.tmpl‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,7 @@ resources:
66
foo:
77
name: test-catalog-$UNIQUE_NAME
88
comment: This is a test catalog
9-
# CANNOT_REMOVE: not force-sent on update (terraform omits it too), so
10-
# clearing it drops the field from the PATCH and the old value drifts back every plan.
11-
custom_max_retention_hours: 48
9+
custom_max_retention_hours: 48 # CAN_REMOVE
1210
managed_encryption_settings:
1311
customer_managed_key_id: 00000000-0000-0000-0000-000000000000
1412
properties:

‎acceptance/bundle/invariant/configs/schema.yml.tmpl‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ resources:
77
catalog_name: main
88
name: test-schema-$UNIQUE_NAME
99
comment: This is a test schema # CAN_REMOVE
10+
custom_max_retention_hours: 240 # CAN_REMOVE
1011
# CANNOT_REMOVE: properties is a map, and ForceSendFields is inert for maps, so
1112
# clearing the whole map drops it from the request and the old value drifts back every plan.
1213
# Kept on one line so a future fix can flip it to CAN_REMOVE and have grep -v strip it cleanly.

‎acceptance/bundle/invariant/configs/schema_tf_compat.yml.tmpl‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
bundle:
2+
name: test-bundle-$UNIQUE_NAME
3+
4+
# A schema without custom_max_retention_hours. The terraform databricks_schema resource cannot set
5+
# that field, so schema.yml (which carries it for the direct field-removal check) is excluded from
6+
# the migrate invariant; this terraform-compatible schema keeps migrate coverage for schemas.
7+
resources:
8+
schemas:
9+
foo:
10+
catalog_name: main
11+
name: test-schema-$UNIQUE_NAME
12+
comment: This is a test schema
13+
grants:
14+
- principal: account users
15+
privileges:
16+
- USE_SCHEMA

‎acceptance/bundle/invariant/continue_293/out.test.toml‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎acceptance/bundle/invariant/delete_idempotent/out.test.toml‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎acceptance/bundle/invariant/destroy_idempotent/out.test.toml‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎acceptance/bundle/invariant/migrate/out.test.toml‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎acceptance/bundle/invariant/migrate/test.toml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,11 @@
33
# handed over to the service (see the TODO in dstate.Open).
44
EnvMatrix.DMS = [""]
55

6+
# schema.yml sets custom_max_retention_hours, which the terraform databricks_schema resource cannot
7+
# set (only direct does), so a terraform-first migrate would drift. schema_tf_compat.yml keeps migrate
8+
# coverage for schemas.
9+
EnvMatrixExclude.no_schema_custom_retention = ["INPUT_CONFIG=schema.yml.tmpl"]
10+
611
# vector_search_endpoints and vector_search_indexes have no terraform converter
712
EnvMatrixExclude.no_vector_search_endpoint = ["INPUT_CONFIG=vector_search_endpoint.yml.tmpl"]
813
EnvMatrixExclude.no_vector_search_index = ["INPUT_CONFIG=vector_search_index.yml.tmpl"]

‎acceptance/bundle/invariant/no_drift/out.test.toml‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)