Skip to content

Add bundle.engine config setting for deployment engine selection#4749

Merged
denik merged 27 commits intomainfrom
denik/bundle-engine-config
Mar 16, 2026
Merged

Add bundle.engine config setting for deployment engine selection#4749
denik merged 27 commits intomainfrom
denik/bundle-engine-config

Conversation

@denik
Copy link
Copy Markdown
Contributor

@denik denik commented Mar 16, 2026

Summary

Add a new engine field at the bundle/target level that allows users to configure the deployment engine ("terraform" or "direct") in their bundle config.

Behavior

  • New bundles (no existing state): uses the configured engine, or defaults to terraform
  • Existing bundles (state present): if the configured engine doesn't match state, a warning is issued pointing to the config source, and the existing state's engine is used. Note, this is different from current behaviour, where databricks CLI refuses to proceed with config/state mismatch.
  • Priority: state > DATABRICKS_BUNDLE_ENGINE env var > bundle.engine config > default

Example

bundle:
  name: my-bundle
  engine: direct

targets:
  production:
    bundle:
      engine: terraform

Test plan

Unit tests and acceptance tests.

🤖 Generated with Claude Code

@eng-dev-ecosystem-bot
Copy link
Copy Markdown
Collaborator

eng-dev-ecosystem-bot commented Mar 16, 2026

Commit: 485496a

Run: 23155445827

Env 🟨​KNOWN 🔄​flaky 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
🟨​ aws linux 7 1 9 268 792 7:15
🟨​ aws windows 7 1 9 270 790 6:53
🔄​ aws-ucws linux 2 7 9 364 707 6:51
🔄​ aws-ucws windows 2 7 9 366 705 6:16
💚​ azure linux 2 11 271 790 5:55
💚​ azure windows 2 11 273 788 6:53
🔄​ azure-ucws linux 2 1 11 369 703 8:15
🔄​ azure-ucws windows 2 1 11 371 701 9:16
💚​ gcp linux 2 11 267 793 5:47
💚​ gcp windows 2 11 269 791 5:01
18 interesting tests: 9 SKIP, 7 KNOWN, 1 flaky, 1 RECOVERED
Test Name aws linux aws windows aws-ucws linux aws-ucws windows azure linux azure windows azure-ucws linux azure-ucws windows gcp linux gcp windows
🟨​ TestAccept 🟨​K 🟨​K 🔄​f 🔄​f 💚​R 💚​R 🔄​f 🔄​f 💚​R 💚​R
🙈​ TestAccept/bundle/resources/permissions 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions 🟨​K 🟨​K 💚​R 💚​R 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct 🟨​K 🟨​K 💚​R 💚​R
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform 🟨​K 🟨​K 💚​R 💚​R
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions 🟨​K 🟨​K 💚​R 💚​R 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct 🟨​K 🟨​K 💚​R 💚​R
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform 🟨​K 🟨​K 💚​R 💚​R
🙈​ TestAccept/bundle/resources/postgres_branches/basic 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/recreate 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/update_protected 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/without_branch_id 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_endpoints/basic 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_endpoints/recreate 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_projects/update_display_name 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/synced_database_tables/basic 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🔄​ TestAccept/ssh/connect-serverless-gpu 🙈​s 🙈​s 🔄​f 🔄​f 🙈​s 🙈​s 🔄​f 🔄​f 🙈​s 🙈​s
💚​ TestAccept/ssh/connection 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 20 slowest tests (at least 2 minutes):
duration env testname
3:45 gcp windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:41 gcp linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:18 aws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:17 gcp linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:07 azure windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:06 gcp windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:52 aws-ucws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:48 aws-ucws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:47 aws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:43 aws-ucws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:43 azure windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:42 aws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:40 aws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:40 aws-ucws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:39 azure linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:18 azure-ucws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:13 azure-ucws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:13 azure linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:08 azure-ucws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:07 azure-ucws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct

@denik denik force-pushed the denik/bundle-engine-config branch from 6d4cadb to bd5d33e Compare March 16, 2026 09:53
@denik denik temporarily deployed to test-trigger-is March 16, 2026 09:54 — with GitHub Actions Inactive
@denik denik temporarily deployed to test-trigger-is March 16, 2026 10:22 — with GitHub Actions Inactive
@denik denik temporarily deployed to test-trigger-is March 16, 2026 11:47 — with GitHub Actions Inactive
@denik denik temporarily deployed to test-trigger-is March 16, 2026 12:19 — with GitHub Actions Inactive
@denik denik temporarily deployed to test-trigger-is March 16, 2026 12:25 — with GitHub Actions Inactive
@denik denik temporarily deployed to test-trigger-is March 16, 2026 12:30 — with GitHub Actions Inactive
@denik denik temporarily deployed to test-trigger-is March 16, 2026 12:44 — with GitHub Actions Inactive
denik added a commit that referenced this pull request Mar 16, 2026
## Changes

- Remove `SkipEngineEnvVar` field from `ProcessOptions` in `process.go`
- In `migrate.go`, clear `DATABRICKS_BUNDLE_ENGINE` in the context via
`env.Set(ctx, engine.EnvVar, "")` before calling `ProcessBundleRet`,
achieving the same effect

## Why
Simplify env var handling for
#4749

## Test plan
Existing tests.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <[email protected]>
denik and others added 13 commits March 16, 2026 13:45
Add `engine` field to the `Deployment` struct, allowing users to configure
the deployment engine ("terraform" or "direct") in their bundle config at
the bundle or target level. The env var DATABRICKS_BUNDLE_ENGINE takes
priority over the config setting.

When state exists with a different engine than configured, a warning is
issued (instead of an error) pointing to the source of the mismatch, and
the existing state's engine is used.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
- Add ValidateEngine to FastValidate checks
- Add enum constraint (terraform/direct) to JSON schema
- Update annotations with description for engine field
- Update migrate command and --plan error messages to mention config

Co-Authored-By: Claude Opus 4.6 <[email protected]>
Engine mismatch between env var and state now produces a warning
instead of an error. Update test scripts to remove musterr expectations
and adjust titles accordingly.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
Revert acceptance test outputs that were regenerated by make test-update
but are unrelated to the engine config feature.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
- Rename EngineRequest to EngineSetting across the codebase
- Rename RequestFromEnv to SettingFromEnv
- Inline engine.Resolve logic into ResolveEngineSetting
- Remove the standalone Resolve function from engine package
- Rename ResolveEngineRequest to ResolveEngineSetting

Co-authored-by: Isaac
denik and others added 11 commits March 16, 2026 13:46
Rename acceptance/bundle/validate/engine-config to invalid-engine-bundle
for clarity, and add a new invalid-engine-target test that validates
engine config errors when set at the target level.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
When the deployment engine setting from DATABRICKS_BUNDLE_ENGINE
mismatches the existing state engine, log at info level (hidden by
default) instead of warning. Config-based mismatches still warn.

Adds acceptance test covering both behaviors and invalid env var error.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Replace IsEnvVar bool with ConfigType EngineType in EngineSetting so the
caller can distinguish whether the config agrees with the state independently
of the effective (env var) setting.

Warning logic: warn only when both the effective engine and the config engine
disagree with the state. If the env var overrides a config that matches the
state, log at info level to avoid noise.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Migration only makes sense when moving from terraform to direct. If the
config explicitly sets engine: terraform, the migrated state would
immediately be overridden on the next deploy. Fail early with a message
directing the user to change the setting to "engine: direct" first.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
@denik denik force-pushed the denik/bundle-engine-config branch from e29caf1 to 1022af2 Compare March 16, 2026 12:47
@denik denik temporarily deployed to test-trigger-is March 16, 2026 12:47 — with GitHub Actions Inactive
@denik denik temporarily deployed to test-trigger-is March 16, 2026 13:51 — with GitHub Actions Inactive
denik and others added 2 commits March 16, 2026 17:41
Ensures forward slashes on Windows in the source path shown in the
engine/state mismatch warning.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
@denik denik temporarily deployed to test-trigger-is March 16, 2026 16:42 — with GitHub Actions Inactive
@denik denik enabled auto-merge March 16, 2026 16:51
@denik denik temporarily deployed to test-trigger-is March 16, 2026 16:51 — with GitHub Actions Inactive
@denik denik added this pull request to the merge queue Mar 16, 2026
Merged via the queue into main with commit e97c00c Mar 16, 2026
18 checks passed
@denik denik deleted the denik/bundle-engine-config branch March 16, 2026 17:34
rauchy pushed a commit that referenced this pull request Mar 17, 2026
## Changes

- Remove `SkipEngineEnvVar` field from `ProcessOptions` in `process.go`
- In `migrate.go`, clear `DATABRICKS_BUNDLE_ENGINE` in the context via
`env.Set(ctx, engine.EnvVar, "")` before calling `ProcessBundleRet`,
achieving the same effect

## Why
Simplify env var handling for
#4749

## Test plan
Existing tests.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <[email protected]>
rauchy pushed a commit that referenced this pull request Mar 17, 2026
…4749)

## Summary

Add a new `engine` field at the bundle/target level that allows users to
configure the deployment engine (`"terraform"` or `"direct"`) in their
bundle config.

### Behavior
- New bundles (no existing state): uses the configured engine, or
defaults to `terraform`
- Existing bundles (state present): if the configured engine doesn't
match state, a **warning** is issued pointing to the config source, and
the existing state's engine is used. Note, this is different from
current behaviour, where databricks CLI refuses to proceed with
config/state mismatch.
- Priority: state > `DATABRICKS_BUNDLE_ENGINE` env var > `bundle.engine`
config > default

### Example
```yaml
bundle:
  name: my-bundle
  engine: direct

targets:
  production:
    bundle:
      engine: terraform
```

## Test plan
Unit tests and acceptance tests.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
deco-sdk-tagging bot added a commit that referenced this pull request Mar 18, 2026
## Release v0.295.0

### Notable Changes

* Databricks Asset Bundles have been renamed to Declarative Automation Bundles (DABs). This is a non-breaking change; no code or configuration modifications are required. See the [FAQ](https://docs.databricks.com/aws/en/dev-tools/bundles/faqs#why-was-databricks-asset-bundles-renamed-to-declarative-automation-bundles).
* Add `bundle.engine` config setting to select the deployment engine (`terraform` or [`direct`](https://docs.databricks.com/aws/en/dev-tools/bundles/direct)). The `bundle.engine` setting takes precedence over the `DATABRICKS_BUNDLE_ENGINE` environment variable. When the configured engine doesn't match existing deployment state, a warning is issued and the existing engine is used ([#4749](#4749), [#4782](#4782))

### CLI
* Add `databricks auth switch` command for setting the default profile ([#4651](#4651))
* Add positional argument support to `auth logout` ([#4744](#4744))
* Strip trailing slash from host in `auth login`, `auth token`, and `configure` commands ([#4633](#4633))

### Bundles
* Standardize `personal_schemas` enum across bundle templates ([#4401](#4401))
* engine/direct: Fix permanent drift on experiment name field ([#4627](#4627))
* engine/direct: Fix permissions state path to match input config schema ([#4703](#4703))
* Add default project name and success message to default-scala template ([#4661](#4661))
* Skip enum validation for unresolved variable references ([#4752](#4752))
* engine/direct: Support references to/from grants ([#4774](#4774))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants