Update the default-python template according to Lakeflow conventions by lennartkats-db · Pull Request #3712 · databricks/cli

lennartkats-db · 2025-10-03T07:06:34Z

Changes

This updates the default-python template according to the latest Lakeflow conventions as established in #3671. Notably, the new template moves away from the use of notebooks for pipeline source code.

The new layout looks as follows when the user selects they want both the sample job and the sample pipeline:

📁 resources
├── sample_job.job.yml
└── sample_etl.pipeline.yml
📁 src
├── 📁 my_project — shared source code for use in jobs and/or pipelines
│ ├── __init__.py
│ └── main.py
└── 📁 my_project_etl — source code for the sample_etl pipeline
├── __init__.py
├── 📁 transformations
│ ├── __init__.py
│ ├── sample_zones_my_project.py
│ └── sample_trips_my_project.py
├── 📁 explorations — exploratory notebooks
│ ├── __init__.py
│ └── sample_exploration.ipynb
└── README.md
📁 tests — unit tests
📁 fixtures — fixtures (these can now be used with load_fixture)
databricks.yml
pyproject.toml
README.md

The template prompts have been updated to cater to this structure. Notably, they include a new prompt to manage the catalog and schema used by the template. These settings are propagated to both the job and the pipeline:

Welcome to the default Python template for Databricks Asset Bundles!

Answer the following questions to customize your project.
You can always change your configuration in the databricks.yml file later.

Note that https://e2-dogfood.staging.cloud.databricks.com is used for initialization.
(For information on how to change your profile, see https://docs.databricks.com/dev-tools/cli/profiles.html.)

Unique name for this project [my_project]: my_project
Include a Lakeflow job that runs a notebook: yes
Include an ETL pipeline: yes
Include a sample Python package that builds into a wheel file: yes
Use serverless compute: yes
Default catalog for any tables created by this project [main]: main
Use a personal schema for each user working on this project.
(This is recommended. Your personal schema will be 'main.lennart_kats'.): yes

✨ Your new project has been created in the 'my_project' directory!

To get started, refer to the project README.md file and the documentation at https://docs.databricks.com/dev-tools/bundles/index.html.

Testing

Standard unit testing, acceptance testing
AI excercised templates and with all permutations of options, deploying/testing/running/inspecting the result
Bug bash of the original lakeflow-pipelines template from Prepare a new "generic" template, use "src layout" for Lakeflow template #3671

eng-dev-ecosystem-bot · 2025-10-03T07:18:09Z

Run: 19064653887

	Env	🔄flaky	💚RECOVERED	🙈SKIP	✅pass	🙈skip
💚	aws linux		1	1	326	591
💚	aws windows		1	1	327	590
💚	aws-ucws linux		1	1	449	486
💚	aws-ucws windows		1	1	450	485
💚	azure linux		1	1	326	590
💚	azure windows		1	1	327	589
💚	azure-ucws linux		1	1	445	485
💚	azure-ucws windows		1	1	446	484
🔄	gcp linux	8	1	1	317	592
🔄	gcp windows	2	1	1	324	591

12 failing tests:

Test Name	aws linux	aws windows	aws-ucws linux	aws-ucws windows	azure linux	azure windows	azure-ucws linux	azure-ucws windows	gcp linux	gcp windows
TestAccept	💚R	💚R	💚R	💚R	💚R	💚R	💚R	💚R	💚R	💚R
TestAccept/bundle/resources/jobs/fail-on-active-runs	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestAccept/bundle/resources/jobs/fail-on-active-runs/DATABRICKS_BUNDLE_ENGINE=direct	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestAccept/bundle/resources/pipelines/lakeflow-pipeline	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestAccept/bundle/resources/pipelines/lakeflow-pipeline/DATABRICKS_BUNDLE_ENGINE=terraform	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestAccept/bundle/resources/pipelines/update/DATABRICKS_BUNDLE_ENGINE=direct	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestAccept/bundle/resources/pipelines/update/DATABRICKS_BUNDLE_ENGINE=terraform	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestAccept/bundle/run/app-with-job	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=no/NBOOK=no/PY=yes	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=yes/NBOOK=no/PY=no	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=direct/UV_PYTHON=3.10	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f
TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=terraform/UV_PYTHON=3.9	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f

Modernize the cli-pipelines template to follow the same conventions as the lakeflow-pipelines template (from PR #3712), while keeping transformations/ in the project root. Key changes: - Add pyproject.toml with modern Python project configuration - Add editable install via environment.dependencies in pipeline config - Add artifacts_dynamic_version preset for development - Simplify databricks.yml with clearer variable descriptions - Fix VS Code settings extraPaths to point to actual directories - Add library/versions.tmpl for reusable template definitions The template now supports edit mode for better development experience while maintaining compatibility with both Python and SQL variations. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

fjakobs

This looks good to me

...nce/bundle/templates/default-python/classic/output/my_default_python/.vscode/extensions.json

Remove unused files (default-python now depends on ../default!)

🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

…n-template # Conflicts: # NEXT_CHANGELOG.md # acceptance/auth/bundle_and_profile/output.txt # acceptance/bundle/templates/default-python/classic/out.plan_prod.direct.json # acceptance/bundle/templates/default-python/integration_classic/output.txt # acceptance/bundle/templates/lakeflow-pipelines/python/output/my_lakeflow_pipelines/resources/lakeflow_pipelines_etl.pipeline.yml # acceptance/bundle/templates/lakeflow-pipelines/python/output/my_lakeflow_pipelines/resources/my_lakeflow_pipelines_etl.pipeline.yml # acceptance/bundle/templates/lakeflow-pipelines/python/output/my_lakeflow_pipelines/src/lakeflow_pipelines_etl/README.md # acceptance/bundle/templates/lakeflow-pipelines/python/output/my_lakeflow_pipelines/src/my_lakeflow_pipelines_etl/README.md # acceptance/bundle/templates/lakeflow-pipelines/sql/output/my_lakeflow_pipelines/resources/lakeflow_pipelines_etl.pipeline.yml # acceptance/bundle/templates/lakeflow-pipelines/sql/output/my_lakeflow_pipelines/resources/my_lakeflow_pipelines_etl.pipeline.yml # acceptance/bundle/templates/lakeflow-pipelines/sql/output/my_lakeflow_pipelines/src/lakeflow_pipelines_etl/README.md # acceptance/bundle/templates/lakeflow-pipelines/sql/output/my_lakeflow_pipelines/src/my_lakeflow_pipelines_etl/README.md # libs/template/template.go # libs/template/templates/default/template/__preamble.tmpl # libs/template/templates/experimental-default-python-vnext/databricks_template_schema.json # libs/template/templates/lakeflow-pipelines/databricks_template_schema.json

…ibility This commit applies several critical fixes to the template system: 1. Non-UC workspace support (deeeb38): Set default_catalog to "hive_metastore" when no UC metastore is available, ensuring templates work on non-UC workspaces. Applied to default-python and lakeflow-pipelines templates. 2. Fix catalog property references (63758b4): Changed template conditionals from using the helper function `default_catalog` to the property `.default_catalog` in pipeline templates. This ensures proper evaluation of the user-provided value. 3. Serverless catalog field (0088b28): Always emit the catalog field for serverless pipelines, changing the comment to clarify that "Serverless compute requires Unity Catalog". Applied to both YAML and Python pipeline templates. 4. PyDABs compatibility: Added missing enable_pydabs property to default-python template schema to support the merged PyDABs infrastructure from main. Acceptance test outputs have been updated to reflect these changes.

.gitignore

Integration tests were failing because when a UC workspace has no metastore default catalog, the template would use "hive_metastore" for serverless pipelines, which the API rejects. Hardcode 'catalog: main' instead of '${var.catalog}' for this case. This provides a clear error if "main" doesn't exist and matches the approach used in origin/main. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

lennartkats-db requested review from andrewnester, anton-107, denik, pietern and shreyas-goenka as code owners October 3, 2025 07:06

lennartkats-db temporarily deployed to test-trigger-is October 3, 2025 07:06 — with GitHub Actions Inactive

lennartkats-db force-pushed the replace-default-python-template branch from 7415181 to 7b37662 Compare October 6, 2025 19:37

lennartkats-db temporarily deployed to test-trigger-is October 6, 2025 19:38 — with GitHub Actions Inactive

lennartkats-db temporarily deployed to test-trigger-is October 6, 2025 19:42 — with GitHub Actions Inactive

lennartkats-db temporarily deployed to test-trigger-is October 7, 2025 07:50 — with GitHub Actions Inactive

lennartkats-db temporarily deployed to test-trigger-is October 7, 2025 08:54 — with GitHub Actions Inactive

lennartkats-db force-pushed the replace-default-python-template branch from 2a8ea44 to 81371f2 Compare October 7, 2025 12:29

lennartkats-db temporarily deployed to test-trigger-is October 7, 2025 12:29 — with GitHub Actions Inactive

lennartkats-db temporarily deployed to test-trigger-is October 7, 2025 12:59 — with GitHub Actions Inactive

Base automatically changed from add-default-template to main October 13, 2025 08:09

lennartkats-db temporarily deployed to test-trigger-is October 13, 2025 09:16 — with GitHub Actions Inactive

lennartkats-db temporarily deployed to test-trigger-is October 13, 2025 09:18 — with GitHub Actions Inactive

lennartkats-db force-pushed the replace-default-python-template branch from ba4fa01 to 603fe28 Compare October 13, 2025 11:48

lennartkats-db temporarily deployed to test-trigger-is October 13, 2025 11:48 — with GitHub Actions Inactive

lennartkats-db temporarily deployed to test-trigger-is October 13, 2025 12:28 — with GitHub Actions Inactive

lennartkats-db temporarily deployed to test-trigger-is October 13, 2025 12:50 — with GitHub Actions Inactive

lennartkats-db temporarily deployed to test-trigger-is October 13, 2025 13:09 — with GitHub Actions Inactive

lennartkats-db temporarily deployed to test-trigger-is October 13, 2025 13:14 — with GitHub Actions Inactive

lennartkats-db temporarily deployed to test-trigger-is October 13, 2025 13:30 — with GitHub Actions Inactive

lennartkats-db temporarily deployed to test-trigger-is October 13, 2025 13:48 — with GitHub Actions Inactive

lennartkats-db temporarily deployed to test-trigger-is October 13, 2025 14:04 — with GitHub Actions Inactive

lennartkats-db temporarily deployed to test-trigger-is October 13, 2025 14:22 — with GitHub Actions Inactive

lennartkats-db temporarily deployed to test-trigger-is October 14, 2025 10:10 — with GitHub Actions Inactive

lennartkats-db temporarily deployed to test-trigger-is October 14, 2025 10:47 — with GitHub Actions Inactive

Fix URL

0910a80

Remove the project_name_short special casing

5e7f05e

lennartkats-db temporarily deployed to test-trigger-is October 16, 2025 13:20 — with GitHub Actions Inactive

lennartkats-db temporarily deployed to test-trigger-is October 16, 2025 13:48 — with GitHub Actions Inactive

lennartkats-db temporarily deployed to test-trigger-is October 17, 2025 09:08 — with GitHub Actions Inactive

lennartkats-db temporarily deployed to test-trigger-is October 17, 2025 16:26 — with GitHub Actions Inactive

lennartkats-db temporarily deployed to test-trigger-is October 21, 2025 10:02 — with GitHub Actions Inactive

lennartkats-db temporarily deployed to test-trigger-is October 22, 2025 08:11 — with GitHub Actions Inactive

lennartkats-db temporarily deployed to test-trigger-is October 22, 2025 18:27 — with GitHub Actions Inactive

lennartkats-db temporarily deployed to test-trigger-is October 22, 2025 18:52 — with GitHub Actions Inactive

lennartkats-db force-pushed the replace-default-python-template branch from 9bee4c9 to cb4b9cf Compare October 22, 2025 18:53

fjakobs reviewed Oct 23, 2025

View reviewed changes

...nce/bundle/templates/default-python/classic/output/my_default_python/.vscode/extensions.json Show resolved Hide resolved

pietern approved these changes Oct 27, 2025

View reviewed changes

lennartkats-db and others added 6 commits November 2, 2025 20:23

Avoid using environemnts in notebook task (PrPr feature)

1e34b95

Cleanup

52c67eb

Remove unused files (default-python now depends on ../default!)

Update acceptance tests for current state

24b914b

🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Add terraform state files to .gitignore

fe7fca4

pietern reviewed Nov 3, 2025

View reviewed changes

.gitignore Outdated Show resolved Hide resolved

pietern reviewed Nov 3, 2025

View reviewed changes

.gitignore Outdated Show resolved Hide resolved

lennartkats-db and others added 3 commits November 4, 2025 09:16

Cleanup .gitignore, add development entries

b5598b4

QA

e7daf19

github-actions bot mentioned this pull request Jan 3, 2026

chore(nix): update flake.lock (act, actionlint, awscli2 +25 more) i9wa4/dotfiles#79

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update the default-python template according to Lakeflow conventions#3712

Update the default-python template according to Lakeflow conventions#3712
lennartkats-db merged 25 commits intomainfrom
replace-default-python-template

lennartkats-db commented Oct 3, 2025 •

edited

Loading

Uh oh!

eng-dev-ecosystem-bot commented Oct 3, 2025 •

edited

Loading

Uh oh!

fjakobs left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

lennartkats-db commented Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Testing

Uh oh!

eng-dev-ecosystem-bot commented Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fjakobs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

lennartkats-db commented Oct 3, 2025 •

edited

Loading

eng-dev-ecosystem-bot commented Oct 3, 2025 •

edited

Loading