Use databricks bundle init without Unity Catalog #3472
Description
Issue description
It should be possible to use databricks bundle init with the default-python template without having Unity Catalog.
Currently, running databricks bundle init default-python gives the following error if Unity Catalog is not available:
"Error: failed to compute file content for {{.project_name}}/resources/{{.project_name}}.pipeline.yml.tmpl. template: :7:20: executing "" at <default_catalog>: error calling default_catalog: Unity Catalog is not available for feature tier STANDARD_TIER."
We have workflows where we would like to create a new asset bundle without being connected to a Unity Catalog. The catalog name is injected during CI/CD, and when initializing the bundle, the Databricks workspace where the bundle is to be deployed might not even have been provisioned yet.
Instead of hard-coding the catalog name in the DAB resource yml files, the yml files should just use a variable placeholder (e.g. {catalog_name}), with the variable value being provided using an environment variable (BUNDLE_VAR_catalog_name) when the DAB is eventually deployed using databricks bundle deploy.
Configuration
Databricks CLI is in a "minimal configuration" state; an unrelated workspace (standard tier, no UC) has been used for databricks configure.
Steps to reproduce the behavior
(Optionally create config-file with values for "include_python", "include_notebook", "include_dlt", "project_name", "serverless", and optionally "default_catalog", "catalog", and/or "catalog_name" - so you do not have to enter values manually)
- Run
databricks bundle init default-python - See error
Expected Behavior
It should be possible to execute databricks bundle init default-python without having Unity Catalog available (e.g. by defining certain variables in the input json config-file).
Actual Behavior
Executing databricks bundle init default-python without having Unity Catalog available gives an error that cannot be circumvented other than
Possible work-arounds
The error occurs because the {{.project_name}}.pipeline.yml.tmpl template file refers to the {{default_catalog}} template helper function.
Our current work-around is to create our own bundle templates that does not have any usages of the {{default_catalog}} helper function.
Preferred solutions:
-
It should be possible to define the catalog used by the
{{.project_name}}.pipeline.yml.tmpltemplate file using the json config-file, such that the{{default_catalog}}helper function is never called. -
Alternatively, maybe the
{{default_catalog}}helper function should not cause a fatal error but instead just print to stderr and return an empty string.
OS and CLI version
Databricks CLI v0.265.0
Is this a regression?
No, not a regression AFAIK.
CLI versions tried:
- Databricks CLI v0.265.0
- Databricks CLI v0.244.0
Debug Logs
Output logs if you run the command with debug logs enabled. Example: databricks bundle deploy --log-level=debug. Redact if needed
❯ databricks bundle init default-python --config-file ../dab_init_vars.json --log-level=debug
13:24:29 Info: start pid=24329 version=0.265.0 args="databricks, bundle, init, default-python, --config-file, ../dab_init_vars.json, --log-level=debug"
13:24:29 Debug: Loading DEFAULT profile from /home/[REDACTED]/.databrickscfg pid=24329 sdk=true
Welcome to the default Python template for Databricks Asset Bundles!
13:24:29 Info: adding skip pattern: __preamble pid=24329 action=initialize-template sdk=true
13:24:29 Info: adding skip pattern: [REDACTED]/src/dlt_pipeline.ipynb pid=24329 action=initialize-template sdk=true
13:24:29 Info: adding skip pattern: [REDACTED]/resources/[REDACTED].pipeline.yml pid=24329 action=initialize-template sdk=true
13:24:29 Info: adding skip pattern: [REDACTED]/resources/.gitkeep pid=24329 action=initialize-template sdk=true
13:24:29 Info: added file to list of possible project files: __preamble pid=24329 action=initialize-template sdk=true
13:24:29 Info: added file to list of possible project files: [REDACTED]/.gitignore pid=24329 action=initialize-template sdk=true
13:24:29 Info: added file to list of possible project files: [REDACTED]/README.md pid=24329 action=initialize-template sdk=true
13:24:29 Debug: GET /api/2.0/preview/scim/v2/Me
< HTTP/2.0 200 OK
< {
< "active": true,
< "displayName": "[REDACTED]",
< "emails": [
< {
< "primary": true,
< "type": "work",
< "value": "[REDACTED]"
< }
< ],
< "externalId": "[REDACTED]",
< "groups": [
< {
< "$ref": "Groups/[REDACTED]",
< "display": "[REDACTED]",
< "type": "direct",
< "value": "[REDACTED]"
< }
< ],
< "id": "[REDACTED]",
< "name": {
< "familyName": "[REDACTED]",
< "givenName": "[REDACTED]"
< },
< "schemas": [
< "urn:ietf:params:scim:schemas:core:2.0:User",
< "urn:ietf:params:scim:schemas:extension:workspace:2.0:User"
< ],
< "userName": "[REDACTED]"
< } pid=24329 sdk=true
13:24:29 Info: added file to list of possible project files: [REDACTED]/databricks.yml pid=24329 action=initialize-template sdk=true
13:24:29 Info: added file to list of possible project files: [REDACTED]/pyproject.toml pid=24329 action=initialize-template sdk=true
13:24:29 Info: added file to list of possible project files: [REDACTED]/.vscode/__builtins__.pyi pid=24329 action=initialize-template sdk=true
13:24:29 Info: added file to list of possible project files: [REDACTED]/.vscode/extensions.json pid=24329 action=initialize-template sdk=true
13:24:29 Info: added file to list of possible project files: [REDACTED]/.vscode/settings.json pid=24329 action=initialize-template sdk=true
13:24:29 Info: added file to list of possible project files: [REDACTED]/fixtures/.gitkeep pid=24329 action=initialize-template sdk=true
13:24:29 Info: added file to list of possible project files: [REDACTED]/resources/.gitkeep pid=24329 action=initialize-template sdk=true
13:24:29 Info: added file to list of possible project files: [REDACTED]/resources/[REDACTED].job.yml pid=24329 action=initialize-template sdk=true
13:24:29 Debug: GET /api/2.1/unity-catalog/current-metastore-assignment
< HTTP/2.0 404 Not Found
< {
< "details": [
< {
< "@type": "type.googleapis.com/google.rpc.RequestInfo",
< "request_id": "[REDACTED]",
< "serving_data": ""
< }
< ],
< "error_code": "FEATURE_DISABLED",
< "message": "Unity Catalog is not available for feature tier STANDARD_TIER."
< } pid=24329 sdk=true
13:24:29 Debug: non-retriable error: Unity Catalog is not available for feature tier STANDARD_TIER. pid=24329 sdk=true
Error: failed to compute file content for {{.project_name}}/resources/{{.project_name}}.pipeline.yml.tmpl. template: :7:20: executing "" at <default_catalog>: error calling default_catalog: Unity Catalog is not available for feature tier STANDARD_TIER.
13:24:29 Info: failed execution pid=24329 exit_code=1 error="failed to compute file content for {{.project_name}}/resources/{{.project_name}}.pipeline.yml.tmpl. template: :7:20: executing \"\" at <default_catalog>: error calling default_catalog: Unity Catalog is not available for feature tier STANDARD_TIER."
13:24:29 Debug: no telemetry logs to upload pid=24329