[FEATURE] DAB: Support for YAML Dictionary variables #1298
Description
When writing Databricks Asset Bundles, it would be great if we could define things like job cluster configuration as one config block:
targets:
dev:
variables:
spark_version: 14.3.x-scala2.12
pool_id: 12334
cluster_config:
spark_version: ${var.spark_version}
instance_pool_id: ${var.pool_id}
data_security_mode: SINGLE_USER
runtime_engine: PHOTON
spark_conf:
spark.master: local[*, 4]
spark.databricks.cluster.profile: singleNode
custom_tags:
ResourceClass: SingleNode
spark_env_vars:
PYSPARK_PYTHON: /databricks/python3/bin/python3
num_workers: 0
prod:
variables:
spark_version: 14.3.x-scala2.12
pool_id: aaabbbbcccc
cluster_config:
spark_version: ${var.spark_version}
instance_pool_id: ${var.pool_id}
data_security_mode: SINGLE_USER
runtime_engine: PHOTON
autoscale:
min_workers: 10
max_workers: 999999
spark_env_vars:
PYSPARK_PYTHON: /databricks/python3/bin/python3
resources:
jobs:
job1:
name: job1
git_source:
git_url: ${var.git_url}
git_provider: ${var.git_provider}
git_branch: ${var.git_branch}
tasks:
- task_key: task1
spark_python_task:
python_file: task1.py
source: GIT
parameters:
- --environment
- ${bundle.target}
job_cluster_key: cluster
job_clusters:
- job_cluster_key: cluster
new_cluster: ${var.cluster_config}It would also be nice to be able to group variables using this method:
variables:
instance_pools:
description: The pools the use for jobs
default:
driver:
large: id here
small: id here
worker:
large: id here
small: id here
cluster_config:
spark_version: ${var.spark_version}
driver_instance_pool_id: ${var.instance_pools.driver.large}
instance_pool_id: ${var.instance_pools.worker.large}
data_security_mode: SINGLE_USER
runtime_engine: PHOTON
autoscale:
min_workers: 10
max_workers: 999999
spark_env_vars:
PYSPARK_PYTHON: /databricks/python3/bin/python3Please let me know if this is in progress, and either way I'll see if I can help contribute :)