Apache Airflow Provider(s)

databricks

Versions of Apache Airflow Providers

apache-airflow-providers-databricks==3.0.0 # Latest

Apache Airflow version

2.3.2 (latest released)

Operating System

Linux

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

What happened

No response

What you think should happen instead

Motivation

In a single airflow deployment, we are looking to have the ability to support multiple databricks connections ( databricks_conn_id) at runtime. This can be helpful to run the same DAG against multiple testing lanes(a.k.a. different development/testing Databricks environments).

Potential Solution

We can pass the connection id via the Airflow DAG run configuration at runtime. For this, databricks_conn_id is required to be a templated field.

How to reproduce

Minor enhancement/new feature

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct