Repository navigation
Allow passing airflow params as job parameter in databricks job #39002
Description
Description
Currently if we need to pass any parameter to spark job we need to hardcode parameters in json config. But we want to pass parameters dynamically through airflow.
This feature allows to pass airflow config paramaters as job parameters in spark job.
Use case/motivation
Dynamically update job parameter in spark job
Related issues
no
Are you willing to submit a PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
potiuk commentedon Mar 2, 2026
@SubhamSinghal We are unassigning you from this issue as part of our updated assignment policy.
This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.
We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.
galafis commentedon Mar 16, 2026
I'd like to take this on. Noticed the previous PR #39007 was closed, and the issue was recently re-labeled as good first issue.
From what I understand, the goal is to allow Airflow DAG params (e.g., from
dag_run.conf) to be passed dynamically as job parameters to Databricks spark jobs, instead of having them hardcoded in the JSON config.I'll look into the
DatabricksSubmitRunOperatorandDatabricksRunNowOperatorto understand howjson/spark_submit_paramsare currently handled, and figure out the best way to inject Airflow params viatemplate_fieldsor an explicit mapping parameter. Will also add tests covering the new behavior.I'll open a PR soon.
laurenhall1526-netizen commentedon Mar 16, 2026
eladkal commentedon May 8, 2026
cc @moomindani not sure if this one is relevant but if is sounds like a very easy fix?
moomindani commentedon May 9, 2026
Hi @eladkal — taking this one on. Opened #66613 which builds on @SubhamSinghal's earlier work in #39007 (closed as stale). Picks up @dirrao's and @Lee-W's review feedback (list-comprehension refactor + use of
params.dump()to resolveParamobjects to their values) and extends the auto-injection fromDatabricksCreateJobsOperatortoDatabricksRunNowOperator(job_parameters) andDatabricksSubmitRunOperator(per-task dict-shaped slots) per @galafis's suggestion. Verified end-to-end against a real Databricks workspace.