Skip to content

Commit e4c73f3

Browse files
committed
Clarify Airflow-params-to-Databricks-params mapping in operator docs
Address Lee-W's review feedback that the auto-injection example was hard to parse. Each operator's section now: - Names the exact Databricks API field being populated and links to its schema (parameters / job_parameters / per-task slots). - States explicitly that each <key>: <value> pair in params becomes one {"name": <key>, "default": <value>} entry (CreateJobs) or is passed through unchanged (RunNow / SubmitRun). - Splits params into a named variable in the CreateJobs example so the key/value to name/default mapping reads top-to-bottom.
1 parent d6da56f commit e4c73f3

3 files changed

Lines changed: 38 additions & 17 deletions

File tree

‎providers/databricks/docs/operators/jobs_create.rst‎

Lines changed: 22 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -61,23 +61,36 @@ Currently the named parameters that ``DatabricksCreateJobsOperator`` supports ar
6161
Forwarding Airflow Dag params as Databricks job parameters
6262
----------------------------------------------------------
6363

64+
The Databricks ``api/2.2/jobs/create`` endpoint accepts a top-level ``parameters`` field
65+
that defines `job-level parameters
66+
<https://docs.databricks.com/api/workspace/jobs/create#parameters>`_ — a list of objects
67+
with a ``name`` (the parameter name) and a ``default`` (its default value), for example
68+
``[{"name": "env", "default": "prod"}]``.
69+
6470
If ``parameters`` is not set in ``json`` and the operator's ``params`` dict is non-empty,
65-
the operator's ``params`` are automatically converted to job-level ``parameters`` (a list
66-
of ``{"name": k, "default": v}`` entries, the shape required by the
67-
``api/2.2/jobs/create`` endpoint) so that Airflow Dag params can be forwarded as
68-
Databricks job parameters without hardcoding them in ``json``. If ``json`` already
69-
contains ``parameters``, it is left untouched.
71+
each key/value pair in ``params`` is converted into one such ``{"name": key, "default":
72+
value}`` entry, so that Airflow Dag params can be forwarded as Databricks job parameters
73+
without hardcoding the API shape in ``json``. If ``json`` already contains ``parameters``,
74+
it is left untouched.
7075

7176
.. code-block:: python
7277
78+
# Airflow Dag params (key/value pairs)
79+
params = {"env": "prod", "batch_size": "100"}
80+
7381
create_job = DatabricksCreateJobsOperator(
7482
task_id="create_job",
7583
json={"name": "my-job", "tasks": [...]},
76-
params={"env": "prod", "batch_size": "100"},
84+
params=params,
7785
)
78-
# The created/reset job will have:
79-
# parameters=[{"name": "env", "default": "prod"},
80-
# {"name": "batch_size", "default": "100"}]
86+
87+
# The Databricks job created/reset by the operator will have:
88+
# parameters=[
89+
# {"name": "env", "default": "prod"},
90+
# {"name": "batch_size", "default": "100"},
91+
# ]
92+
# i.e. each "<key>: <value>" in params becomes one
93+
# {"name": "<key>", "default": "<value>"} entry in the job definition.
8194
8295
8396
Examples

‎providers/databricks/docs/operators/run_now.rst‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -52,11 +52,15 @@ All other parameters are optional and described in documentation for ``Databrick
5252
Forwarding Airflow Dag params as Databricks job parameters
5353
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
5454

55+
The Databricks ``api/2.2/jobs/run-now`` endpoint accepts a top-level `job_parameters
56+
<https://docs.databricks.com/api/workspace/jobs/runnow#job_parameters>`_ field — a plain
57+
``Dict[str, str]`` mapping parameter name to value — that overrides the job's defaults
58+
for this run.
59+
5560
If ``job_parameters`` is not set in ``json`` and the operator's ``params`` dict is
56-
non-empty, the operator's ``params`` are automatically forwarded as ``job_parameters``
57-
(a ``Dict[str, str]``, the shape required by the ``api/2.2/jobs/run-now`` endpoint) so
58-
that Airflow Dag params can be passed dynamically to Databricks runs without hardcoding
59-
them in ``json``. If ``json`` already contains ``job_parameters``, it is left untouched.
61+
non-empty, ``params`` is forwarded as ``job_parameters`` as-is, so Airflow Dag params can
62+
be passed dynamically to a run without hardcoding them in ``json``. If ``json`` already
63+
contains ``job_parameters``, it is left untouched.
6064

6165
.. code-block:: python
6266
@@ -67,6 +71,7 @@ them in ``json``. If ``json`` already contains ``job_parameters``, it is left un
6771
)
6872
# The triggered run receives:
6973
# job_parameters={"env": "staging", "batch_size": "42"}
74+
# i.e. the same dict, passed straight through to the run-now request body.
7075
7176
7277
DatabricksRunNowDeferrableOperator

‎providers/databricks/docs/operators/submit_run.rst‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -107,12 +107,14 @@ Another way to do is use the param tasks to pass array of objects to instantiate
107107
Forwarding Airflow Dag params as task parameters
108108
------------------------------------------------
109109

110-
The ``api/2.2/jobs/runs/submit`` endpoint has no top-level parameter slot — each task in
111-
``tasks`` carries its own parameters whose shape depends on the task type. If the
112-
operator's ``params`` dict is non-empty, it is automatically forwarded into the
110+
Unlike ``api/2.2/jobs/create`` and ``api/2.2/jobs/run-now``, the
111+
``api/2.2/jobs/runs/submit`` endpoint has no top-level parameter slot — each task in
112+
``tasks`` carries its own parameters whose shape depends on the task type.
113+
114+
If the operator's ``params`` dict is non-empty, it is forwarded as-is into the
113115
dict-shaped parameter slot of every task in ``json`` whose corresponding field is empty:
114116

115-
* ``notebook_task.base_parameters``
117+
* ``notebook_task.base_parameters`` (e.g. for ``notebook_task``)
116118
* ``python_wheel_task.named_parameters``
117119
* ``sql_task.parameters``
118120
* ``run_job_task.job_parameters``
@@ -132,6 +134,7 @@ or ``tasks`` argument.
132134
)
133135
# The submitted run's notebook_task.base_parameters becomes:
134136
# {"env": "dev", "shard": "1"}
137+
# i.e. the same dict, copied into the task's dict-shaped parameter slot.
135138
136139
137140
Examples

0 commit comments

Comments
 (0)