Skip to content

Commit c614c57

Browse files
authored
Decode deadline alert interval and callback without generic deserialization (#72651)
* Decode deadline alert interval and callback without generic deserialization decode_deadline_alert passed the Dag-author controlled interval and callback to airflow.sdk.serde.deserialize, which imports the class named in the payload and instantiates it with the encoded arguments. These decoders run in the scheduler and the API server whenever a serialized Dag is loaded, so any class under the airflow.* allow list could be constructed there. The security model says a Dag author reaches those processes only through registered plugins and providers, and the codebase enforces that at decode time for timetables, priority weight strategies and operator extra links. Deadline fields had no equivalent gate. Both fields are now rebuilt from their encoded form directly: * interval accepts a number, a timedelta payload, or a variable-interval payload carrying a key, each reconstructed from primitives. * callback accepts AsyncCallback or SyncCallback, selected from a fixed map rather than imported by name, with path as a string, queue/executor as optional strings, and unexpected fields refused. Neither reaches serde.deserialize, so the class a Dag author names in either field is never imported. Filtering in front of deserialize was tried first and was not sufficient. serde normalises the legacy {__type, __var} shape into __classname__ *inside* deserialize, so a payload inspected beforehand carries no class name to reject. Payloads are normalised before inspection here, and that case is tested. Known residual, deliberately not closed here: callback kwargs are still passed through generic deserialization, so a legitimate callback can carry an arbitrary allow-listed class under its kwargs. Deferring that decode to the process that runs the callback would close it, but the kwargs are consumed through two paths using two different encodings, and getting either wrong hands user code an encoded dict in place of its argument. The residual is not specific to deadlines -- it is the general property of deserializing Dag-author data, shared with every other serde call site. A test asserts the current behaviour so the gap stays visible and any change to it has to be deliberate. Tests assert the class is never constructed rather than that an error is raised. Against unpatched sources the callback case reports DID NOT RAISE and the interval case names the instance that had already been built. * Accept pre-3.2 callback paths and validate fields per callback class Review feedback on the deadline decoding gate. Callbacks moved out of airflow.sdk.definitions.deadline in 3.2, so alerts serialized by an earlier version name the old module. The allow list only held the current path, which would have made those rows undecodable on upgrade -- the same backward-compatibility case the interval allow list already covers. The permitted callback fields were a hardcoded set covering both subclasses at once, so a payload could carry queue on a SyncCallback or executor on an AsyncCallback. The set is per class, and each class already declares its own via serialized_fields(), so ask it rather than restating the answer here and letting the two drift. That also turns a TypeError raised from inside the rebuild into the intended refusal, and the check now runs before anything is reconstructed from the payload. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_012zrnHJHPchB83FtwrYRf5q
1 parent 7911e76 commit c614c57

3 files changed

Lines changed: 316 additions & 22 deletions

File tree

‎airflow-core/src/airflow/serialization/decoders.py‎

Lines changed: 129 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,7 @@
5656
from airflow.partition_mappers.base import PartitionMapper
5757
from airflow.partition_mappers.wait_policy import WaitPolicy
5858
from airflow.partition_mappers.window import Window
59+
from airflow.sdk.definitions.callback import AsyncCallback, SyncCallback # noqa: SDK001
5960
from airflow.timetables.base import Timetable as CoreTimetable
6061

6162
R = TypeVar("R")
@@ -182,15 +183,138 @@ def decode_deadline_reference(reference_data: dict):
182183
return reference_class.deserialize_reference(reference_data)
183184

184185

186+
_TIMEDELTA_CLASSNAME = "datetime.timedelta"
187+
_VARIABLE_INTERVAL_CLASSNAMES = frozenset(
188+
{
189+
"airflow.sdk.definitions.deadline.VariableInterval",
190+
"airflow.serialization.definitions.deadline.SerializedVariableInterval",
191+
}
192+
)
193+
# The callback classes lived here until 3.2 moved them to ``...definitions.callback``,
194+
# so alerts serialized by an earlier version name this module instead.
195+
_LEGACY_CALLBACK_MODULE = "airflow.sdk.definitions.deadline"
196+
197+
198+
def _normalised_payload(encoded: Any, field: str) -> tuple[str, Any]:
199+
"""
200+
Return ``(classname, data)`` for a serde payload, in either encoding.
201+
202+
``serde`` accepts a legacy ``{"__type": ..., "__var": ...}`` shape and rewrites it
203+
into the current one *inside* ``deserialize``. Anything inspecting the payload before
204+
that call therefore has to normalise it first, or the legacy spelling carries no
205+
``__classname__`` at the moment it is looked at and slips past unexamined.
206+
"""
207+
from airflow.sdk.serde import CLASSNAME, DATA, _convert
208+
209+
if not isinstance(encoded, dict):
210+
raise ValueError(f"Deadline {field} is not a serialized object.")
211+
converted = _convert(encoded)
212+
if not isinstance(converted, dict):
213+
raise ValueError(f"Deadline {field} is not a serialized object.")
214+
if CLASSNAME not in converted:
215+
raise ValueError(f"Deadline {field} does not name a class.")
216+
return converted[CLASSNAME], converted.get(DATA)
217+
218+
219+
def _decode_deadline_interval(raw_interval: Any) -> datetime.timedelta | SerializedVariableInterval:
220+
"""
221+
Build the interval from its encoded form without importing what the payload names.
222+
223+
Only three shapes are legitimate, and each is reconstructed from primitives directly.
224+
Nothing here reaches ``serde.deserialize``, so no class named by a Dag author is
225+
imported or instantiated in the scheduler or the API server.
226+
"""
227+
# Backward compatibility: previously stored as total_seconds().
228+
if isinstance(raw_interval, (int, float)) and not isinstance(raw_interval, bool):
229+
return datetime.timedelta(seconds=raw_interval)
230+
231+
classname, data = _normalised_payload(raw_interval, "interval")
232+
233+
if classname == _TIMEDELTA_CLASSNAME:
234+
if isinstance(data, (int, float)) and not isinstance(data, bool):
235+
return datetime.timedelta(seconds=data)
236+
raise ValueError("Deadline interval timedelta payload is not a number.")
237+
238+
if classname in _VARIABLE_INTERVAL_CLASSNAMES:
239+
key = data.get("key") if isinstance(data, dict) else None
240+
if not isinstance(key, str):
241+
raise ValueError("Deadline interval variable payload has no string key.")
242+
return SerializedVariableInterval(key=key)
243+
244+
raise ValueError(
245+
f"Refusing to deserialize {classname!r} as a deadline interval. "
246+
f"Permitted: {_TIMEDELTA_CLASSNAME}, {', '.join(sorted(_VARIABLE_INTERVAL_CLASSNAMES))}."
247+
)
248+
249+
250+
def _decode_deadline_callback(raw_callback: Any) -> AsyncCallback | SyncCallback:
251+
"""
252+
Build the callback from its encoded form without importing what the payload names.
253+
254+
``kwargs`` is still passed through generic deserialization, and that is a deliberate,
255+
documented limit rather than an oversight. Leaving it encoded would close a real
256+
residual -- a legitimate callback can carry an arbitrary allow-listed class under its
257+
kwargs, which serde constructs while the outer payload looks entirely valid -- but the
258+
kwargs are consumed through two different paths that use two different encodings
259+
(``BaseSerialization`` in the triggerer, serde here), and deferring the decode without
260+
getting both exactly right silently hands user code an encoded dict in place of its
261+
argument. That residual is not specific to deadlines: it is the general property of
262+
deserializing Dag-author data, shared with every other serde call site. Closing it
263+
belongs with that broader work, not smuggled in here.
264+
"""
265+
from airflow.sdk.definitions.callback import (
266+
AsyncCallback,
267+
SyncCallback,
268+
_SerializedCallbackPath,
269+
)
270+
271+
permitted = {
272+
f"{module}.{cls.__qualname__}": cls
273+
for cls in (AsyncCallback, SyncCallback)
274+
for module in (cls.__module__, _LEGACY_CALLBACK_MODULE)
275+
}
276+
classname, data = _normalised_payload(raw_callback, "callback")
277+
callback_cls = permitted.get(classname)
278+
if callback_cls is None:
279+
raise ValueError(
280+
f"Refusing to deserialize {classname!r} as a deadline callback. "
281+
f"Permitted: {', '.join(sorted(permitted))}."
282+
)
283+
if not isinstance(data, dict):
284+
raise ValueError("Deadline callback payload is not a mapping.")
285+
286+
# The fields each class declares are the fields it can be rebuilt from, so an unknown
287+
# one means the payload does not describe this callback and nothing below should run.
288+
# ``queue`` and ``executor`` belong to one subclass each, never to both.
289+
permitted_fields = set(callback_cls.serialized_fields())
290+
unexpected = set(data) - permitted_fields
291+
if unexpected:
292+
raise ValueError(f"Unexpected deadline callback fields: {', '.join(sorted(unexpected))}.")
293+
294+
path = data.get("path")
295+
if not isinstance(path, str):
296+
raise ValueError("Deadline callback payload has no string path.")
297+
298+
from airflow.sdk.serde import deserialize
299+
300+
raw_kwargs = data.get("kwargs") or {}
301+
fields: dict[str, Any] = {"kwargs": deserialize(raw_kwargs) if raw_kwargs else {}}
302+
for optional in sorted(permitted_fields - {"path", "kwargs"}):
303+
if optional in data:
304+
value = data[optional]
305+
if value is not None and not isinstance(value, str):
306+
raise ValueError(f"Deadline callback {optional} is not a string.")
307+
fields[optional] = value
308+
309+
return callback_cls(callback_callable=_SerializedCallbackPath(path), **fields)
310+
311+
185312
def decode_deadline_alert(encoded_data: dict):
186313
"""
187314
Decode a previously serialized deadline alert.
188315
189316
:meta private:
190317
"""
191-
from airflow.sdk.definitions.deadline import VariableInterval
192-
from airflow.sdk.serde import deserialize
193-
194318
data = encoded_data.get(Encoding.VAR, encoded_data)
195319

196320
reference_data = data[DeadlineAlertFields.REFERENCE]
@@ -204,28 +328,12 @@ def decode_deadline_alert(encoded_data: dict):
204328
"from a version that supports VariableInterval. Downgrade is not fully reversible."
205329
)
206330

207-
interval: datetime.timedelta | SerializedVariableInterval
208-
209-
# Backward compatibility: previously interval was stored as total_seconds() (float/int).
210-
# Handle numeric values by converting to timedelta.
211-
if isinstance(raw_interval, (int, float)):
212-
interval = datetime.timedelta(seconds=raw_interval)
213-
else:
214-
deserialized = deserialize(raw_interval)
215-
216-
if isinstance(deserialized, datetime.timedelta):
217-
interval = deserialized
218-
elif isinstance(deserialized, SerializedVariableInterval):
219-
interval = deserialized
220-
elif isinstance(deserialized, VariableInterval):
221-
interval = SerializedVariableInterval(key=deserialized.key)
222-
else:
223-
raise TypeError(f"Invalid interval type: {type(deserialized).__name__}")
331+
interval = _decode_deadline_interval(raw_interval)
224332

225333
return SerializedDeadlineAlert(
226334
reference=reference,
227335
interval=interval,
228-
callback=deserialize(data[DeadlineAlertFields.CALLBACK]),
336+
callback=_decode_deadline_callback(data[DeadlineAlertFields.CALLBACK]),
229337
name=data.get(DeadlineAlertFields.NAME),
230338
)
231339

‎airflow-core/tests/unit/serialization/test_serialized_objects.py‎

Lines changed: 186 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,7 @@
7979
AsyncCallback,
8080
DeadlineAlert,
8181
DeadlineReference,
82+
SyncCallback,
8283
VariableInterval,
8384
)
8485
from airflow.sdk.definitions.decorators import task
@@ -1632,3 +1633,188 @@ def test_serialized_dag_getitem_returns_task_group(dag_maker):
16321633
ser_dag = DagSerialization.from_dict(ser_dict)
16331634
assert isinstance(ser_dag, SerializedDAG)
16341635
assert ser_dag["section"].group_id == tg.group_id
1636+
1637+
1638+
class _DeadlineGadget:
1639+
"""Stands in for any allow-listed class a Dag author could name in a deadline field.
1640+
1641+
Deliberately constructible by serde -- it carries the ``serialize`` /
1642+
``deserialize`` pair serde requires. A gadget serde cannot build would make the
1643+
test pass for the wrong reason: deserialization would fail on its own and the
1644+
"was it constructed" assertion would never be exercised.
1645+
"""
1646+
1647+
instantiated = False
1648+
1649+
def __init__(self, **kwargs):
1650+
type(self).instantiated = True
1651+
self.kwargs = kwargs
1652+
1653+
def serialize(self):
1654+
return {}
1655+
1656+
@staticmethod
1657+
def deserialize(data, version):
1658+
return _DeadlineGadget(**(data or {}))
1659+
1660+
1661+
def _encode_as(classname: str) -> dict:
1662+
"""Build a serde payload naming ``classname``."""
1663+
from airflow.sdk.serde import CLASSNAME, DATA, VERSION
1664+
1665+
return {CLASSNAME: classname, VERSION: 1, DATA: {}}
1666+
1667+
1668+
@pytest.mark.parametrize("field", [DeadlineAlertFields.CALLBACK, DeadlineAlertFields.INTERVAL])
1669+
def test_deadline_fields_refuse_unexpected_classes_without_constructing_them(field):
1670+
"""A Dag author must not be able to name an arbitrary class in a deadline field.
1671+
1672+
These decoders run in the scheduler and API server whenever a serialized Dag is
1673+
loaded, and the Security Model says a Dag author reaches those processes only
1674+
through registered plugins. The assertion that matters is that the class is never
1675+
*constructed*: `deserialize()` instantiates before returning, so a check on the
1676+
returned object would already be too late for a class with side effects in
1677+
`__init__`.
1678+
"""
1679+
from unittest import mock
1680+
1681+
from airflow.sdk import serde
1682+
1683+
gadget_name = f"{_DeadlineGadget.__module__}.{_DeadlineGadget.__qualname__}"
1684+
_DeadlineGadget.instantiated = False
1685+
1686+
valid = DeadlineAlert(
1687+
reference=DeadlineReference.DAGRUN_QUEUED_AT,
1688+
interval=timedelta(hours=1),
1689+
callback=AsyncCallback(TEST_CALLBACK_PATH, kwargs=TEST_CALLBACK_KWARGS),
1690+
)
1691+
serialized = BaseSerialization.serialize(valid)
1692+
serialized[Encoding.VAR][field] = _encode_as(gadget_name)
1693+
1694+
# Make the gadget acceptable to serde itself, so the refusal comes from the
1695+
# deadline gate rather than from the general deserialization allow list.
1696+
with mock.patch.object(serde, "_extra_allowed", serde._extra_allowed | {gadget_name}):
1697+
with pytest.raises(ValueError, match="Refusing to deserialize"):
1698+
BaseSerialization.deserialize(serialized)
1699+
1700+
assert _DeadlineGadget.instantiated is False, "the class was constructed before being rejected"
1701+
1702+
1703+
def test_deadline_callback_kwargs_still_construct_nested_classes():
1704+
"""Documents a known residual: kwargs are still generically deserialized.
1705+
1706+
The outer payload is a valid AsyncCallback and the class sits under ``kwargs``,
1707+
which serde deserializes recursively. This is NOT closed here, and the test asserts
1708+
the current behaviour rather than the desired one so the gap is visible and a future
1709+
change has to update it deliberately.
1710+
1711+
It is not specific to deadlines -- it is the general property of deserializing
1712+
Dag-author data, shared with every other serde call site. Closing it means deferring
1713+
the kwargs decode to the process that runs the callback, which spans two consumption
1714+
paths using two different encodings and belongs with that broader work.
1715+
"""
1716+
from unittest import mock
1717+
1718+
from airflow.sdk import serde
1719+
1720+
gadget_name = f"{_DeadlineGadget.__module__}.{_DeadlineGadget.__qualname__}"
1721+
_DeadlineGadget.instantiated = False
1722+
1723+
valid = DeadlineAlert(
1724+
reference=DeadlineReference.DAGRUN_QUEUED_AT,
1725+
interval=timedelta(hours=1),
1726+
callback=AsyncCallback(TEST_CALLBACK_PATH, kwargs=TEST_CALLBACK_KWARGS),
1727+
)
1728+
serialized = BaseSerialization.serialize(valid)
1729+
callback_payload = serialized[Encoding.VAR][DeadlineAlertFields.CALLBACK]
1730+
callback_payload[serde.DATA]["kwargs"] = {"evil": _encode_as(gadget_name)}
1731+
1732+
with mock.patch.object(serde, "_extra_allowed", serde._extra_allowed | {gadget_name}):
1733+
BaseSerialization.deserialize(serialized)
1734+
1735+
assert _DeadlineGadget.instantiated is True, (
1736+
"kwargs deserialization behaviour changed -- if this is now closed, update the "
1737+
"docstring on _decode_deadline_callback and this test together"
1738+
)
1739+
1740+
1741+
@pytest.mark.parametrize("field", [DeadlineAlertFields.CALLBACK, DeadlineAlertFields.INTERVAL])
1742+
def test_deadline_fields_refuse_legacy_encoded_classes(field):
1743+
"""The legacy ``{"__type", "__var"}`` spelling must not slip past the check.
1744+
1745+
serde rewrites that shape into ``__classname__`` *inside* deserialize, so a payload
1746+
inspected beforehand carries no ``__classname__`` at all and an unnormalised check
1747+
sees nothing to reject.
1748+
"""
1749+
from unittest import mock
1750+
1751+
from airflow.sdk import serde
1752+
from airflow.sdk._shared.serialization import OLD_DATA, OLD_TYPE
1753+
1754+
gadget_name = f"{_DeadlineGadget.__module__}.{_DeadlineGadget.__qualname__}"
1755+
_DeadlineGadget.instantiated = False
1756+
1757+
valid = DeadlineAlert(
1758+
reference=DeadlineReference.DAGRUN_QUEUED_AT,
1759+
interval=timedelta(hours=1),
1760+
callback=AsyncCallback(TEST_CALLBACK_PATH, kwargs=TEST_CALLBACK_KWARGS),
1761+
)
1762+
serialized = BaseSerialization.serialize(valid)
1763+
serialized[Encoding.VAR][field] = {OLD_TYPE: gadget_name, OLD_DATA: {}}
1764+
1765+
with mock.patch.object(serde, "_extra_allowed", serde._extra_allowed | {gadget_name}):
1766+
with pytest.raises(ValueError, match="Refusing to deserialize"):
1767+
BaseSerialization.deserialize(serialized)
1768+
1769+
assert _DeadlineGadget.instantiated is False
1770+
1771+
1772+
@pytest.mark.parametrize("callback_cls", [AsyncCallback, SyncCallback])
1773+
def test_deadline_callback_accepts_pre_3_2_module_path(callback_cls):
1774+
"""Callbacks serialized before 3.2 name the module they were defined in back then.
1775+
1776+
3.2 moved them out of ``airflow.sdk.definitions.deadline`` into
1777+
``...definitions.callback``, so an alert stored by an earlier version carries the old
1778+
path. Rejecting it would make those rows undecodable on upgrade.
1779+
"""
1780+
from airflow.sdk import serde
1781+
1782+
valid = DeadlineAlert(
1783+
reference=DeadlineReference.DAGRUN_QUEUED_AT,
1784+
interval=timedelta(hours=1),
1785+
callback=callback_cls(TEST_CALLBACK_PATH, kwargs=TEST_CALLBACK_KWARGS),
1786+
)
1787+
serialized = BaseSerialization.serialize(valid)
1788+
payload = serialized[Encoding.VAR][DeadlineAlertFields.CALLBACK]
1789+
payload[serde.CLASSNAME] = f"airflow.sdk.definitions.deadline.{callback_cls.__qualname__}"
1790+
1791+
decoded = BaseSerialization.deserialize(serialized)
1792+
1793+
assert isinstance(decoded.callback, callback_cls)
1794+
assert decoded.callback.path == TEST_CALLBACK_PATH
1795+
assert decoded.callback.kwargs == TEST_CALLBACK_KWARGS
1796+
1797+
1798+
@pytest.mark.parametrize(
1799+
("callback_cls", "foreign_field"),
1800+
[(AsyncCallback, "executor"), (SyncCallback, "queue")],
1801+
ids=["async-rejects-executor", "sync-rejects-queue"],
1802+
)
1803+
def test_deadline_callback_rejects_field_belonging_to_the_other_subclass(callback_cls, foreign_field):
1804+
"""``queue`` and ``executor`` belong to one subclass each, not to both.
1805+
1806+
Accepting either for both classes would hand the constructor an argument it does not
1807+
take, turning a malformed payload into a TypeError from deep inside the rebuild.
1808+
"""
1809+
from airflow.sdk import serde
1810+
1811+
valid = DeadlineAlert(
1812+
reference=DeadlineReference.DAGRUN_QUEUED_AT,
1813+
interval=timedelta(hours=1),
1814+
callback=callback_cls(TEST_CALLBACK_PATH, kwargs=TEST_CALLBACK_KWARGS),
1815+
)
1816+
serialized = BaseSerialization.serialize(valid)
1817+
serialized[Encoding.VAR][DeadlineAlertFields.CALLBACK][serde.DATA][foreign_field] = "something"
1818+
1819+
with pytest.raises(ValueError, match=f"Unexpected deadline callback fields: {foreign_field}"):
1820+
BaseSerialization.deserialize(serialized)

‎generated/known_sdk_imports_in_core.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ airflow-core/src/airflow/models/xcom_arg.py::1
2323
airflow-core/src/airflow/plugins_manager.py::1
2424
airflow-core/src/airflow/providers_manager.py::5
2525
airflow-core/src/airflow/secrets/__init__.py::1
26-
airflow-core/src/airflow/serialization/decoders.py::2
26+
airflow-core/src/airflow/serialization/decoders.py::3
2727
airflow-core/src/airflow/serialization/definitions/baseoperator.py::1
2828
airflow-core/src/airflow/serialization/definitions/dag.py::1
2929
airflow-core/src/airflow/serialization/definitions/mappedoperator.py::5

0 commit comments

Comments
 (0)