Skip to content

Commit 63c494f

Browse files
pankajastroclaude
andcommitted
Fix watcher ORM ban on Airflow 3.0 by detecting the supervisor at runtime
The previous version gate (airflow_supports_task_sdk_runtime, >= 3.1) routed all of Airflow 3.0 to the direct-DB ORM path for both the producer task-state fetch and the triggerer's XCom fetch. That premise -- "a triggerer is not a task runner, so SUPERVISOR_COMMS is never bound on 3.0" -- is wrong: - TriggererJobRunner.init_comms() binds task_runner.SUPERVISOR_COMMS and the triggerer handles GetXCom and GetTaskStates over its comms channel, so the task-SDK APIs DO work in a real 3.0 triggerer. - A real 3.0 worker and triggerer both FORBID direct ORM access: create_session raises "Direct database access via the ORM is not allowed in Airflow 3.0" (supervisor.py). - SUPERVISOR_COMMS is unbound only with no supervisor (dag.test() inline), where the task-SDK raises ImportError/NameError and the ORM IS permitted. A single version gate cannot tell "in a supervisor" from "not", so it crashed 3.0 in both the worker sensor poke (state fetch) and the triggerer (XCom fetch) with the ORM-ban RuntimeError. Decide at call time instead, on Airflow >= 3.0: - state.py build_producer_state_fetcher: try get_task_states; on NameError (no supervisor) fall back to the metadata-DB ORM. - triggerer.py get_xcom_val: try XCom.get_one; on ImportError/NameError fall back to the metadata-DB ORM. Airflow < 3.0 keeps the ORM-only path (always permitted). Remove the now-unused and misleading airflow_supports_task_sdk_runtime predicate from constants. Update tests: producer-state AF3 tests run at 3.0.0 (the actual bug version) plus a no-supervisor ORM-fallback case; get_xcom_val branch test expects the task-SDK path on 3.0 plus an ImportError/NameError ORM-fallback case. Refs apache/airflow#51816, apache/airflow#59093, BOSS-405 Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
1 parent e7f5262 commit 63c494f

5 files changed

Lines changed: 124 additions & 44 deletions

File tree

‎cosmos/constants.py‎

Lines changed: 0 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -10,18 +10,6 @@
1010
AIRFLOW_VERSION = Version(airflow.__version__)
1111

1212

13-
def airflow_supports_task_sdk_runtime(version: Version = AIRFLOW_VERSION) -> bool:
14-
"""Whether Airflow's task-SDK runtime XCom / task-state APIs are usable in the triggerer.
15-
16-
``XCom.get_one`` and ``RuntimeTaskInstance.get_task_states`` import ``SUPERVISOR_COMMS``
17-
from ``airflow.sdk.execution_time.task_runner`` -- a symbol that only exists on Airflow
18-
3.1+. On Airflow 3.0 those APIs raise outside a task-SDK supervisor process (most notably
19-
``dag.test()``, which runs triggers/tasks inline), so callers must read XCom and producer
20-
task state directly from the metadata DB instead. See apache/airflow#51816 and #59093.
21-
"""
22-
return version >= Version("3.1.0")
23-
24-
2513
# dbt materialization for models that are inlined as a CTE into downstream models and never written to the warehouse.
2614
DBT_EPHEMERAL_MATERIALIZATION = "ephemeral"
2715

‎cosmos/operators/_watcher/state.py‎

Lines changed: 36 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -145,16 +145,25 @@ def build_producer_state_fetcher(
145145
producer_task_id: str,
146146
logger: logging.Logger,
147147
) -> ProducerStateFetcher | None:
148-
"""Return a callable that fetches the producer task state for the given Airflow version."""
149-
150-
# Imported lazily: a module-level import of cosmos.constants here triggers a circular import
151-
# during Airflow plugin discovery (cosmos.plugin imports from a half-initialized constants).
152-
from cosmos.constants import airflow_supports_task_sdk_runtime
148+
"""Return a callable that fetches the producer task state for the given Airflow version.
149+
150+
On Airflow 3.x the same callable is used from two very different runtime contexts, and
151+
the correct data source differs between them -- so the choice is made at call time, not
152+
by version alone:
153+
154+
* Inside a task-SDK worker (the sensor's synchronous ``poke()``), direct ORM access is
155+
forbidden -- ``create_session`` raises ``RuntimeError: Direct database access via the
156+
ORM is not allowed in Airflow 3.0`` -- but ``RuntimeTaskInstance.get_task_states`` works
157+
because the worker binds ``SUPERVISOR_COMMS``.
158+
* Outside a supervisor (``dag.test()`` running inline, or the Airflow 3.0 triggerer process
159+
where ``SUPERVISOR_COMMS`` is never bound), ``get_task_states`` raises ``NameError`` while
160+
ORM access *is* permitted.
161+
162+
So on Airflow >= 3.0 we prefer ``get_task_states`` and fall back to the metadata DB only on
163+
``NameError`` (no supervisor). See apache/airflow#51816, #59093.
164+
"""
153165

154-
# Airflow < 3.1 reads producer task state directly from the metadata DB. The task-SDK
155-
# ``RuntimeTaskInstance.get_task_states`` raises on Airflow 3.0 outside a supervisor
156-
# (e.g. dag.test()), so it is only usable on 3.1+. See apache/airflow#51816, #59093.
157-
if not airflow_supports_task_sdk_runtime(airflow_version):
166+
def build_orm_fetcher() -> ProducerStateFetcher | None:
158167
try:
159168
TaskInstance, create_session = _load_airflow2_dependencies()
160169
except ImportError as exc: # pragma: no cover - defensive guard for stripped test envs
@@ -178,11 +187,18 @@ def fetch_state_orm() -> str | None:
178187

179188
return fetch_state_orm
180189

190+
# Airflow < 3.0 has no task-SDK runtime: read producer task state directly from the
191+
# metadata DB, which is always permitted there.
192+
if airflow_version < Version("3.0.0"):
193+
return build_orm_fetcher()
194+
181195
try:
182196
RuntimeTaskInstance = _load_airflow3_dependencies()
183197
except (ImportError, NameError) as exc: # pragma: no cover - Airflow 3 libs missing
184-
logger.warning("Could not load Airflow 3 RuntimeTaskInstance: %s", exc)
185-
return None
198+
logger.warning("Could not load Airflow 3 RuntimeTaskInstance, falling back to ORM: %s", exc)
199+
return build_orm_fetcher()
200+
201+
orm_fetcher = build_orm_fetcher()
186202

187203
def fetch_state_airflow3() -> str | None:
188204
try:
@@ -191,9 +207,15 @@ def fetch_state_airflow3() -> str | None:
191207
task_ids=[producer_task_id],
192208
run_ids=[run_id],
193209
)
194-
except NameError as exc: # pragma: no cover - Airflow 3.0 missing supervisor comms
195-
logger.warning("RuntimeTaskInstance.get_task_states unavailable due to NameError: %s", exc)
196-
return None
210+
except NameError as exc:
211+
# No task-SDK supervisor (e.g. dag.test() inline, or the Airflow 3.0 triggerer):
212+
# SUPERVISOR_COMMS is unbound. Read producer state from the metadata DB, which is
213+
# permitted in these non-supervisor contexts.
214+
logger.debug(
215+
"RuntimeTaskInstance.get_task_states unavailable (no supervisor: %s); reading state from the metadata DB",
216+
exc,
217+
)
218+
return orm_fetcher() if orm_fetcher is not None else None
197219
state = task_states.get(run_id, {}).get(producer_task_id)
198220
if state is not None:
199221
return str(state)

‎cosmos/operators/_watcher/triggerer.py‎

Lines changed: 14 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,9 @@
77

88
from airflow.triggers.base import BaseTrigger, TriggerEvent
99
from asgiref.sync import sync_to_async
10+
from packaging.version import Version
1011

11-
from cosmos.constants import _DBT_STARTUP_EVENTS_XCOM_KEY, AIRFLOW_VERSION, airflow_supports_task_sdk_runtime
12+
from cosmos.constants import _DBT_STARTUP_EVENTS_XCOM_KEY, AIRFLOW_VERSION
1213
from cosmos.listeners.dag_run_listener import EventStatus
1314
from cosmos.log import get_logger
1415
from cosmos.operators._watcher.state import (
@@ -116,11 +117,19 @@ async def get_xcom_val(self, key: str) -> Any | None:
116117
self.run_id,
117118
self.map_index,
118119
)
119-
# Airflow 3.0's task-SDK XCom API raises outside a supervisor (e.g. dag.test()), so only
120-
# use it on 3.1+; Airflow < 3.1 reads XCom directly from the metadata DB. See #51816.
121-
if airflow_supports_task_sdk_runtime(AIRFLOW_VERSION):
120+
# Airflow < 3.0 has no task-SDK runtime: read XCom directly from the metadata DB.
121+
if AIRFLOW_VERSION < Version("3.0.0"):
122+
return await self.get_xcom_val_af2(key)
123+
# Airflow >= 3.0: prefer the task-SDK XCom API. A real triggerer binds SUPERVISOR_COMMS
124+
# (TriggererJobRunner.init_comms) and forbids direct ORM access -- create_session raises
125+
# "Direct database access via the ORM is not allowed in Airflow 3.0" -- so XCom.get_one
126+
# is the only path that works there. Only outside a supervisor (dag.test() running
127+
# inline) is SUPERVISOR_COMMS unbound, making XCom.get_one raise ImportError/NameError;
128+
# there ORM access IS permitted, so fall back to the metadata DB. A version gate cannot
129+
# tell these apart. See apache/airflow#51816, #59093.
130+
try:
122131
return await self.get_xcom_val_af3(key)
123-
else:
132+
except (ImportError, NameError):
124133
return await self.get_xcom_val_af2(key)
125134

126135
async def _get_node_status(self) -> Any | None:

‎tests/operators/_watcher/test_triggerer.py‎

Lines changed: 36 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -133,9 +133,12 @@ async def test_parse_dbt_node_status_for_test_sensor(self):
133133
@pytest.mark.parametrize(
134134
"airflow_version, expected_val",
135135
[
136-
(Version("2.11.0"), "af2"), # Airflow < 3.1 uses the direct-DB get_xcom_val_af2
137-
(Version("3.0.0"), "af2"), # Airflow 3.0 lacks SUPERVISOR_COMMS, so also uses get_xcom_val_af2
138-
(Version("3.1.0"), "af3"), # Airflow >= 3.1 uses the task-SDK get_xcom_val_af3
136+
(Version("2.11.0"), "af2"), # Airflow < 3.0 reads XCom directly from the metadata DB
137+
# Airflow >= 3.0 uses the task-SDK get_xcom_val_af3: a real triggerer binds
138+
# SUPERVISOR_COMMS (init_comms) and forbids ORM access, so the SDK is the only path
139+
# that works there.
140+
(Version("3.0.0"), "af3"),
141+
(Version("3.1.0"), "af3"),
139142
],
140143
)
141144
async def test_get_xcom_val_branches(self, airflow_version, expected_val):
@@ -149,6 +152,19 @@ async def test_get_xcom_val_branches(self, airflow_version, expected_val):
149152
val = await self.trigger.get_xcom_val("key")
150153
assert val == "af3"
151154

155+
@pytest.mark.parametrize("raised", [ImportError("no SUPERVISOR_COMMS"), NameError("SUPERVISOR_COMMS")])
156+
async def test_get_xcom_val_af3_falls_back_to_orm_without_supervisor(self, raised):
157+
# On Airflow >= 3.0 the task-SDK XCom path is primary, but outside a supervisor
158+
# (dag.test() running inline) SUPERVISOR_COMMS is unbound and XCom.get_one raises
159+
# ImportError/NameError. There ORM access is permitted, so we must fall back to it.
160+
with patch("cosmos.operators._watcher.triggerer.AIRFLOW_VERSION", Version("3.0.0")):
161+
with patch.object(self.trigger, "get_xcom_val_af3", AsyncMock(side_effect=raised)):
162+
with patch.object(self.trigger, "get_xcom_val_af2", AsyncMock(return_value="af2")) as mock_af2:
163+
val = await self.trigger.get_xcom_val("key")
164+
165+
assert val == "af2"
166+
mock_af2.assert_awaited_once_with("key")
167+
152168
@pytest.mark.parametrize(
153169
"xcom_status_val, producer_state, expected",
154170
[
@@ -264,19 +280,30 @@ async def test_get_producer_task_status_airflow3_missing_state(self):
264280
AIRFLOW_VERSION < Version("3.0.0"), reason="RuntimeTaskInstance import exists on Airflow >= 3.0"
265281
)
266282
@pytest.mark.asyncio
267-
async def test_get_producer_task_status_airflow3_import_error(self):
283+
async def test_get_producer_task_status_airflow3_import_error_falls_back_to_orm(self):
284+
# If the task-SDK RuntimeTaskInstance cannot be imported (stripped env), the fetcher must
285+
# fall back to the metadata-DB ORM path rather than degrade to None.
268286
def _import_side_effect(name: str, *args, **kwargs):
269287
if name == "airflow.sdk.execution_time.task_runner":
270288
raise ModuleNotFoundError("missing runtime")
271289
return _real_import(name, *args, **kwargs)
272290

273-
# Pin to 3.1.0 so the task-SDK branch is taken; that is where the RuntimeTaskInstance
274-
# import is attempted (and here blocked), which must degrade to None rather than raise.
291+
orm_ti = MagicMock()
292+
orm_ti.state = "success"
293+
session = MagicMock()
294+
session.query.return_value.filter_by.return_value.one_or_none.return_value = orm_ti
295+
create_session_cm = MagicMock()
296+
create_session_cm.return_value.__enter__.return_value = session
297+
275298
with patch("cosmos.operators._watcher.triggerer.AIRFLOW_VERSION", Version("3.1.0")):
276-
with patch("builtins.__import__", side_effect=_import_side_effect):
277-
state = await self.trigger._get_producer_task_status()
299+
with patch(
300+
"cosmos.operators._watcher.state._load_airflow2_dependencies",
301+
return_value=(MagicMock(), create_session_cm),
302+
):
303+
with patch("builtins.__import__", side_effect=_import_side_effect):
304+
state = await self.trigger._get_producer_task_status()
278305

279-
assert state is None
306+
assert state == "success"
280307

281308
@pytest.mark.asyncio
282309
async def test_run_producer_success_model_not_run(self, caplog):

‎tests/operators/test_watcher.py‎

Lines changed: 38 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1064,8 +1064,8 @@ def test_get_producer_task_status_airflow2_missing_instance(self):
10641064
fetcher.assert_called_once_with()
10651065
assert status is None
10661066

1067-
@pytest.mark.skipif(AIRFLOW_VERSION < Version("3.1.0"), reason="task-SDK get_task_states path is Airflow >= 3.1")
1068-
@patch("cosmos.operators._watcher.base.AIRFLOW_VERSION", new=Version("3.1.0"))
1067+
@pytest.mark.skipif(AIRFLOW_VERSION < Version("3.0.0"), reason="task-SDK get_task_states path is Airflow >= 3.0")
1068+
@patch("cosmos.operators._watcher.base.AIRFLOW_VERSION", new=Version("3.0.0"))
10691069
@patch("airflow.sdk.execution_time.task_runner.RuntimeTaskInstance.get_task_states")
10701070
def test_get_producer_task_status_airflow3(self, mock_get_task_states):
10711071
sensor = self.make_sensor()
@@ -1085,8 +1085,8 @@ def test_get_producer_task_status_airflow3(self, mock_get_task_states):
10851085
dag_id="example_dag", task_ids=[sensor.producer_task_id], run_ids=["run_3"]
10861086
)
10871087

1088-
@pytest.mark.skipif(AIRFLOW_VERSION < Version("3.1.0"), reason="task-SDK get_task_states path is Airflow >= 3.1")
1089-
@patch("cosmos.operators._watcher.base.AIRFLOW_VERSION", new=Version("3.1.0"))
1088+
@pytest.mark.skipif(AIRFLOW_VERSION < Version("3.0.0"), reason="task-SDK get_task_states path is Airflow >= 3.0")
1089+
@patch("cosmos.operators._watcher.base.AIRFLOW_VERSION", new=Version("3.0.0"))
10901090
@patch("airflow.sdk.execution_time.task_runner.RuntimeTaskInstance.get_task_states")
10911091
def test_get_producer_task_status_airflow3_missing_state(self, mock_get_task_states):
10921092
sensor = self.make_sensor()
@@ -1106,6 +1106,40 @@ def test_get_producer_task_status_airflow3_missing_state(self, mock_get_task_sta
11061106
dag_id="example_dag", task_ids=[sensor.producer_task_id], run_ids=["run_3_missing"]
11071107
)
11081108

1109+
@pytest.mark.skipif(AIRFLOW_VERSION < Version("3.0.0"), reason="task-SDK get_task_states path is Airflow >= 3.0")
1110+
@patch("cosmos.operators._watcher.base.AIRFLOW_VERSION", new=Version("3.0.0"))
1111+
@patch("airflow.sdk.execution_time.task_runner.RuntimeTaskInstance.get_task_states")
1112+
def test_get_producer_task_status_airflow3_falls_back_to_orm_without_supervisor(self, mock_get_task_states):
1113+
# Regression for the Airflow 3.0 worker crash: inside a task-SDK worker get_task_states
1114+
# works, but with no supervisor (e.g. dag.test()) it raises NameError because
1115+
# SUPERVISOR_COMMS is unbound. There the metadata DB IS readable, so the fetcher must
1116+
# fall back to the ORM instead of propagating the NameError.
1117+
sensor = self.make_sensor()
1118+
sensor._get_producer_task_status = DbtConsumerWatcherSensor._get_producer_task_status.__get__(
1119+
sensor, DbtConsumerWatcherSensor
1120+
)
1121+
ti = MagicMock()
1122+
ti.dag_id = "example_dag"
1123+
context = self.make_context(ti, run_id="run_3_no_supervisor")
1124+
1125+
mock_get_task_states.side_effect = NameError("name 'SUPERVISOR_COMMS' is not defined")
1126+
1127+
orm_ti = MagicMock()
1128+
orm_ti.state = "success"
1129+
session = MagicMock()
1130+
session.query.return_value.filter_by.return_value.one_or_none.return_value = orm_ti
1131+
create_session_cm = MagicMock()
1132+
create_session_cm.return_value.__enter__.return_value = session
1133+
1134+
with patch(
1135+
"cosmos.operators._watcher.state._load_airflow2_dependencies",
1136+
return_value=(MagicMock(), create_session_cm),
1137+
):
1138+
status = sensor._get_producer_task_status(context)
1139+
1140+
assert status == "success"
1141+
mock_get_task_states.assert_called_once()
1142+
11091143
@pytest.mark.skipif(AIRFLOW_VERSION < Version("3.0.0"), reason="Database lookup path in Airflow < 3.0")
11101144
@patch("cosmos.operators._watcher.base.AIRFLOW_VERSION", new=Version("3.0.0"))
11111145
def test_get_producer_task_status_airflow3_import_error(self):

0 commit comments

Comments
 (0)