Skip to content

Commit 63f5845

Browse files
zach-overflowashb
andauthored
Migrate Task SDK to httpx2 (#73116)
* Depend on httpx2 instead of httpx in the Task SDK httpx has stalled: fixes that landed in httpx2 (a fork of httpx 0.28.1) have not shipped upstream, and starlette has already migrated, so FastAPI following is a matter of time. Moving the Execution API client now keeps the SDK on a maintained HTTP stack. Objects do not cross the httpx/httpx2 boundary, so the in-process Execution API transports move with the client that consumes them; the rest of airflow-core stays on httpx and migrates separately. related: #70522 * Add newsfragment for the Task SDK httpx2 migration * Match the in-process test client transport to the installed Task SDK Compat tests pair the repo's devel-common helpers with a released Airflow whose Task SDK is still on httpx. Pinning the helper to httpx2 hands those clients a transport from the wrong package, and httpx rejects the response stream it gets back. * Rename newsfragment to the new PR number * Pin the Task SDK ast-grep hook to a frozen revision Every other third-party hook in the repo pins an immutable SHA; a mutable tag lets the hook change under CI without a diff. * Remove unneeded newsfragment * Verify Task SDK HTTP logging and simplify test helpers Both HTTP stacks can coexist while compatibility jobs use older SDK distributions. The migration needs regression coverage for its logging behavior without exposing development-only helpers as public APIs. * Use httpx2 in the Dag run note client test * Use httpx2 in the execution API shutdown tests Tests added on main for in-process shutdown still referenced httpx, which this branch no longer imports in the module. * Trust the system certificate store in the Task SDK API client httpx2 verifies against the system trust store via truststore, so keeping certifi for the Execution API client would add a direct dependency only to maintain a second, different set of trusted roots in the same worker. * Ban httpx imports in the Task SDK with ruff Ruff already enforces banned imports, so a separate ast-grep hook and its extra pre-commit dependency are unnecessary. extend-banned-api adds to the root banned-api table instead of replacing it. * Use httpx2 in the server termination acknowledgement test The test arrived on main after the Task SDK moved to httpx2 and still referenced httpx, which the module no longer imports. * Test the Task SDK SSL context against a real CA file Mocking load_verify_locations while also writing a CA file to disk tested neither path convincingly. Keeping the certificate-generation imports inside the fixture limits them to the one place that needs them. * Note when the in-process httpx resolver can be removed * Apply suggestion from @ashb Co-authored-by: Ash Berlin-Taylor <[email protected]> * Use httpx2 in the supervisor launch error test helper The helper arrived on main after the Task SDK moved to httpx2 and still referenced httpx, which the module no longer imports. * Remove the duplicated in-process httpx resolver note * Use httpx2 in the supervisor execution timeout test The test arrived on main after the Task SDK moved to httpx2 and still referenced httpx, which the module no longer imports. * Use httpx2 in the request handler tests The tests arrived on main after the Task SDK moved to httpx2; the client rejects httpx transports. --------- Co-authored-by: Ash Berlin-Taylor <[email protected]>
1 parent a1c3755 commit 63f5845

19 files changed

Lines changed: 666 additions & 553 deletions

File tree

‎airflow-core/docs/security/jwt_token_authentication.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -317,7 +317,7 @@ The token flows through the execution stack as follows:
317317
2. The workload JSON is passed to the worker process (via the executor-specific mechanism:
318318
Celery message, Kubernetes Pod spec, local subprocess arguments, etc.).
319319
3. The worker's ``execute_workload()`` function reads the workload JSON and extracts the token.
320-
4. The ``supervise_task()`` function receives the token and creates an ``httpx.Client`` instance
320+
4. The ``supervise_task()`` function receives the token and creates an ``httpx2.Client`` instance
321321
with ``BearerAuth(token)`` for all Execution API HTTP requests.
322322
5. The worker calls the ``/run`` endpoint with the ``workload``-scoped token to mark the task
323323
as running. The server responds with a fresh ``execution``-scoped token in the
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
Task SDK API client trusts the system certificate store instead of ``certifi``
2+
3+
With ``[api] client_use_public_certs`` enabled (the default), the Task SDK client that talks to the
4+
Execution API now trusts the operating system's certificate store through ``truststore`` instead of
5+
the ``certifi`` CA bundle. CAs from ``[api] ssl_ca_file`` and ``[api] ssl_cert`` are still trusted
6+
in addition to the system store, and ``certifi`` is no longer a Task SDK dependency.
7+
8+
Workers must have the public CAs they rely on installed in the system store (for example, the
9+
``ca-certificates`` package on Debian-based images). On macOS and Windows, certificate validation
10+
is delegated to the operating system, which can reject server certificates that OpenSSL accepted,
11+
such as private-CA certificates without the ``serverAuth`` extended key usage.

‎airflow-core/pyproject.toml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -103,6 +103,9 @@ dependencies = [
103103
# parsed with no field-count or field-size bound at all.
104104
"starlette>=1.3.1",
105105
"httpx>=0.25.0",
106+
# Only for InProcessExecutionAPI's transports, which feed httpx2-based Task SDK clients.
107+
# The rest of airflow-core is still on httpx; tracked at https://github.com/apache/airflow/issues/70522
108+
"httpx2>=2.0.0",
106109
'importlib_metadata>=6.5;python_version<"3.12"',
107110
'importlib_metadata>=7.0;python_version>="3.12"',
108111
"itsdangerous>=2.0",

‎airflow-core/src/airflow/api_fastapi/execution_api/in_process.py‎

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@
3030
from starlette.requests import Request
3131

3232
if TYPE_CHECKING:
33-
import httpx
33+
import httpx2
3434
from fastapi import FastAPI
3535

3636
logger = structlog.get_logger(logger_name=__name__)
@@ -86,7 +86,7 @@ class InProcessExecutionAPI:
8686
A helper class to make it possible to run the ExecutionAPI "in-process".
8787
8888
The sync version of this makes use of a2wsgi which runs the async loop in a separate thread. This is
89-
needed so that we can use the sync httpx client
89+
needed so that we can use the sync httpx2 client
9090
"""
9191

9292
_app: FastAPI | None = None
@@ -109,9 +109,10 @@ def app(self):
109109

110110
return self._app
111111

112+
# httpx2, not httpx: the only consumers are Task SDK clients, which subclass httpx2.Client.
112113
@cached_property
113-
def transport(self) -> httpx.WSGITransport:
114-
import httpx
114+
def transport(self) -> httpx2.WSGITransport:
115+
import httpx2
115116
from a2wsgi import ASGIMiddleware
116117

117118
from airflow import settings
@@ -133,7 +134,7 @@ async def start_lifespan(cm: AsyncExitStack, app: FastAPI):
133134
# Wait for startup so callers see a ready app and the finalizer can close an entered context.
134135
asyncio.run_coroutine_threadsafe(start_lifespan(cm, self.app), loop).result()
135136

136-
transport = httpx.WSGITransport(app=middleware) # type: ignore[arg-type]
137+
transport = httpx2.WSGITransport(app=middleware) # type: ignore[arg-type]
137138

138139
# Callers can retain the transport after dropping this instance; finalizing on self would
139140
# stop the loop while requests still need it.
@@ -142,7 +143,7 @@ async def start_lifespan(cm: AsyncExitStack, app: FastAPI):
142143
return transport
143144

144145
@cached_property
145-
def atransport(self) -> httpx.ASGITransport:
146-
import httpx
146+
def atransport(self) -> httpx2.ASGITransport:
147+
import httpx2
147148

148-
return httpx.ASGITransport(app=self.app)
149+
return httpx2.ASGITransport(app=self.app)

‎airflow-core/src/airflow/config_templates/config.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2067,7 +2067,7 @@ api:
20672067
default: ~
20682068
client_use_public_certs:
20692069
description: |
2070-
Enable loading of public CA certificates from certifi into the client SSL context.
2070+
Enable trusting public CA certificates from the system trust store in the client SSL context.
20712071
version_added: ~
20722072
type: boolean
20732073
example: ~

‎airflow-core/tests/unit/api_fastapi/execution_api/test_app.py‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@
2323
from unittest import mock
2424
from uuid import UUID
2525

26-
import httpx
26+
import httpx2
2727
import pytest
2828
from fastapi import FastAPI, Request, status
2929
from fastapi.params import Security as SecurityParam
@@ -81,7 +81,7 @@ def test_server_and_in_process_apps_isolate_auth(in_process_first, monkeypatch):
8181

8282
with (
8383
TestClient(server_app) as server,
84-
httpx.Client(transport=api.transport, base_url="http://in-process.invalid") as local,
84+
httpx2.Client(transport=api.transport, base_url="http://in-process.invalid") as local,
8585
):
8686
response = local.get("/variables/execution_api_hosting")
8787
assert response.status_code == 200
@@ -208,7 +208,7 @@ def test_routes_with_task_instance_id_param_enforce_ti_self(client):
208208
def test_in_process_execution_api_runs_without_jwt_secret():
209209
"""The in-process API must not require ``api_auth/jwt_secret`` to be configured."""
210210
api = InProcessExecutionAPI()
211-
with httpx.Client(transport=api.transport) as client:
211+
with httpx2.Client(transport=api.transport) as client:
212212
response = client.get("http://localhost/health")
213213
assert response.status_code == 200
214214

@@ -301,7 +301,7 @@ async def lifespan(app):
301301

302302
app.router.lifespan_context = lifespan
303303
api = InProcessExecutionAPI(app)
304-
with httpx.Client(transport=api.transport) as client:
304+
with httpx2.Client(transport=api.transport) as client:
305305
assert client.get("http://localhost/").json() == 1
306306
del client, api
307307
gc.collect()
@@ -315,7 +315,7 @@ def test_session_factory_remains_usable_after_in_process_shutdown(in_process_db_
315315
app, opened, closed = in_process_db_app
316316
engine, factory = settings.async_engine, settings.AsyncSession
317317
api = InProcessExecutionAPI(app)
318-
with httpx.Client(transport=api.transport) as client:
318+
with httpx2.Client(transport=api.transport) as client:
319319
assert client.get("http://localhost/").json() == 1
320320
del client, api
321321
gc.collect()

‎devel-common/src/tests_common/test_utils/in_process_taskrun.py‎

Lines changed: 32 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -39,11 +39,15 @@
3939

4040
from __future__ import annotations
4141

42+
import importlib
4243
import json
4344
from typing import TYPE_CHECKING, Any
4445

4546
if TYPE_CHECKING:
4647
from collections.abc import Callable
48+
from types import ModuleType
49+
50+
import httpx2
4751

4852
from airflow.sdk.api.client import Client
4953
from airflow.sdk.execution_time.supervisor import TaskRunResult
@@ -55,27 +59,45 @@
5559
_XCOM_PATH_PARTS = 5 # /xcoms/{dag_id}/{run_id}/{task_id}/{key}
5660

5761

62+
def _resolve_sdk_httpx() -> ModuleType:
63+
"""Return the httpx package the *installed* Task SDK ``Client`` subclasses.
64+
65+
Needed as these test_utils run against multiple versions of Airflow/sdk. This can be
66+
removed when the oldest version we test against is Airflow 3.4.
67+
68+
Compat jobs pair this helper with a released SDK still on ``httpx``. A transport from the
69+
wrong package trips httpx's own stream-type assertion.
70+
"""
71+
from airflow.sdk.api.client import Client
72+
73+
for base in Client.__mro__:
74+
root = base.__module__.partition(".")[0]
75+
if root in ("httpx", "httpx2"):
76+
return importlib.import_module(root)
77+
raise RuntimeError("Task SDK Client is built on neither httpx nor httpx2")
78+
79+
5880
def _remembering_handler(store: dict, run_context_json: bytes) -> Callable:
5981
"""A dry-run transport handler: valid run-context + XCom round-trip from ``store``, else no-op."""
60-
import httpx
82+
httpx_impl = _resolve_sdk_httpx()
6183

6284
from airflow.sdk.api.client import noop_handler
6385

64-
def handler(request: httpx.Request) -> httpx.Response:
86+
def handler(request: httpx2.Request) -> httpx2.Response:
6587
path = request.url.path
6688
if path.startswith("/task-instances/") and path.endswith("/run"):
67-
return httpx.Response(200, content=run_context_json)
89+
return httpx_impl.Response(200, content=run_context_json)
6890
parts = path.strip("/").split("/")
6991
if len(parts) == _XCOM_PATH_PARTS and parts[0] == "xcoms":
7092
dag_id, run_id, task_id, key = parts[1:]
7193
sig = (dag_id, run_id, task_id, key)
7294
if request.method == "POST":
7395
store[sig] = json.loads(request.content)
74-
return httpx.Response(201, json={"ok": True})
96+
return httpx_impl.Response(201, json={"ok": True})
7597
if request.method == "GET":
7698
if sig in store:
77-
return httpx.Response(200, json={"key": key, "value": store[sig]})
78-
return httpx.Response(404, json={"detail": "XCom not found"})
99+
return httpx_impl.Response(200, json={"key": key, "value": store[sig]})
100+
return httpx_impl.Response(404, json={"detail": "XCom not found"})
79101
return noop_handler(request)
80102

81103
return handler
@@ -87,7 +109,7 @@ def build_in_memory_client(ti_context) -> Client:
87109
``ti_context`` (a ``TIRunContext``) is replayed for the task-start request. Pushed XCom
88110
values are exposed as ``client.pushed_xcoms`` keyed by ``(dag_id, run_id, task_id, key)``.
89111
"""
90-
import httpx
112+
httpx_impl = _resolve_sdk_httpx()
91113

92114
from airflow.sdk.api.client import Client
93115

@@ -96,7 +118,9 @@ def build_in_memory_client(ti_context) -> Client:
96118
base_url=None,
97119
dry_run=True,
98120
token="",
99-
transport=httpx.MockTransport(_remembering_handler(store, ti_context.model_dump_json().encode())),
121+
transport=httpx_impl.MockTransport(
122+
_remembering_handler(store, ti_context.model_dump_json().encode())
123+
),
100124
)
101125
client.pushed_xcoms = store # type: ignore[attr-defined]
102126
return client

‎scripts/ci/prek/generate_execution_api_schema.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,15 +27,15 @@
2727
os.environ["_AIRFLOW__AS_LIBRARY"] = "1"
2828
sys.path.insert(0, str(Path("airflow-core/src").resolve()))
2929

30-
import httpx
30+
import httpx2
3131

3232
# Keep this import compatible with the target branch: the version check runs this
3333
# generator against both checkouts.
3434
from airflow.api_fastapi.execution_api.app import InProcessExecutionAPI
3535

3636
app = InProcessExecutionAPI()
3737
version = app.app.versions.version_values[0]
38-
client = httpx.Client(transport=app.transport)
38+
client = httpx2.Client(transport=app.transport)
3939
response = client.get(f"http://localhost/openapi.json?version={version}")
4040
response.raise_for_status()
4141
print(json.dumps(response.json()))

‎shared/logging/src/airflow_shared/logging/structlog.py‎

Lines changed: 7 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -420,22 +420,16 @@ def structlog_processors(
420420
# structlog to ignore.
421421

422422
import contextlib
423+
import importlib
423424

424425
import click
425426

426427
suppress: tuple[ModuleType, ...] = (click, contextlib)
427-
try:
428-
import httpcore
429-
430-
suppress = (*suppress, httpcore)
431-
except ImportError:
432-
pass
433-
try:
434-
import httpx
435-
436-
suppress = (*suppress, httpx)
437-
except ImportError:
438-
pass
428+
# airflow-core is still on httpx while the Task SDK is on httpx2, so either (or both) may
429+
# be installed; tracked at https://github.com/apache/airflow/issues/70522
430+
for module_name in ("httpcore", "httpcore2", "httpx", "httpx2"):
431+
with contextlib.suppress(ImportError):
432+
suppress = (*suppress, importlib.import_module(module_name))
439433

440434
if json_output:
441435
dict_exc_formatter = structlog.tracebacks.ExceptionDictTransformer(
@@ -690,6 +684,7 @@ def is_atty():
690684
"airflow": {"level": log_level.upper()},
691685
# These ones are too chatty even at info
692686
"httpx": {"level": "WARN"},
687+
"httpx2": {"level": "WARN"},
693688
"sqlalchemy.engine": {"level": "WARN"},
694689
"alembic.runtime.plugins": {"level": "WARN"},
695690
}

‎shared/logging/tests/logging/test_structlog.py‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@
2626
import textwrap
2727
from datetime import UTC, datetime
2828
from pathlib import Path
29+
from types import ModuleType
2930
from unittest import mock
3031

3132
import pytest
@@ -73,6 +74,34 @@ def configurer(**kwargs):
7374
return configurer
7475

7576

77+
@pytest.mark.parametrize("json_output", [True, False], ids=["json", "rich"])
78+
@pytest.mark.parametrize(
79+
"installed_modules",
80+
[
81+
pytest.param(("httpx2",), id="httpx2-only"),
82+
pytest.param(("httpcore2",), id="httpcore2-only"),
83+
pytest.param(("httpcore2", "httpx2"), id="new-stack"),
84+
pytest.param(("httpcore", "httpcore2", "httpx", "httpx2"), id="both-stacks"),
85+
],
86+
)
87+
@mock.patch("structlog.dev.RichTracebackFormatter", autospec=True)
88+
@mock.patch("structlog.tracebacks.ExceptionDictTransformer", autospec=True)
89+
def test_httpx2_tracebacks_are_suppressed(
90+
mock_dict_transformer, mock_rich_formatter, monkeypatch, installed_modules, json_output
91+
):
92+
modules = {name: ModuleType(name) for name in installed_modules}
93+
for name in ("httpcore", "httpcore2", "httpx", "httpx2"):
94+
monkeypatch.setitem(sys.modules, name, modules.get(name))
95+
monkeypatch.setenv("DEV", "1")
96+
97+
structlog_module.structlog_processors.__wrapped__(json_output=json_output)
98+
99+
formatter = mock_dict_transformer if json_output else mock_rich_formatter
100+
formatter.assert_called_once()
101+
suppressed = formatter.call_args.kwargs["suppress"]
102+
assert {module.__name__ for module in suppressed} == {"click", "contextlib", *installed_modules}
103+
104+
76105
@pytest.mark.parametrize(
77106
("get_logger", "config_kwargs", "extra_kwargs", "extra_output"),
78107
[
@@ -474,6 +503,21 @@ def test_alembic_runtime_plugin_setup_logs_are_suppressed(structlog_config):
474503
assert sio.getvalue() == "[alembic.runtime.plugins] Visible warning\n"
475504

476505

506+
def test_httpx2_logs_below_warning_are_suppressed(structlog_config, monkeypatch):
507+
logger = logging.getLogger("httpx2")
508+
monkeypatch.setattr(logger, "level", logging.INFO)
509+
510+
with structlog_config(
511+
colors=False,
512+
log_format="[%(name)s] %(message)s",
513+
log_level="INFO",
514+
) as sio:
515+
logger.info("Filtered request message")
516+
logger.warning("Visible warning")
517+
518+
assert sio.getvalue() == "[httpx2] Visible warning\n"
519+
520+
477521
def test_excepthook_installed_when_json_output_true(structlog_config):
478522
import sys
479523

0 commit comments

Comments
 (0)