Skip to content

Commit 352ca60

Browse files
rjgoylnhenry3260
andauthored
Fix API client errors losing detail behind a reverse proxy (#75493)
* Fix airflowctl tracebacks on JSON errors behind a reverse proxy airflowctl compared the error response content-type against the exact string "application/json". Any reverse proxy, ingress, or API gateway that returns "application/json; charset=utf-8" — or emits its own "application/problem+json" error — fell outside that comparison, so ServerResponseError was never constructed and the friendly "not found" messages in the dags and tasks commands were skipped. The bare httpx.HTTPStatusError raised instead is not handled by safe_call_command, so users saw a raw traceback. Airflow's own API server does not send a charset, which is why this only surfaces once something sits in front of it. Media types are case-insensitive and may carry parameters (RFC 9110), so the header has to be parsed rather than compared whole. Matching more responses also routes bodies into the error path that are valid JSON but not objects, which dict() rejected with a ValueError from that same unprotected path, and bodies that do not decode at all. * Move media type rationale from docstring into an inline comment * Apply the same JSON media type handling to the Task SDK client The Task SDK client carries the identical exact-string content-type comparison, so a task talking to the API server through a proxy that declares a charset loses the server's error detail and, because the raised exception is then a plain httpx.HTTPStatusError, no longer hits the status-specific handling in the supervisor and the task runner. * Cross-reference the twin JSON media type checks The airflowctl and Task SDK copies are the same three lines with different comments, and they already drifted in the commits that created them. A pointer in each makes the next person notice there are two. * Add newsfragment for the API client error handling fix Requested during review: the Task SDK half of the change ships in airflow-core, and both effects are visible to users running behind a proxy. * Use httpx2 in the Task SDK JSON media type check The Task SDK moved from httpx to httpx2 (#73116) after this branch was written, so the rebased helper and its tests referred to a module that is no longer imported there. * Describe the airflowctl symptom as it is on main now Since #73307 airflowctl catches the bare HTTPStatusError and prints one line instead of a traceback, so the newsfragment overstated it; the server's own message is still what gets lost. * Keep only the airflowctl test cases the fix changes The bare media type and the 404/500 axis repeat what test_error_parsing_with_unread_body already covers, and dict() of an object body worked before; none of them fail without the fix. * Explain the non-object log guard by what it protects The previous comment described the dict() call the guard replaced, which is no longer in the code. * Limit the newsfragment to the Task SDK fix airflow-core's release notes ship only the Task SDK side; airflowctl's changelog is built from git log. The task runner never handled ServerResponseError, so it is no longer named. * Rename the newsfragment to this PR's number * Shorten the media type comments The longer versions restated what the diff and the PR already explain, and the two _has_json_body copies now carry the same note. --------- Co-authored-by: Henry Chen <[email protected]>
1 parent e5f2c87 commit 352ca60

6 files changed

Lines changed: 119 additions & 6 deletions

File tree

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Fix the Task SDK client raising a bare ``HTTPStatusError`` instead of ``ServerResponseError`` when a reverse proxy rewrites the ``content-type`` of a JSON error response, for example to ``application/json; charset=utf-8``. The supervisor's and the client's status-specific handling, such as treating a 404 as a missing connection or variable, now applies to those responses.

‎airflow-ctl/src/airflowctl/api/client.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -113,7 +113,9 @@ def get_json_error(response: httpx.Response):
113113
# This part is used in integration tests to verify the error message
114114
# If you are updating here don't forget to update the airflow-ctl-tests
115115
if not response.request.extensions.get("airflowctl_suppress_error_log"):
116-
log.warning("Server error ", extra=dict(err.response.json()))
116+
detail = err.response.json()
117+
# ``extra`` needs a mapping, but a JSON error body can be any JSON value.
118+
log.warning("Server error ", extra=detail if isinstance(detail, dict) else {"detail": detail})
117119
raise err
118120

119121

‎airflow-ctl/src/airflowctl/api/operations.py‎

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,14 @@ def _build_query_params(**values: Any) -> dict[str, Any]:
106106
return {name: _serialize_query_param(value) for name, value in values.items() if value is not None}
107107

108108

109+
def _has_json_body(response: httpx.Response) -> bool:
110+
"""Check whether a response declares a JSON body."""
111+
# Kept in sync with airflow.sdk.api.client._has_json_body.
112+
# Media types are case-insensitive and may carry parameters (RFC 9110).
113+
media_type = response.headers.get("content-type", "").partition(";")[0].strip().lower()
114+
return media_type == "application/json" or media_type.endswith("+json")
115+
116+
109117
# Generic Server Response Error
110118
class ServerResponseError(httpx.HTTPStatusError):
111119
"""Server response error (Generic)."""
@@ -115,16 +123,22 @@ def from_response(cls, response: httpx.Response) -> ServerResponseError | None:
115123
if response.status_code < 400:
116124
return None
117125

118-
if response.headers.get("content-type") != "application/json":
126+
if not _has_json_body(response):
119127
return None
120128

121129
# httpx runs response event hooks before it reads the body, so the body has to be
122130
# pulled in explicitly here or ``.json()`` raises ``httpx.ResponseNotRead``.
123131
response.read()
124132

133+
try:
134+
detail = response.json()
135+
except ValueError:
136+
# A JSON content-type does not guarantee a JSON body; leave it to raise_for_status().
137+
return None
138+
125139
error_kind = "Client" if response.status_code < 500 else "Server"
126140
return cls(
127-
message=f"{error_kind} error message: {response.json()}",
141+
message=f"{error_kind} error message: {detail}",
128142
request=response.request,
129143
response=response,
130144
)

‎airflow-ctl/tests/airflow_ctl/api/test_client.py‎

Lines changed: 61 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@
2323
import subprocess
2424
import sys
2525
import tempfile
26+
from typing import Any
2627
from unittest.mock import MagicMock, patch
2728

2829
import httpx
@@ -57,7 +58,9 @@ def handle_request(request: httpx.Request) -> httpx.Response:
5758
return Client(base_url="", token="", mounts={"'http://": httpx.MockTransport(handle_request)})
5859

5960

60-
def make_unread_json_response(status_code: int, payload: dict, **kwargs) -> httpx.Response:
61+
def make_unread_json_response(
62+
status_code: int, payload: Any, content_type: str = "application/json", **kwargs
63+
) -> httpx.Response:
6164
"""
6265
Build a JSON response whose body has not been read yet.
6366
@@ -67,7 +70,7 @@ def make_unread_json_response(status_code: int, payload: dict, **kwargs) -> http
6770
"""
6871
return httpx.Response(
6972
status_code,
70-
headers={"content-type": "application/json"},
73+
headers={"content-type": content_type},
7174
content=iter([json.dumps(payload).encode()]),
7275
**kwargs,
7376
)
@@ -144,6 +147,62 @@ def test_error_parsing_with_unread_body(self, status_code, expected_message):
144147

145148
assert err.value.args == (expected_message,)
146149

150+
@pytest.mark.parametrize(
151+
"content_type",
152+
[
153+
pytest.param("application/json; charset=utf-8", id="charset"),
154+
pytest.param("application/json;charset=UTF-8", id="charset-no-space"),
155+
pytest.param("Application/JSON", id="mixed-case"),
156+
pytest.param("application/problem+json", id="problem-json"),
157+
],
158+
)
159+
def test_error_parsing_json_media_type_variants(self, content_type):
160+
response = make_unread_json_response(
161+
404,
162+
{"detail": "boom"},
163+
content_type=content_type,
164+
request=httpx.Request("GET", "http://error"),
165+
)
166+
167+
with pytest.raises(ServerResponseError):
168+
get_json_error(response)
169+
170+
def test_error_parsing_media_type_merely_prefixed_with_json(self):
171+
response = make_unread_json_response(
172+
404,
173+
{"detail": "boom"},
174+
content_type="application/jsonp",
175+
request=httpx.Request("GET", "http://error"),
176+
)
177+
178+
assert get_json_error(response) is None
179+
180+
def test_error_parsing_undecodable_json_body(self):
181+
response = httpx.Response(
182+
404,
183+
headers={"content-type": "application/json"},
184+
content=iter([b"{truncated"]),
185+
request=httpx.Request("GET", "http://error"),
186+
)
187+
188+
assert get_json_error(response) is None
189+
190+
@pytest.mark.parametrize(
191+
("payload", "expected_extra"),
192+
[
193+
pytest.param(["a", "b"], {"detail": ["a", "b"]}, id="array"),
194+
pytest.param("plain message", {"detail": "plain message"}, id="string"),
195+
],
196+
)
197+
@patch("airflowctl.api.client.log.warning", autospec=True)
198+
def test_error_log_accepts_non_object_json_body(self, mock_warning, payload, expected_extra):
199+
response = make_unread_json_response(404, payload, request=httpx.Request("GET", "http://error"))
200+
201+
with pytest.raises(ServerResponseError):
202+
get_json_error(response)
203+
204+
assert mock_warning.call_args.kwargs["extra"] == expected_extra
205+
147206
@pytest.mark.parametrize(
148207
("suppress_error_log", "expected_warning_count"),
149208
[

‎task-sdk/src/airflow/sdk/api/client.py‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1419,6 +1419,14 @@ def dags(self) -> DagsOperations:
14191419
return DagsOperations(self)
14201420

14211421

1422+
def _has_json_body(response: httpx2.Response) -> bool:
1423+
"""Check whether a response declares a JSON body."""
1424+
# Kept in sync with airflowctl.api.operations._has_json_body.
1425+
# Media types are case-insensitive and may carry parameters (RFC 9110).
1426+
media_type = response.headers.get("content-type", "").partition(";")[0].strip().lower()
1427+
return media_type == "application/json" or media_type.endswith("+json")
1428+
1429+
14221430
# This is only used for parsing. ServerResponseError is raised instead
14231431
class _ErrorBody(BaseModel):
14241432
detail: list[RemoteValidationError] | dict[str, Any] | str
@@ -1446,7 +1454,7 @@ def from_response(cls, response: httpx2.Response) -> ServerResponseError | None:
14461454
if not (400 <= response.status_code < 600):
14471455
return None
14481456

1449-
if response.headers.get("content-type") != "application/json":
1457+
if not _has_json_body(response):
14501458
return None
14511459

14521460
detail: list[RemoteValidationError] | dict[str, Any] | None = None

‎task-sdk/tests/task_sdk/api/test_client.py‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -200,6 +200,35 @@ def test_error_parsing_other_json(self):
200200
assert err.value.args == ("Not found",)
201201
assert err.value.detail is None
202202

203+
@pytest.mark.parametrize(
204+
"content_type",
205+
[
206+
pytest.param("application/json; charset=utf-8", id="charset"),
207+
pytest.param("application/json;charset=UTF-8", id="charset-no-space"),
208+
pytest.param("Application/JSON", id="mixed-case"),
209+
pytest.param("application/problem+json", id="problem-json"),
210+
],
211+
)
212+
def test_error_parsing_json_media_type_variants(self, content_type):
213+
responses = [
214+
httpx2.Response(404, json={"detail": "Not found"}, headers={"content-type": content_type})
215+
]
216+
client = make_client_w_responses(responses)
217+
218+
with pytest.raises(ServerResponseError) as err:
219+
client.get("http://error")
220+
assert err.value.args == ("Not found",)
221+
222+
def test_error_parsing_media_type_merely_prefixed_with_json(self):
223+
responses = [
224+
httpx2.Response(404, json={"detail": "Not found"}, headers={"content-type": "application/jsonp"})
225+
]
226+
client = make_client_w_responses(responses)
227+
228+
with pytest.raises(httpx2.HTTPStatusError) as err:
229+
client.get("http://error")
230+
assert not isinstance(err.value, ServerResponseError)
231+
203232
def test_server_response_error_pickling(self):
204233
responses = [httpx2.Response(404, json={"detail": {"message": "Invalid input"}})]
205234
client = make_client_w_responses(responses)

0 commit comments

Comments
 (0)