Repository navigation
Make Lang SDKs refuse mapped stub arg bindings they cannot bind - #76967
Open
jason810496 wants to merge 20 commits into
Open
jason810496 wants to merge 20 commits into
jason810496 wants to merge 20 commits into
Conversation
Restore the mapped (.expand()) stub arg-binding support split out of the unmapped PR: the Dag serializer captures per-parameter metadata (declaration order, defaults, value schemas) from the stub signature via the get_mapped_serialized_fields hook, ti_run derives per-map-index bindings from it with the map-index decomposition on SchedulerDictOfListsExpandInput, and XComArgBinding regains the map_index/element_index delivery fields across the task-sdk and ts-sdk generated models.
ADR-0016 keeps the mapping arithmetic in core, so a Lang SDK never re-implements it: the server picks each map index's element and sends it as an arg binding. The change ships in Execution API 2027-02-28 because 2026-10-30 is already released.
ti_run loads the task instance as a column row, which has no task attribute, so every mapped stub that needed producer contexts failed with a 500. An expand_kwargs() mapping entry must override a partial() value, as it does in Python. Exposing the mapped metadata as a property keeps callers off a private attribute, and the mapped prefix tells the per-map-index path apart from the parse-time one. Co-Authored-By: Claude <[email protected]>
Two fields describing the same per-parameter bindings made every reader branch on which one
a stub had, and the dedicated field needed {__type, __var} encoding that nothing in it requires.
A param kind keeps one plain-JSON list in declaration order, matching ADR-0016.
A typing.TypedDict annotation on Python < 3.12 used to ship with no value schema silently,
so it now fails parsing with the fix in the message.
Co-Authored-By: Claude <[email protected]>
The 2026-10-30 supervisor schema still has the arg_bindings field, so a plain model_dump puts it back as its None default after the downgrade removes it. Co-Authored-By: Claude <[email protected]>
The old name described where the argument gets resolved, not what it is, and read as if it covered only map(). The kind also covers zip() nesting that no single binding can hold. Co-Authored-By: Claude <[email protected]>
The supervisor now narrows each binding to its value and the runtime applies map() callables, so the server no longer needs an escape hatch for arguments it cannot finish. Shapes no single binding can carry fail at parse time instead, where the Dag author sees them, and an expand_kwargs() key now wins over its partial() value as it does in Python. Co-Authored-By: Claude <[email protected]>
A review of the mapped stub-task bindings found cases where the server and the Python task runner disagree, where a Dag fails only at run time, or where an older client silently runs a task with no arguments. Stored Dags may re-order expand() keys, strict expand_kwargs() and zip().map() were not honoured, and deterministic binding errors were retried as server errors. Co-Authored-By: Claude <[email protected]>
A second review found that a mapped upstream with missing rows could bind the wrong element, that old runtimes could still run a task with no arguments, and that some server faults were reported as Dag errors the client must not retry. The Python task handler binder also accepted binding shapes it cannot honour. Co-Authored-By: Claude <[email protected]>
Reviewers had to trace where match-pattern captures such as producer came from, and Any on the validator values hid which expand shapes they accept. Co-Authored-By: Claude <[email protected]>
The round-trip test only checked _arg_bindings, so losing partial_kwargs, the expand input or the strict flag in serialization would go unnoticed. Co-Authored-By: Claude <[email protected]>
A stale group id in a stored binding surfaced as a bare 500 KeyError instead of the 422 that other undeliverable bindings return. Co-Authored-By: Claude <[email protected]>
Upstream added a field to the same 2027-02-28 schema version, so the generated Go and TypeScript models have to include both changes. Co-Authored-By: Claude <[email protected]>
Keep this change focused on the API server; the Go, TypeScript and Java runtimes learn to refuse per-index bindings in a stacked PR. Co-Authored-By: Claude <[email protected]>
Mapped task support for stub tasks is planned, so a binding the handler cannot read is a missing feature rather than an invalid value. Co-Authored-By: Claude <[email protected]>
Mapped stub tasks now get bindings with per-index fields; ignoring those fields would bind an argument to the wrong value. Co-Authored-By: Claude <[email protected]>
Mapped stub-task arg bindings only reach runtimes at this version, and the supervisor refuses to start such a task on an older one. Co-Authored-By: Claude <[email protected]>
A mapped stub task's binding can need its map() callables applied or a key read from an expand_kwargs() mapping; ignoring those fields would bind the argument to the wrong value. Co-Authored-By: Claude <[email protected]>
The capability note said Airflow sends no arguments to mapped stubs, which is no longer true; the SDKs now refuse the per-index bindings instead. Co-Authored-By: Claude <[email protected]>
Lang SDK 1.1.0 adds mapped task support on the Airflow 3.5.0 timeline, so the error should say the gap is planned rather than imply a wrong upgrade. Co-Authored-By: Claude <[email protected]>
1 task done
jason810496
marked this pull request as ready for review
November 12, 2026 05:33
jason810496
requested review from
XD-DENG,
amoghrajesh,
ashb,
bolkedebruin,
bugraoz93,
gopidesupavan,
jscheffl,
potiuk and
uranusjr
as code owners
November 12, 2026 05:33
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
#70570 sends mapped stub tasks arg bindings with per-index fields, and a runtime that ignores those fields binds an argument to the wrong value instead of failing.
This PR bumps the Java SDK to supervisor schema
2027-02-28, and makes the Go, TypeScript and Java SDKs raise a not-implemented error as a sentinel for the mapped task feature gap, which Lang SDK 1.1.0 fills on the Airflow 3.5.0 timeline.Was generative AI tooling used to co-author this PR?