Repository navigation
Support dynamic task mapping on stub tasks - #70570
Open
jason810496 wants to merge 15 commits into
Open
jason810496 wants to merge 15 commits into
jason810496 wants to merge 15 commits into
Conversation
This was referenced Jul 28, 2026
This was referenced Aug 4, 2026
1 task done
jason810496
force-pushed
the
feature/lang-sdk/taskflow-stub-dag-mapped
branch
from
August 12, 2026 08:52
2c949d8 to
b786830
Compare
18 of 91 tasks
Contributor
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. |
1 task done
jason810496
force-pushed
the
feature/lang-sdk/taskflow-stub-dag-mapped
branch
from
November 3, 2026 07:04
b786830 to
7518b09
Compare
Restore the mapped (.expand()) stub arg-binding support split out of the unmapped PR: the Dag serializer captures per-parameter metadata (declaration order, defaults, value schemas) from the stub signature via the get_mapped_serialized_fields hook, ti_run derives per-map-index bindings from it with the map-index decomposition on SchedulerDictOfListsExpandInput, and XComArgBinding regains the map_index/element_index delivery fields across the task-sdk and ts-sdk generated models.
ADR-0016 keeps the mapping arithmetic in core, so a Lang SDK never re-implements it: the server picks each map index's element and sends it as an arg binding. The change ships in Execution API 2027-02-28 because 2026-10-30 is already released.
ti_run loads the task instance as a column row, which has no task attribute, so every mapped stub that needed producer contexts failed with a 500. An expand_kwargs() mapping entry must override a partial() value, as it does in Python. Exposing the mapped metadata as a property keeps callers off a private attribute, and the mapped prefix tells the per-map-index path apart from the parse-time one. Co-Authored-By: Claude <[email protected]>
Two fields describing the same per-parameter bindings made every reader branch on which one
a stub had, and the dedicated field needed {__type, __var} encoding that nothing in it requires.
A param kind keeps one plain-JSON list in declaration order, matching ADR-0016.
A typing.TypedDict annotation on Python < 3.12 used to ship with no value schema silently,
so it now fails parsing with the fix in the message.
Co-Authored-By: Claude <[email protected]>
The 2026-10-30 supervisor schema still has the arg_bindings field, so a plain model_dump puts it back as its None default after the downgrade removes it. Co-Authored-By: Claude <[email protected]>
The old name described where the argument gets resolved, not what it is, and read as if it covered only map(). The kind also covers zip() nesting that no single binding can hold. Co-Authored-By: Claude <[email protected]>
The supervisor now narrows each binding to its value and the runtime applies map() callables, so the server no longer needs an escape hatch for arguments it cannot finish. Shapes no single binding can carry fail at parse time instead, where the Dag author sees them, and an expand_kwargs() key now wins over its partial() value as it does in Python. Co-Authored-By: Claude <[email protected]>
A review of the mapped stub-task bindings found cases where the server and the Python task runner disagree, where a Dag fails only at run time, or where an older client silently runs a task with no arguments. Stored Dags may re-order expand() keys, strict expand_kwargs() and zip().map() were not honoured, and deterministic binding errors were retried as server errors. Co-Authored-By: Claude <[email protected]>
A second review found that a mapped upstream with missing rows could bind the wrong element, that old runtimes could still run a task with no arguments, and that some server faults were reported as Dag errors the client must not retry. The Python task handler binder also accepted binding shapes it cannot honour. Co-Authored-By: Claude <[email protected]>
Reviewers had to trace where match-pattern captures such as producer came from, and Any on the validator values hid which expand shapes they accept. Co-Authored-By: Claude <[email protected]>
The round-trip test only checked _arg_bindings, so losing partial_kwargs, the expand input or the strict flag in serialization would go unnoticed. Co-Authored-By: Claude <[email protected]>
A stale group id in a stored binding surfaced as a bare 500 KeyError instead of the 422 that other undeliverable bindings return. Co-Authored-By: Claude <[email protected]>
Upstream added a field to the same 2027-02-28 schema version, so the generated Go and TypeScript models have to include both changes. Co-Authored-By: Claude <[email protected]>
jason810496
force-pushed
the
feature/lang-sdk/taskflow-stub-dag-mapped
branch
from
November 12, 2026 03:21
3f8a2a5 to
89fa2d7
Compare
Keep this change focused on the API server; the Go, TypeScript and Java runtimes learn to refuse per-index bindings in a stacked PR. Co-Authored-By: Claude <[email protected]>
Mapped task support for stub tasks is planned, so a binding the handler cannot read is a missing feature rather than an invalid value. Co-Authored-By: Claude <[email protected]>
1 task done
jason810496
marked this pull request as ready for review
November 12, 2026 05:33
jason810496
requested review from
XD-DENG,
amoghrajesh,
ashb,
bolkedebruin,
bugraoz93,
gopidesupavan,
jscheffl,
potiuk and
uranusjr
as code owners
November 12, 2026 05:33
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Support dynamic task mapping on stub tasks
A
@task.stubcan now be mapped with every Python mapping primitive. Atti_runthe API server works out which upstream row and element each map index reads and sends it inarg_bindings, as ADR-0016 decides, so no Lang SDK re-implements the mapping arithmetic. The server sends coordinates, never XCom values.Argument lifecycle: how parse time and the API server split the work for the example above.
Write path:
serialization/stub_arg_bindings.pybuilds_arg_bindingsfrom the live operator at parse time.Read path:
execution_api/services/stub_arg_bindings.pyderives this map index's bindings atti_run.expand()cross product,zip()(a newtuplebinding, one part per zipped argument),concat(),expand_kwargs(), and a stub inside a mapped task group.map_indexandelement_indexlocate one element of an upstream output,apply_mapmarksmap()callables still to run on it, andkwargandfallbackserveexpand_kwargs()over an upstream output, whose keys are unknown until it runs.sequencebinding, so an unmapped stub or apartial()argument can read a mapped upstream; parse time used to reject this.expand_keys, because a JSONB store reorders dict keys and the order fixes the cross product.Follow-ups, per ADR-0016:
2027-02-28.xcomandtuplebindings through the XCom backend and hands the runtime aliteral.ApplyMapruns a Python Dag'smap()callables for a stub TaskHandler.needs_mapped_lengthlets the supervisor measure a Lang-SDK producer's map length.Until the supervisor change lands, no runtime reads the new fields, so the stack has to land before the
2027-02-28versions are released.Compatibility: the change ships in Execution API version
2027-02-28and in the supervisor schema version of the same date.arg_bindingsfor a mapped stub, the old ignored-arguments behaviour. For an unmapped stub whose bindings need the new kinds or fields, such as asequenceor an argument of a mapped task group,ti_runanswers 422.2027-02-28, and asks to upgrade the SDK.NotImplementedErrorfor a binding kind or per-index field that mapped tasks need, instead of reading the wrong value. The Go, TypeScript and Java runtimes do the same in Make Lang SDKs refuse mapped stub arg bindings they cannot bind #76967, pointing at Lang SDK 1.1.0.Was generative AI tooling used to co-author this PR?