Add scoped Execution API access for Dag processors - #74370

* Resolve a trigger's XCom requests as its deferred task instance

Note: _None_ of this is a security boundary, and should not be treated as
such.

The Execution API resolves XCom reads and writes for a task inside a loop
to the caller's own iteration, using the task identity in the caller's JWT.
Triggers have no such identity: the triggerer talks to an in-process API
that treats every request as the all-zero task instance. A trigger's XCom
requests therefore resolve as an anonymous caller rather than as the
deferred task it runs for, so it can neither read (a 400) nor write the
XComs of a task inside a loop.

Triggers can also share one runner coroutine and one comms channel (the
"shared streams" idea), so only the asyncio task running a trigger's code
knows which trigger it is; a context variable carries that trigger's task
instance id to each request frame. Shared-stream groups are started from
whichever trigger subscribes first but serve several triggers, so they start
with it cleared rather than inheriting that trigger's id. This isn't ideal,
but will do "for now".

The trigger runner process (the one running async code) sets the task
instance id, and since that process runs user code a trigger can claim to be
any task instance. The (in-process) API therefore checks the claim. A write
acts as the task instance only when it is deferred and its trigger is
already assigned to the calling triggerer. A read acts as it whenever that
try is still live, because after a user marks a deferred task failed or
successful the trigger row is gone and the triggerer still runs the
trigger's on_kill(). The most a misbehaving trigger can do is write as
another task that is deferred on the same triggerer, or read as any live
task, and it can already read and write every task's XComs by coordinates
through the in-process API. As stated, this is _not_ a security boundary we
can rely on

Requests that fail the check are rejected rather than falling back to the
anonymous identity, deliberately: a trigger whose task instance is no
longer deferred, or whose trigger was reassigned, should not still be
writing its task's XComs, and falling back would only work for tasks
outside loops. Clearing a deferred task archives its try before on_kill()
runs, so on_kill() after a clear cannot read its task's XComs; a trigger
that needs a job id there should keep it on the trigger, as the in-tree
triggers already do.

Only the in-process API that the triggerer creates knows which triggerer
is calling, so every other API rejects the header rather than trusting it.
A triggerer that eventually talks to a real API server will need its own
credential before this can work there.

Only the XCom routes honour the trigger's task instance id for now.
get_previous_task_instance also resolves relative to the caller, so a
trigger asking for its previous task instance still gets the latest pass
rather than the one at its own loop position. Every other request from the
triggerer keeps its current identity, so access rules and the live-attempt
check on those routes are unchanged.

* fixup! Resolve a trigger's XCom requests as its deferred task instance

Pin the HEAD and DELETE rules in the trigger claim check and assert which
check denied each 403, so loosening the read-method test no longer passes.

* fixup! Resolve a trigger's XCom requests as its deferred task instance

Ignore a trigger failure reported by a triggerer that no longer owns the
trigger. The XCom write check now answers 403 to a stale copy on its
previous owner, which raises before yielding and would otherwise have
Trigger.submit_failure send the task to __fail__ while the new owner is
still running it.

(cherry picked from commit 61f1d4b)

Adapted to v3-4-test's CurrentTIToken: the typed execution tokens from the
Dag processor scoped access work (#74370) are not on this branch.

…) (#74868)

* Resolve a trigger's XCom requests as its deferred task instance

Note: _None_ of this is a security boundary, and should not be treated as
such.

The Execution API resolves XCom reads and writes for a task inside a loop
to the caller's own iteration, using the task identity in the caller's JWT.
Triggers have no such identity: the triggerer talks to an in-process API
that treats every request as the all-zero task instance. A trigger's XCom
requests therefore resolve as an anonymous caller rather than as the
deferred task it runs for, so it can neither read (a 400) nor write the
XComs of a task inside a loop.

Triggers can also share one runner coroutine and one comms channel (the
"shared streams" idea), so only the asyncio task running a trigger's code
knows which trigger it is; a context variable carries that trigger's task
instance id to each request frame. Shared-stream groups are started from
whichever trigger subscribes first but serve several triggers, so they start
with it cleared rather than inheriting that trigger's id. This isn't ideal,
but will do "for now".

The trigger runner process (the one running async code) sets the task
instance id, and since that process runs user code a trigger can claim to be
any task instance. The (in-process) API therefore checks the claim. A write
acts as the task instance only when it is deferred and its trigger is
already assigned to the calling triggerer. A read acts as it whenever that
try is still live, because after a user marks a deferred task failed or
successful the trigger row is gone and the triggerer still runs the
trigger's on_kill(). The most a misbehaving trigger can do is write as
another task that is deferred on the same triggerer, or read as any live
task, and it can already read and write every task's XComs by coordinates
through the in-process API. As stated, this is _not_ a security boundary we
can rely on

Requests that fail the check are rejected rather than falling back to the
anonymous identity, deliberately: a trigger whose task instance is no
longer deferred, or whose trigger was reassigned, should not still be
writing its task's XComs, and falling back would only work for tasks
outside loops. Clearing a deferred task archives its try before on_kill()
runs, so on_kill() after a clear cannot read its task's XComs; a trigger
that needs a job id there should keep it on the trigger, as the in-tree
triggers already do.

Only the in-process API that the triggerer creates knows which triggerer
is calling, so every other API rejects the header rather than trusting it.
A triggerer that eventually talks to a real API server will need its own
credential before this can work there.

Only the XCom routes honour the trigger's task instance id for now.
get_previous_task_instance also resolves relative to the caller, so a
trigger asking for its previous task instance still gets the latest pass
rather than the one at its own loop position. Every other request from the
triggerer keeps its current identity, so access rules and the live-attempt
check on those routes are unchanged.

* fixup! Resolve a trigger's XCom requests as its deferred task instance

Pin the HEAD and DELETE rules in the trigger claim check and assert which
check denied each 403, so loosening the read-method test no longer passes.

* fixup! Resolve a trigger's XCom requests as its deferred task instance

Ignore a trigger failure reported by a triggerer that no longer owns the
trigger. The XCom write check now answers 403 to a stale copy on its
previous owner, which raises before yielding and would otherwise have
Trigger.submit_failure send the task to __fail__ while the new owner is
still running it.

(cherry picked from commit 61f1d4b)

Adapted to v3-4-test's CurrentTIToken: the typed execution tokens from the
Dag processor scoped access work (#74370) are not on this branch.

…) (#74868)

* Resolve a trigger's XCom requests as its deferred task instance

Note: _None_ of this is a security boundary, and should not be treated as
such.

The Execution API resolves XCom reads and writes for a task inside a loop
to the caller's own iteration, using the task identity in the caller's JWT.
Triggers have no such identity: the triggerer talks to an in-process API
that treats every request as the all-zero task instance. A trigger's XCom
requests therefore resolve as an anonymous caller rather than as the
deferred task it runs for, so it can neither read (a 400) nor write the
XComs of a task inside a loop.

Triggers can also share one runner coroutine and one comms channel (the
"shared streams" idea), so only the asyncio task running a trigger's code
knows which trigger it is; a context variable carries that trigger's task
instance id to each request frame. Shared-stream groups are started from
whichever trigger subscribes first but serve several triggers, so they start
with it cleared rather than inheriting that trigger's id. This isn't ideal,
but will do "for now".

The trigger runner process (the one running async code) sets the task
instance id, and since that process runs user code a trigger can claim to be
any task instance. The (in-process) API therefore checks the claim. A write
acts as the task instance only when it is deferred and its trigger is
already assigned to the calling triggerer. A read acts as it whenever that
try is still live, because after a user marks a deferred task failed or
successful the trigger row is gone and the triggerer still runs the
trigger's on_kill(). The most a misbehaving trigger can do is write as
another task that is deferred on the same triggerer, or read as any live
task, and it can already read and write every task's XComs by coordinates
through the in-process API. As stated, this is _not_ a security boundary we
can rely on

Requests that fail the check are rejected rather than falling back to the
anonymous identity, deliberately: a trigger whose task instance is no
longer deferred, or whose trigger was reassigned, should not still be
writing its task's XComs, and falling back would only work for tasks
outside loops. Clearing a deferred task archives its try before on_kill()
runs, so on_kill() after a clear cannot read its task's XComs; a trigger
that needs a job id there should keep it on the trigger, as the in-tree
triggers already do.

Only the in-process API that the triggerer creates knows which triggerer
is calling, so every other API rejects the header rather than trusting it.
A triggerer that eventually talks to a real API server will need its own
credential before this can work there.

Only the XCom routes honour the trigger's task instance id for now.
get_previous_task_instance also resolves relative to the caller, so a
trigger asking for its previous task instance still gets the latest pass
rather than the one at its own loop position. Every other request from the
triggerer keeps its current identity, so access rules and the live-attempt
check on those routes are unchanged.

* fixup! Resolve a trigger's XCom requests as its deferred task instance

Pin the HEAD and DELETE rules in the trigger claim check and assert which
check denied each 403, so loosening the read-method test no longer passes.

* fixup! Resolve a trigger's XCom requests as its deferred task instance

Ignore a trigger failure reported by a triggerer that no longer owns the
trigger. The XCom write check now answers 403 to a stale copy on its
previous owner, which raises before yielding and would otherwise have
Trigger.submit_failure send the task to __fail__ while the new owner is
still running it.

(cherry picked from commit 61f1d4b)

Adapted to v3-4-test's CurrentTIToken: the typed execution tokens from the
Dag processor scoped access work (#74370) are not on this branch.

Manual backport: pruned subset of #75689 excluding the #64523 (REST metrics) and #74370 (dag-processor scoped Execution API) pieces that are not on v3-4-test. Keeps the fastapi>=0.142/cadwyn>=8.1 bump + native-OTel-disable + lazy-router adaptations.

(cherry picked from commit b70a439)

…5715)

Manual backport: pruned subset of #75689 excluding the #64523 (REST metrics) and #74370 (dag-processor scoped Execution API) pieces that are not on v3-4-test. Keeps the fastapi>=0.142/cadwyn>=8.1 bump + native-OTel-disable + lazy-router adaptations.

(cherry picked from commit b70a439)