Correlate executor task events by attempt UUID - #73916

…g` list

Since #73916 landed, BaseExecutor now keeps a UUID-to-coordinates map for each
attempt and drops an entry once the key is no longer queued, running or in the
event buffer. The issue was that LocalExecutor never added dispatched work to
`running`, so the entry was dropped while the task was still executing. The
scheduler then received the final event with no coordinates ("Received executor
event with state success for task instance <uuid> (coordinates=None)").

This PR fixes that, and addresses a few gotchas that could, in edge cases
around callbacks or post-task finalization, lead to a forever-dead/locked
worker slot.

Holding the key in `running` until the worker finishes needs LocalExecutor
to know when that is, and workers share one activity queue so the parent
cannot tell which worker took which workload. Results now carry the
worker's pid and START tells the parent which pid owns the key. That lets
it:

- release the slot only on a terminal result from the owning worker,
ignoring results from unknown pids or pids that don't own the key
- fail the workload when its worker dies, instead of leaving it in
`running` until the scheduler's heartbeat timeout

A resumed task reuses its key, so a per-key dispatch count stops the old
run's terminal result from releasing the new run's slot.

…g` list

Since #73916 landed, BaseExecutor now keeps a UUID-to-coordinates map for each
attempt and drops an entry once the key is no longer queued, running or in the
event buffer. The issue was that LocalExecutor never added dispatched work to
`running`, so the entry was dropped while the task was still executing. The
scheduler then received the final event with no coordinates ("Received executor
event with state success for task instance <uuid> (coordinates=None)").

This PR fixes that, and addresses a few gotchas that could, in edge cases
around callbacks or post-task finalization, lead to a forever-dead/locked
worker slot.

Holding the key in `running` until the worker finishes needs LocalExecutor
to know when that is, and workers share one activity queue so the parent
cannot tell which worker took which workload. Results now carry the
worker's pid and START tells the parent which pid owns the key. That lets
it:

- release the slot only on a terminal result from the owning worker,
ignoring results from unknown pids or pids that don't own the key
- fail the workload when its worker dies, instead of leaving it in
`running` until the scheduler's heartbeat timeout

A resumed task reuses its key, so a per-key dispatch count stops the old
run's terminal result from releasing the new run's slot.

…g` list

Since apache#73916 landed, BaseExecutor now keeps a UUID-to-coordinates map for each
attempt and drops an entry once the key is no longer queued, running or in the
event buffer. The issue was that LocalExecutor never added dispatched work to
`running`, so the entry was dropped while the task was still executing. The
scheduler then received the final event with no coordinates ("Received executor
event with state success for task instance <uuid> (coordinates=None)").

This PR fixes that, and addresses a few gotchas that could, in edge cases
around callbacks or post-task finalization, lead to a forever-dead/locked
worker slot.

Holding the key in `running` until the worker finishes needs LocalExecutor
to know when that is, and workers share one activity queue so the parent
cannot tell which worker took which workload. Results now carry the
worker's pid and START tells the parent which pid owns the key. That lets
it:

- release the slot only on a terminal result from the owning worker,
ignoring results from unknown pids or pids that don't own the key
- fail the workload when its worker dies, instead of leaving it in
`running` until the scheduler's heartbeat timeout

A resumed task reuses its key, so a per-key dispatch count stops the old
run's terminal result from releasing the new run's slot.

…g` list (#74142)

* Improve LocalExecutor bookkeeping: correctly add tasks to the `running` list

Since #73916 landed, BaseExecutor now keeps a UUID-to-coordinates map for each
attempt and drops an entry once the key is no longer queued, running or in the
event buffer. The issue was that LocalExecutor never added dispatched work to
`running`, so the entry was dropped while the task was still executing. The
scheduler then received the final event with no coordinates ("Received executor
event with state success for task instance <uuid> (coordinates=None)").

This PR fixes that, and addresses a few gotchas that could, in edge cases
around callbacks or post-task finalization, lead to a forever-dead/locked
worker slot.

Holding the key in `running` until the worker finishes needs LocalExecutor
to know when that is, and workers share one activity queue so the parent
cannot tell which worker took which workload. Results now carry the
worker's pid and START tells the parent which pid owns the key. That lets
it:

- release the slot only on a terminal result from the owning worker,
ignoring results from unknown pids or pids that don't own the key
- fail the workload when its worker dies, instead of leaving it in
`running` until the scheduler's heartbeat timeout

A resumed task reuses its key, so a per-key dispatch count stops the old
run's terminal result from releasing the new run's slot.

* fixup! Improve LocalExecutor bookkeeping: correctly add tasks to the `running` list

* fixup! Improve LocalExecutor bookkeeping: correctly add tasks to the `running` list