ExecuteTask activity token can expire before the task starts running #53713

2025-07-23 01:27:58,629 {_client.py:1025} INFO - HTTP Request: PATCH http://127.0.0.1:8085/airflow/execution/task-instances/019835de-0967-7bc4-b053-e6663b1a45ec/state "HTTP/1.1 204 No Content"
2025-07-23 01:27:58 [info     ] Task finished                  [supervisor] duration=1076.4064389939886 exit_code=0 final_state=success
2025-07-23 01:27:58 [info     ] Secrets backends loaded for worker [supervisor] backend_classes=['SecretsManagerBackend', 'EnvironmentVariablesBackend'] count=2
2025-07-23 01:27:58,919 {_client.py:1025} INFO - HTTP Request: PATCH http://127.0.0.1:8085/airflow/execution/task-instances/019835de-0955-74d4-8db6-2aea4332ab1b/run "HTTP/1.1 403 Forbidden"
2025-07-23 01:27:58 [warning  ] Server error                   [airflow.sdk.api.client] detail=None
2025-07-23 01:27:58 [info     ] Process exited                 [supervisor] exit_code=<Negsignal.SIGKILL: -9> pid=1258981 signal_sent=SIGKILL
2025-07-23 01:27:58,929 {local_executor.py:96} ERROR - uhoh
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/executors/local_executor.py", line 92, in _run_worker
    _execute_work(log, workload)
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/executors/local_executor.py", line 120, in _execute_work
    supervise(
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1683, in supervise
    process = ActivitySubprocess.start(
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 851, in start
    proc._on_child_started(ti=what, dag_rel_path=dag_rel_path, bundle_info=bundle_info)
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 862, in _on_child_started
    ti_context = self.client.task_instances.start(ti.id, self.pid, start_date)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 149, in start
    resp = self.client.patch(f"task-instances/{id}/run", content=body.model_dump_json())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 1218, in patch
    return self.request(
           ^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 336, in wrapped_f
    return copy(f, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 475, in __call__
    do = self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 376, in iter
    result = action(retry_state)
             ^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 398, in <lambda>
    self._add_action_func(lambda rs: rs.outcome.result())
                                     ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 478, in __call__
    result = fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 707, in request
    return super().request(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 825, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 914, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 942, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 999, in _send_handling_redirects
    raise exc
  File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 982, in _send_handling_redirects
    hook(response)
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 120, in raise_on_4xx_5xx_with_note
    return get_json_error(response) or response.raise_for_status()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 110, in get_json_error
    raise err
airflow.sdk.api.client.ServerResponseError: Invalid auth token: Signature has expired
Correlation-id=019835f7-a470-78d0-b268-b388c323926f