Skip to content

triggerer: TriggerCommsDecoder.asend() asyncio.Lock is not thread-safe — concurrent sync_to_async calls cause "Response read out of order" and fatal TriggerRunner crash #64620

Closed

Description

@hkc-8010

Apache Airflow Provider(s)

core (triggerer)

Versions of Apache Airflow Providers

Observed in Apache Airflow 3.1.8 (Astronomer Runtime 3.1-14).

Apache Airflow version

3.1.8

Operating System

Debian GNU/Linux 12 (bookworm)

Deployment

Astronomer (managed)

Deployment details

Managed Astronomer deployment with triggerer enabled, using deferrable operators including CloudDataTransferServiceS3ToGCSOperator (Google Cloud Storage Transfer Service) and BigQuery deferrable operators. Single triggerer replica, DEFAULT_CAPACITY=1000, 0.5 vCPU / 1.92 GiB memory.

What happened

The triggerer's TriggerRunner subprocess crashes with RuntimeError: Response read out of order! Got frame.id=N, expect_id=N+1 raised from TriggerCommsDecoder._aget_response. When this exception propagates up through TriggerRunner.arun() → sync_state_to_supervisor(), it kills the entire TriggerRunner subprocess rather than just the individual failing trigger. After this fatal crash, the triggerer pod remains alive (its HTTP server continues serving requests) but the TriggerRunner subprocess does not restart, leaving all deferred tasks stuck indefinitely.

The error was observed repeatedly — at least a dozen times over an 11-hour window — across multiple TriggerRunner subprocess restarts. The final fatal crash propagated to arun() itself and the subprocess did not recover:

2026-04-02T05:43:00.310571Z [error] Trigger runner failed [airflow.jobs.triggerer_job_runner]
RuntimeError: Response read out of order! Got frame.id=18314, expect_id=18315
  File triggerer_job_runner.py:880, in arun
  File triggerer_job_runner.py:1090, in sync_state_to_supervisor
  File triggerer_job_runner.py:1101, in asend
  File triggerer_job_runner.py:801, in asend
  File triggerer_job_runner.py:791, in _aget_response

An earlier instance at 02:37:35 UTC shows the full call chain, originating from a BigQuery trigger calling a synchronous method from within an async context:

RuntimeError: Response read out of order! Got frame.id=25902, expect_id=25903
  task: <Task finished name='kingpd-dimensions/manual__2026-04-01T00:00:00+00:00/d_kingpd_flavour_f_act_sum.insert-step-1/-1/1 (ID 158353)'
        exception=RuntimeError('Response read out of order! Got frame.id=25902, expect_id=25903')>

  Full traceback:
    greenback/_impl.py:116   greenback_shim
    greenback/_impl.py:201   _greenback_shim
    greenback/_impl.py:81    trampoline
    outcome/_impl.py:185     send
    triggerer_job_runner.py:1152  run_trigger
    providers/google/cloud/triggers/bigquery.py:199   run
    providers/google/cloud/triggers/bigquery.py:157   safe_to_cancel
    providers/google/cloud/triggers/bigquery.py:131   get_task_state
    asgiref/sync.py:439      __call__           ← sync_to_async wrapping a sync fn
    greenback/_impl.py:210   _greenback_shim
    concurrent/futures/thread.py:59  run        ← runs in thread pool
    asgiref/sync.py:491      thread_handler
    sdk/execution_time/task_runner.py:514  get_task_states
    triggerer_job_runner.py:772   send           ← TriggerCommsDecoder.send()
    asgiref/sync.py:262      __call__           ← async_to_sync()
    concurrent/futures/_base.py:449  result
    concurrent/futures/_base.py:401  __get_result
    asgiref/sync.py:300      main_wrap
    triggerer_job_runner.py:801   asend
    triggerer_job_runner.py:791   _aget_response ← frame.id mismatch raised here

After the fatal 05:43 crash, the triggerer pod served only HTTP 404 Not Found responses for trigger log requests — confirming no triggers were being executed — for the remainder of the observation window (07:30–07:53+ UTC).

Root Cause Analysis

The bug is a thread-safety violation in TriggerCommsDecoder.asend().

# triggerer_job_runner.py
async def asend(self, msg: ToTriggerSupervisor) -> ToTriggerRunner | None:
    frame = _RequestFrame(id=next(self.id_counter), body=msg.model_dump())
    bytes = frame.as_bytes()

    async with self._async_lock:          # ← asyncio.Lock: only safe within ONE event loop
        self._async_writer.write(bytes)
        return await self._aget_response(frame.id)

TriggerCommsDecoder.send() calls async_to_sync(self.asend)(msg). When async_to_sync is called from a thread (e.g., from asgiref.sync_to_async running a synchronous method of a trigger), it spins up a new asyncio event loop in that thread. The asyncio.Lock (self._async_lock) is bound to a single event loop and provides no mutual exclusion across threads — it only serializes coroutines within the same event loop.

When two concurrent callers invoke TriggerCommsDecoder.send():

  1. Caller A: sync_state_to_supervisor() from the main TriggerRunner.arun() event loop
  2. Caller B: a BigQuery trigger's get_task_state() via sync_to_async → thread → async_to_sync(asend) on a separate event loop

Both callers write their request frames and then await the response for their own frame ID. Because the writes and reads are not mutually exclusive across threads, Caller A reads the response intended for Caller B (frame IDs arrive out of order), raising RuntimeError: Response read out of order! Got frame.id=N, expect_id=N+1.

The specific trigger type that initiates the cross-thread send is any trigger that:

  • calls a synchronous method from its async def run() loop (via asgiref.sync_to_async or greenback)
  • and that synchronous method calls TriggerCommsDecoder.send() (e.g., via task_runner.get_task_states)

In the observed incidents, BigQueryTableExistenceTrigger calling safe_to_cancel → get_task_state → task_runner.get_task_states is the initiating trigger. But the victim can be any trigger running concurrently in the same TriggerRunner subprocess.

What you think should happen instead

TriggerCommsDecoder.asend() should be safe to call from multiple threads simultaneously. The asyncio.Lock should be replaced with a threading.Lock (or a threading.RLock) that provides mutual exclusion across threads, not just within a single event loop. Alternatively, the communication channel could be restructured so that cross-thread sends use a different mechanism (e.g., asyncio.run_coroutine_threadsafe with the parent event loop rather than async_to_sync).

A minimal fix would be:

async def asend(self, msg: ToTriggerSupervisor) -> ToTriggerRunner | None:
    frame = _RequestFrame(id=next(self.id_counter), body=msg.model_dump())
    bytes = frame.as_bytes()

    with self._thread_lock:             # threading.Lock — cross-thread safe
        self._async_writer.write(bytes)
        return await self._aget_response(frame.id)

where self._thread_lock = threading.Lock() is added alongside the existing asyncio.Lock.

Note: self.id_counter (a itertools.count) is also shared across threads; it should be verified as thread-safe (it is in CPython due to the GIL, but worth noting).

How to reproduce

  1. Run a deployment with multiple deferrable operators active concurrently, including at least one trigger type that calls task_runner.get_task_states (or any synchronous SDK method) from inside async def run() via sync_to_async or greenback.
  2. Set AIRFLOW__TRIGGERER__DEFAULT_CAPACITY high (e.g., 1000) to maximize concurrency.
  3. Observe RuntimeError: Response read out of order! in triggerer logs when the race condition fires.
  4. Observe Trigger runner failed with the same error logged from arun when it propagates to the main loop.
  5. After the fatal crash, all deferred tasks remain stuck in DEFERRED state indefinitely; the triggerer pod is alive but serves only 404 Not Found for trigger log requests.

Relationship to existing issues

This is related to but distinct from #64213, which covers a different RuntimeError from the same TriggerCommsDecoder.send() path (Task got Future attached to a different loop). Both issues share the same root cause (thread-unsafe asyncio.Lock in asend), but produce different error messages depending on which async/thread boundary is crossed first.

Anything else

  • The error repeats across multiple TriggerRunner subprocess restarts (observed ~12 times in 11 hours before the fatal crash).
  • The fatal variant (Trigger runner failed propagating through arun) is more severe than non-fatal variants: the TriggerRunner subprocess does not restart, requiring a full triggerer pod restart to recover.
  • AIRFLOW__TRIGGERER__DEFAULT_CAPACITY=1000 on a 0.5 vCPU pod was a contributing factor — higher concurrency increases the probability of the race condition.
  • Confirmed on Airflow 3.1.8 / Python 3.12 / asgiref 3.x / greenback installed.

Activity

  1. phanikumv commented on Apr 2, 2026

    @phanikumv
    Contributor
  2. ashb commented on Apr 2, 2026

    @ashb
    Member

    How reproducible is this? Is there any way we can make it happen more often than "a few times an hour"?

    After the fatal crash, all deferred tasks remain stuck in DEFERRED state indefinitely; the triggerer pod is alive but serves

    That is def a big issue we need to fix.

    The idea behind the async Lock is to try and reuse a single event loop -- it's mean to "bubble" back up to reuse the existing async event loop - we should never be creating more than one async event loop per process.

    Can you show the original un-edited callstack you show here:

      Full traceback:
        greenback/_impl.py:116   greenback_shim
        greenback/_impl.py:201   _greenback_shim
        greenback/_impl.py:81    trampoline
        outcome/_impl.py:185     send
    

    This isn't a python native stack trace (the formatting is slightly off)

  3. tirkarthi commented on Apr 3, 2026

    @tirkarthi
    Contributor

    We also started seeing similar failures in production. One of our environments has 400-600 triggers running at a time. We started noticing this in ExternalTaskSensor in deferred mode with default poll_interval of 2 seconds. The tasks used to run for days waiting for the external task to finish. After Airflow 3 upgrade this runs for few hours like 3 to 7 hours and fails randomly. Full traceback as below. For now we recommended increasing the poll_interval to 300 seconds or set deferrable=False since these are long running jobs to see if it helps.

    We also have other environment in production where the same sensor runs but the running triggers are around 10-20 at a time. We noticed this occurring once or twice maybe there. So this could be due to high load and something our non-production didn't catch due to lack of load in non-production.

    Airflow version : 3.1.5
    apache-airflow-providers-standard : 1.10.0

    [2026-04-02 21:33:12] ERROR - Trigger failed:
    Traceback (most recent call last):
      File "/opt/conda/envs/airflow_app/lib/python3.12/site-packages/airflow/jobs/triggerer_job_runner.py", line 999, in cleanup_finished_triggers
        result = details["task"].result()
                 ^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/conda/envs/airflow_app/lib/python3.12/site-packages/greenback/_impl.py", line 116, in greenback_shim
        return await _greenback_shim(orig_coro, next_send)  # type: ignore
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/conda/envs/airflow_app/lib/python3.12/site-packages/greenback/_impl.py", line 201, in _greenback_shim
        next_yield, resume_greenlet = resume_greenlet.switch(next_send)
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/conda/envs/airflow_app/lib/python3.12/site-packages/greenback/_impl.py", line 81, in trampoline
        next_yield: Any = next_send.send(orig_coro)  # type: ignore
                          ^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/conda/envs/airflow_app/lib/python3.12/site-packages/outcome/_impl.py", line 185, in send
        return gen.send(self.value)
               ^^^^^^^^^^^^^^^^^^^^
      File "/opt/conda/envs/airflow_app/lib/python3.12/site-packages/airflow/jobs/triggerer_job_runner.py", line 1113, in run_trigger
        async for event in trigger.run():
      File "/opt/conda/envs/airflow_app/lib/python3.12/site-packages/airflow/providers/standard/triggers/external_task.py", line 125, in run
        allowed_count = await get_count_func(self.allowed_states)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/conda/envs/airflow_app/lib/python3.12/site-packages/airflow/providers/standard/triggers/external_task.py", line 138, in _get_count_af_3
        count = await sync_to_async(RuntimeTaskInstance.get_ti_count)(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/conda/envs/airflow_app/lib/python3.12/site-packages/asgiref/sync.py", line 504, in __call__
        ret = await asyncio.shield(exec_coro)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/conda/envs/airflow_app/lib/python3.12/site-packages/greenback/_impl.py", line 210, in _greenback_shim
        next_send = outcome.Value((yield next_yield))
                                   ^^^^^^^^^^^^^^^^
      File "/opt/conda/envs/airflow_app/lib/python3.12/concurrent/futures/thread.py", line 58, in run
        result = self.fn(*self.args, **self.kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/conda/envs/airflow_app/lib/python3.12/site-packages/asgiref/sync.py", line 559, in thread_handler
        return func(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^
      File "/opt/conda/envs/airflow_app/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 502, in get_ti_count
        response = SUPERVISOR_COMMS.send(
                   ^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/conda/envs/airflow_app/lib/python3.12/site-packages/airflow/jobs/triggerer_job_runner.py", line 770, in send
        return async_to_sync(self.asend)(msg)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/conda/envs/airflow_app/lib/python3.12/site-packages/asgiref/sync.py", line 325, in __call__
        return call_result.result()
               ^^^^^^^^^^^^^^^^^^^^
      File "/opt/conda/envs/airflow_app/lib/python3.12/concurrent/futures/_base.py", line 449, in result
        return self.__get_result()
               ^^^^^^^^^^^^^^^^^^^
      File "/opt/conda/envs/airflow_app/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
        raise self._exception
      File "/opt/conda/envs/airflow_app/lib/python3.12/site-packages/asgiref/sync.py", line 365, in main_wrap
        result = await awaitable
                 ^^^^^^^^^^^^^^^
      File "/opt/conda/envs/airflow_app/lib/python3.12/site-packages/airflow/jobs/triggerer_job_runner.py", line 799, in asend
        return await self._aget_response(frame.id)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/conda/envs/airflow_app/lib/python3.12/site-packages/airflow/jobs/triggerer_job_runner.py", line 789, in _aget_response
        raise RuntimeError(f"Response read out of order! Got {frame.id=}, {expect_id=}")
    
    [2026-04-02 21:33:12] ERROR - Task failed with exception
    TaskDeferralError: Trigger failure
    File "/opt/conda/envs/airflow_app/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 953 in run
    File "/opt/conda/envs/airflow_app/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1344 in _execute_task
    File "/opt/conda/envs/airflow_app/lib/python3.12/site-packages/airflow/sdk/bases/sensor.py", line 255 in resume_execution
    File "/opt/conda/envs/airflow_app/lib/python3.12/site-packages/airflow/sdk/bases/operator.py", line 1630 in resume_execution
    
  4. added 2 commits that reference this issue on Apr 7, 2026
    810fc59
    3fc0dc7
  5. kalluripradeep commented on Apr 7, 2026

    @kalluripradeep
    Contributor

    I’ve been digging into this and it looks like the asyncio.Lock in TriggerCommsDecoder.asend() is definitely the root cause. This method gets called across different threads and loops (via sync_to_async), but because asyncio.Lock is bound to a single event loop, it wasn't actually protecting the socket access across threads, leading to those out-of-order response crashes.

    I've pushed a PR that swaps this out for a threading.Lock acquired asynchronously. This ensures that only one thread can use the socket at a time, regardless of which event loop it's running in. I also added a drain() call to make sure messages are fully sent before we wait for the supervisor's response.

  6. added 3 commits that reference this issue on Apr 7, 2026
    0ee6fa9
    6e9dd34
    02bc550
  7. added 8 commits that reference this issue on Apr 8, 2026
    867f4ad
    840c212
    9d8fdbd
    e881d35
    2981839
    cd75817
    647ba3f
    9281794
  8. andrewhharmon commented on Apr 9, 2026

    @andrewhharmon
    Contributor

    we just encountered this yesterday as well. I can't easily recreate. we are running Astro Runtime. 3.1-12 (based on Airflow v3.1.7). Let me know if there's any additional information I could provide to help. Thanks

  9. uranusjr commented on Apr 13, 2026

    @uranusjr
    Member

    @kalluripradeep I don’t think we want to swap the call out to threading.Lock; this would make all async calls blocking even if they are used in the same event loop. Instead, I think we should only lock in the sync calls. Reading what CommsDecoder (the equivalent used in task workers) does, I thought it’d be possible to add the threading lock on top of the asyncio lock, but when I did that (#64882), it’s reported the triggerer would enter a state where it no longer takes any triggers. I think that needs fixing too.

  10. added a commit that references this issue on Apr 30, 2026
    e9b37e9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions