GlueJobOperator in deffered mode does not return error message #64095

Apache Airflow version

main (development)

If "Other Airflow 3 version" selected, which one?

No response

What happened?

We are using on_failure_callback with a custom implemented function that extracts the error message from the context of a failed task and posts it as a card to our Teams channel. exception = context.get('exception')

When a glue job fails while the task is in deferred status it will only pick up that the state has failed and our callback simply extracts "Trigger failure".

This is an issue because in our Teams error notifications we want to immediately be able to see the high level cause of failure. Currently we would need to either go to Glue logs directly or via the Airflow logs.

What you think should happen instead?

The error message from the Glue job should be picked up and passed to the operator, written to logs and available in context for use with callback functions.

How to reproduce

1. Create a Glue job that raises an error message.
2. Create a DAG that triggers above job using GlueJobOperator in deferred mode.
3. Run DAG
4. DAG will fail with error "Trigger failure"

Operating System

MWAA 2.10.3

Versions of Apache Airflow Providers

MWAA 2.10.3

Deployment

Amazon (AWS) MWAA

Deployment details

No response

Anything else?

First raised in #63706

Work already started in #64085

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct