Repository navigation
GlueJobOperator - resume_glue_job_on_retry doesn't seem to work on MWAA 2.11.0 #62353
Description
Apache Airflow version
2.11.X
If "Other Airflow 3 version" selected, which one?
MWAA 2.11.0
What happened?
Hi,
I tried to run the GlueJobOperator with resume_glue_job_on_retry=True but somehow during retry when the task failed, it creates a new glue job ID while the existing glue job ID is still running.
Sample of the GlueJobOperator:
drug_program = GlueJobOperator(
task_id="drug_program",
job_name="gtm-core-dl-dev-euc1-glue-job-glo_informa_prd",
verbose=False,
script_args={
"--name": var_glue_job_name,
"--db_name": var_database_name,
"--tbl_name": "drug_program",
},
stop_job_run_on_kill=False,
deferrable=False,
pool=var_glue_pool,
botocore_config=botocore_overide_config,
resume_glue_job_on_retry=True,
)1st Run:
Create a new glue job ID as per screenshot
2nd Run (1st Retry):
Job failed due to internal errors, during retry, it didn't find back the existing glue job ID but created 1 new glue job ID. End results, 2 same glue jobs are running together.
What you think should happen instead?
During task retry, it should check the existing glue job ID, if found, it should not create a new glue job ID and resume run status in Airflow Task
How to reproduce
- Install the apache-airflow-providers-amazon==9.22.0rc1 in MWAA Airflow 2.11 environment
2.. Create GlueJobOperator Dag
Sample of the GlueJobOperator:
drug_program = GlueJobOperator(
task_id="drug_program",
job_name="gtm-core-dl-dev-euc1-glue-job-glo_informa_prd",
verbose=False,
script_args={
"--name": var_glue_job_name,
"--db_name": var_database_name,
"--tbl_name": "drug_program",
},
stop_job_run_on_kill=False,
deferrable=False,
pool=var_glue_pool,
botocore_config=botocore_overide_config,
resume_glue_job_on_retry=True,
)- Run the glue job task.
- During 1st run, it failed (to test, produce by marking the task as failed)
- Task went to for retry but somehow it creates a new glue job ID
Operating System
Amazon Linux 2023
Versions of Apache Airflow Providers
apache-airflow-providers-amazon==9.22.0rc1
Deployment
Amazon (AWS) MWAA
Deployment details
MWAA 2.11.0 deployment with requirements to install apache-airflow-providers-amazon==9.22.0rc1
Anything else?
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
wilsonhooi86 commentedon Feb 23, 2026
Hi @henry3260 , just to share this is related to the recent feature: #59392
Thank you so much and let me know if you need more info
henry3260 commentedon Feb 23, 2026
Ｉwill take a look. Thanks for sharing!
wilsonhooi86 commentedon Feb 24, 2026
Thank you so much, @henry3260 .
This feature will be very helpful for our current use cases. Really appreciate your support.
henry3260 commentedon Feb 25, 2026
Can I ask Did the first run fail before the XCom push?
shivaam commentedon Feb 26, 2026
@henry3260, I see you're looking into this. I have some bandwidth this weekend and would love to help fix this bug if you haven't started on a PR yet. Would you like me to take a stab at a reproduction case or the fix itself?
wilsonhooi86 commentedon Feb 27, 2026
Hi @henry3260 , I not too sure on that. It was hard for me to test by having an internal failure that caused the GlueJobOperator's task to fail on its own.
So, to simulate task's retry, what I did was below steps:
FAILED