Repository navigation
AIRFLOW__SCHEDULER__TASK_QUEUED_TIMEOUT configuration ignored #59553
Description
Apache Airflow version
3.1.5
If "Other Airflow 3 version" selected, which one?
No response
What happened?
For my workload, I ingest video files and then create tasks to process each one. It is normal to create 30+ tasks that will get processed in the span of 40+min. On backfill, it's absolutely normal for my GPU workers to not be available and for a task to wait 1h+ to be executed.
This is why I changed AIRFLOW__SCHEDULER__TASK_QUEUED_TIMEOUT to 10 000 (see image below).
I also changed my task definition to allow a task to be in processing for 24h, so this is not the issue either.
However, even when setting that, my tasks fail and the retry mechanism kicks in. I've bandaged fixed it by increasing the number of retries, but it's not optimal.
What you think should happen instead?
AIRFLOW__SCHEDULER__TASK_QUEUED_TIMEOUT should be taken into account, and tasks should not timeout on around 15min.
All tasks eventually complete. With 20 runs × 2.5 min each and 1 worker, the last task should wait ~50 min in the queue - well within the 100,000s timeout.
How to reproduce
1. Install Airflow
uv pip install "apache-airflow[celery]==3.1.5" \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.1.5/constraints-3.12.txt"2. Initialize and configure
airflow db migrate
sed -i 's/^executor = .*/executor = CeleryExecutor/' airflow.cfg
sed -i 's/^task_queued_timeout = .*/task_queued_timeout = 100000.0/' airflow.cfg3. Create the DAG
Save as dags/long_running_task_dag.py:
from datetime import datetime, timedelta
from time import sleep
from airflow.sdk import dag, task
@dag(
dag_id="long_running_task_bug_repro",
schedule=None,
start_date=datetime(2025, 1, 1),
catchup=False,
default_args={
"owner": "airflow",
"retries": 0,
},
)
def long_running_task_bug_repro():
@task(
queue="gpu_queue",
execution_timeout=timedelta(hours=24),
)
def gpu_simulation():
"""Simulate GPU processing - sleeps for 2.5 minutes."""
sleep(150)
gpu_simulation()
long_running_task_bug_repro()4. Start services
Terminal 1 - Scheduler:
airflow api-server --port 8080 &
airflow scheduler &
airflow dag-processor &
airflow triggerer &
airflow celery worker --queues defaultTerminal 2 - Single worker (simulates limited GPU resources):
airflow celery worker --queues gpu_queue --concurrency 15. Trigger multiple DAG runs
for i in $(seq 1 20); do
airflow dags trigger long_running_task_bug_repro
done6. Wait ~15 minutes
Operating System
Ubuntu LTS 22.04
Versions of Apache Airflow Providers
apache-airflow-providers-celery==3.14.0
apache-airflow-providers-common-compat==1.10.0
apache-airflow-providers-common-io==1.7.0
apache-airflow-providers-common-sql==1.30.0
apache-airflow-providers-smtp==2.4.0
apache-airflow-providers-standard==1.10.0
Deployment
Virtualenv installation
Deployment details
No response
Anything else?
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
boring-cyborg commentedon Dec 17, 2025
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
sania-2912 commentedon Dec 22, 2025
Hi @potiuk,
I’ve gone through the reproduction steps and can consistently reproduce the issue on Airflow 3.1.5 with CeleryExecutor.
From my view, it looks like task_queued_timeout is not being honored correctly for Celery tasks waiting in a non-default queue with limited worker concurrency, leading to premature retries (~15 min) despite a much higher configured timeout.
I’d like to work on a fix for this and submit a PR. My plan is to:
Please let me know if there are any architectural considerations or preferred areas to start with. Happy to proceed.
Thanks!
potiuk commentedon Dec 29, 2025
Proceed. The discussion is better when there is PR and I am not necessarily the best person to comment but when you open PR it will likely drag attention of those who can take a look (just please don't tag me as it might be someone else to look at it )
Add token refresh mechanism for Execution API (apache#59553)
Add token refresh mechanism for Execution API (apache#59553)
anishgirianish commentedon Jan 4, 2026
I've opened a PR to fix this issue: #60108
The fix adds a token refresh mechanism that detects expired tokens before task execution and refreshes them via a new
/execution/token/refreshendpoint.Would appreciate a review when you get a chance. Thanks!
4 remaining items
apache#59553 (fix): token expiry in long running task
apache#59553 (fix): revert unecessary changes
apache#59553 (fix): fix failing test
apache#59553 (fix): token expiry in long running task
apache#59553 (fix): revert unecessary changes
apache#59553 (fix): fix failing test
Two-token mechanism for task execution to prevent token expiration wh…
Two-token mechanism for task execution to prevent token expiration wh…
Two-token mechanism for task execution to prevent token expiration wh…
Two-token mechanism for task execution to prevent token expiration wh…
Two-token mechanism for task execution to prevent token expiration wh…
Two-token mechanism for task execution to prevent token expiration wh…
Two-token mechanism for task execution to prevent token expiration wh…