Repository navigation
Conversation
Closes apache#72387 Currently, when a DAG reaches its max_active_runs limit, the scheduler sets DagModel.exceeds_max_non_backfill without logging anything, so operators have no way to tell which DagRun/TaskInstance is occupying the active slots from the scheduler logs alone. This adds a targeted, one-time log in SchedulerJobRunner._set_exceeds_max_active_runs: the moment a DAG newly reaches (or exceeds) max_active_runs, it queries the blocking run_ids and the task_ids/states holding them, and logs that context. The query only runs on the False->True transition, so there is no added database load during normal, healthy scheduling loops, and no repeated log spam while the DAG stays stuck. Adds two unit tests covering the log content and the no-repeat-log behavior on a subsequent call. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01VxiX5rvn9qaori5ghyBg6D
potiuk
added
the
closed because of open PR limit
|
Hello @Kunal8954 - thank you for your contributions to Apache Airflow! The Airflow community has introduced a limit of 5 open pull requests at a time for contributors without write access to the repository. You currently have 8 open pull requests, so - as a one-time step of introducing the limit - we closed the ones where maintainers have not engaged yet:
These pull requests stay open because maintainers are already engaged in them - they count towards your limit:
This is not a judgement of you or of your changes. We never told contributors before that opening many pull requests at once was a problem, so there is nothing to feel bad about - and nothing is lost: your branches, commits and the review history stay where they are. What we ask you to do is to make your first prioritization decision: choose which of the pull requests above matter most to you, and reopen them (up to 5 open at a time, including the ones still open) with the "Reopen pull request" button or While your pull requests are waiting for review, the most valuable thing you can do is help in other ways - reviewing other contributors' pull requests, helping with issues, and taking part in the discussions on the devlist and Slack. Why we introduced the limit, what it means for you and how to reopen or restore a pull request is explained in https://github.com/apache/airflow/blob/main/contributing-docs/32_open_pull_request_limit.rst. Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting |
The promotion queries (get_queued_dag_runs_to_set_running, the scheduler's _start_queued_dagruns) both count running runs per (dag_id, backfill_id), so a running backfill run doesn't hold back a manual trigger's own max_active_runs slot. This check was counting every running run regardless of backfill_id, which could log a false "will not be scheduled yet" for a manual trigger that would actually be promoted on the next scheduler loop. Also corrected the docstring: _set_exceeds_max_active_runs doesn't log anything on main -- that one-shot-logging behavior was from apache#72403, which was closed without merging, not something actually in the codebase to avoid duplicating. Consolidated the three single-case tests into one parametrized test covering the fix above, and switched the negative assertions to the membership form instead of iterating caplog.records, which is marked for removal. Co-Authored-By: Claude <[email protected]>
The promotion queries (get_queued_dag_runs_to_set_running, the scheduler's _start_queued_dagruns) both count running runs per (dag_id, backfill_id), so a running backfill run doesn't hold back a manual trigger's own max_active_runs slot. This check was counting every running run regardless of backfill_id, which could log a false "will not be scheduled yet" for a manual trigger that would actually be promoted on the next scheduler loop. Also corrected the docstring: _set_exceeds_max_active_runs doesn't log anything on main -- that one-shot-logging behavior was from apache#72403, which was closed without merging, not something actually in the codebase to avoid duplicating. Consolidated the three single-case tests into one parametrized test covering the fix above, and switched the negative assertions to the membership form instead of iterating caplog.records, which is marked for removal. Co-Authored-By: Claude <[email protected]>
Closes #72387
Currently, when a DAG reaches its max_active_runs limit, the scheduler
sets DagModel.exceeds_max_non_backfill without logging anything, so
operators have no way to tell which DagRun/TaskInstance is occupying
the active slots from the scheduler logs alone.
This adds a targeted, one-time log in
SchedulerJobRunner._set_exceeds_max_active_runs: the moment a DAG
newly reaches (or exceeds) max_active_runs, it queries the blocking
run_ids and the task_ids/states holding them, and logs that context.
The query only runs on the False->True transition, so there is no
added database load during normal, healthy scheduling loops, and no
repeated log spam while the DAG stays stuck.
Adds two unit tests covering the log content and the no-repeat-log
behavior on a subsequent call.