Repository navigation
Conversation
1fanwang
requested review from
XD-DENG,
amoghrajesh,
ashb and
kaxil
as code owners
May 12, 2026 15:44
1fanwang
force-pushed
the
fix/reschedule-start-date-multi-scheduler
branch
from
May 13, 2026 18:55
d89cd36 to
7ac3847
Compare
1 task done
choo121600
added
the
ready for maintainer review
1fanwang
force-pushed
the
fix/reschedule-start-date-multi-scheduler
branch
from
May 16, 2026 18:44
7ac3847 to
c9e4c09
Compare
1 task done
1fanwang
force-pushed
the
fix/reschedule-start-date-multi-scheduler
branch
2 times, most recently
from
June 14, 2026 05:55
1aba313 to
6b3de28
Compare
When a sensor runs in mode="reschedule", the supervisor sends start_date=utcnow() on every poke. The ti_run execution-API endpoint wrote that value through unconditionally, so ti.start_date drifted forward on each re-execution. dagrun.first_task_scheduling_delay (computed from start_date - queued_at) collapsed to ~0 for any DAG fronted by a reschedule-mode sensor. Add a reschedule guard in ti_run that mirrors the existing deferral guard: when start_date is present in the payload and the task has prior TaskReschedule rows, restore start_date from the first row instead of accepting the supervisor's utcnow(). The lookup is scoped by ti_id, and prepare_db_for_next_try clears TaskReschedule rows and rotates ti.id on each retry, so try-number scoping is implicit and no stale rows leak across tries. Also mirror context.start_date in the response so the supervisor pins context["ti"].start_date to the first poke as well. Also drop the unreliable state guard on the same path in _check_and_change_state_before_execution. In the normal multi-scheduler flow the scheduler advances UP_FOR_RESCHEDULE -> QUEUED before the worker calls this method, so refresh_from_db returns QUEUED and the guard never fires. The TaskReschedule lookup is harmless for non-rescheduled tasks (returns no rows), so the guard can go. Closes apache#66784 Co-authored-by: peachchen0716 <[email protected]> Signed-off-by: 1fanwang <[email protected]>
1fanwang
force-pushed
the
fix/reschedule-start-date-multi-scheduler
branch
from
June 19, 2026 18:40
6b3de28 to
9b9dfd4
Compare
Contributor
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. |
github-actions
Bot
added
the
stale
…t-date-multi-scheduler
github-actions
Bot
removed
the
stale
This was referenced Sep 25, 2026
potiuk
added
the
closed because of open PR limit
Member
|
Hello @1fanwang - thank you for your contributions to Apache Airflow! The Airflow community has introduced a limit of 5 open pull requests at a time for contributors without write access to the repository. You currently have 34 open pull requests, so - as a one-time step of introducing the limit - we closed the ones where maintainers have not engaged yet:
These pull requests stay open because maintainers are already engaged in them - they count towards your limit:
This is not a judgement of you or of your changes. We never told contributors before that opening many pull requests at once was a problem, so there is nothing to feel bad about - and nothing is lost: your branches, commits and the review history stay where they are. What we ask you to do is to make your first prioritization decision: choose which of the pull requests above matter most to you, and reopen them (up to 5 open at a time, including the ones still open) with the "Reopen pull request" button or While your pull requests are waiting for review, the most valuable thing you can do is help in other ways - reviewing other contributors' pull requests, helping with issues, and taking part in the discussions on the devlist and Slack. Why we introduced the limit, what it means for you and how to reopen or restore a pull request is explained in https://github.com/apache/airflow/blob/main/contributing-docs/32_open_pull_request_limit.rst. Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting |
This was referenced Sep 25, 2026
This was referenced Sep 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #66784.
A sensor in
mode="reschedule"loses its first-pokestart_dateon every re-execution: each poke writesstart_date=utcnow()through the run path, so the value drifts forward to the last poke.dagrun.first_task_scheduling_delaythen grows by the reschedule wait time, which makes scheduling-delay dashboards too high for any Dag fronted by a reschedule-mode sensor.What
When a task has prior reschedule history, its first-poke
start_dateis restored instead of accepting the per-pokeutcnow(). This applies to both the worker run path and the legacy execution path. Deferred tasks already had an equivalent guard; rescheduled tasks did not. The older reschedule guard never fired because the scheduler advances the task pastUP_FOR_RESCHEDULEbefore the worker runs it.End-to-end repro
Real
airflow standalone(scheduler -> LocalExecutor -> supervisor -> Execution API), with a scheduled Dag fronted by a reschedule-modePythonSensorgated to succeed after 3 reschedules.start_dateand the emitted StatsD metric are read from the live run, not fabricated rows.start_datedriftRaw before / after snapshots
Tests
test_ti_run_restores_start_date_for_rescheduled_tasktest_ti_run_uses_payload_start_date_when_no_reschedule_rowstest_check_and_change_state_before_execution_restores_reschedule_start_datePrior effort
Same fix attempted in #64816 by peachchen0716 (auto-closed after inactivity); credited as co-author.