Skip to content

Report failure causes and bound infrastructure replacements - #66405

Closed
1fanwang wants to merge 2 commits into
apache:mainfrom
1fanwang:1fanwang/aip97-failure-details
Closed

Report failure causes and bound infrastructure replacements#66405
1fanwang wants to merge 2 commits into
apache:mainfrom
1fanwang:1fanwang/aip97-failure-details

Conversation

@1fanwang

@1fanwang 1fanwang commented May 5, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

An infrastructure interruption can stop healthy task code and consume an ordinary retry. This combined prototype reports failure causes to listeners and metrics, and lets administrators grant bounded replacement attempts for confirmed infrastructure failures.

[core] max_infra_retries defaults to 0; Dag authors get no new parameter. The policy reuses existing retry fields without a migration or an AIP-103 counter. Ordinary attempts and clears can exhaust the allowance early. Raising ordinary retries can reopen an early refusal, but not once the prior-attempt count reaches the cap. Unclassified failures do not qualify.

This implements the combined reference for AIP-97 classification and AIP-116 bounded infrastructure retries. Classification is independently reviewable in #73350.

Testing Done

Real Kubernetes Task SDK workers made authenticated starts and later heartbeats before interruption. PostgreSQL snapshots, listener records, callbacks, and UDP metrics showed:

Scenario Default cap 0 Cap 1
Confirmed taint-manager interruption, retries=0 Failed on attempt 1 Succeeded on attempt 2; one grant
Workload storage-limit eviction Failed, unclassified Same; no grant
Application, timeout, and manual controls Failed without replacement Same
Ordinary retry followed by a clear Attempts 1, 2, 3; callbacks retry, success, success Same

The runnable workflow, complete fixture, provenance, and sanitized evidence include setup. These commands ran on Linux:

SOURCE_SHA=77b67ade969b160f7550437e6b5108926d3ecc15
flags=(--policy)
python fixture/run.py --source "$GITHUB_WORKSPACE/source" --commit "$SOURCE_SHA" "${flags[@]}"
runs=(fixture/runs/*)
python fixture/verify_evidence.py "${runs[0]}" --candidate "$SOURCE_SHA" "${flags[@]}"
Raw runtime observations 
{"utc": "2026-09-22T18:10:19.136083+00:00", "origin": "scheduler", "dag_id": "aip97_classification_infra", "task_id": "probe", "run_id": "manual__infra_88043252", "try_number": 1, "kind": "infra", "reason": "DeletionByTaintManager"}
aip97.ti_infra_retry_granted:1|c|#dag_id:aip97_classification_infra,run_type:manual,task_id:probe|c:in-28585
{"utc": "2026-09-22T18:10:39.039483+00:00", "event": "CASE_FINISHED", "case": "infra", "final": {"id": "01a0ca4f-bf7f-76c1-980f-af503b623d78", "dag_id": "aip97_classification_infra", "task_id": "probe", "run_id": "manual__infra_88043252", "state": "success", "try_number": 2, "max_tries": 1, "queued_by_job_id": 2, "start_date": "2026-09-22 18:10:26.996839+00:00", "end_date": "2026-09-22 18:10:28.877524+00:00", "last_heartbeat_at": "2026-09-22 18:10:28.630957+00:00", "pid": 15}, "pre_clear": null}

The receipt identifies the exact candidate SHA used by the runtime processes. The selected lines above are unchanged; workspace-normalized linked output is labeled sanitized.

The run used kind and PostgreSQL. It observed forced termination under a 30-second grace configuration, not cooperative SIGTERM completion, and did not exercise Celery or AWS.

Was generative AI tooling used to co-author this PR?
  • Yes: GitHub Copilot CLI 1.0.80 (GPT-5.6 Sol)

Generated-by: GitHub Copilot CLI 1.0.80 (GPT-5.6 Sol) following the guidelines

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@1fanwang
1fanwang force-pushed the 1fanwang/aip97-failure-details branch from 20d6bd9 to 8c53810 Compare May 5, 2026 10:21
This was referenced May 5, 2026
Closed
Closed
Closed
Closed
Closed
Closed
@seanmuth seanmuth mentioned this pull request Jun 26, 2026
Closed
@1fanwang
1fanwang force-pushed the 1fanwang/aip97-failure-details branch from 57f338e to fcc393a Compare July 16, 2026 05:33
@1fanwang 1fanwang closed this Jul 16, 2026
@1fanwang 1fanwang reopened this Jul 16, 2026
@1fanwang
1fanwang force-pushed the 1fanwang/aip97-failure-details branch from fcc393a to bd8e7ec Compare July 24, 2026 19:28
@1fanwang 1fanwang changed the title AIP-97 (draft): add FailureDetails primitive for infrastructure-side failure context AIP-97 (draft): failure context propagation and transparent infra retry Jul 24, 2026
@1fanwang
1fanwang force-pushed the 1fanwang/aip97-failure-details branch from bd8e7ec to e87f0ba Compare July 24, 2026 19:38
@1fanwang 1fanwang mentioned this pull request Jul 24, 2026
Closed
@1fanwang
1fanwang force-pushed the 1fanwang/aip97-failure-details branch from e87f0ba to 7d568c9 Compare July 24, 2026 22:39
This was referenced Jul 24, 2026
Closed
Closed
Closed
@1fanwang 1fanwang changed the title AIP-97 (draft): failure context propagation and transparent infra retry AIP-97 (draft): Disruption Readiness (Failure Context Propagation and Transparent Infra Retries) Jul 27, 2026
@1fanwang
1fanwang force-pushed the 1fanwang/aip97-failure-details branch 2 times, most recently from ab413e1 to c822f45 Compare July 27, 2026 19:25
This was referenced Jul 27, 2026
Closed
Closed
@1fanwang
1fanwang force-pushed the 1fanwang/aip97-failure-details branch 2 times, most recently from aa82a98 to 0de4133 Compare July 28, 2026 01:13
@1fanwang 1fanwang closed this Jul 28, 2026
@1fanwang 1fanwang reopened this Jul 28, 2026
@1fanwang
1fanwang force-pushed the 1fanwang/aip97-failure-details branch from 0de4133 to 4a545fd Compare July 28, 2026 07:45
@1fanwang
1fanwang force-pushed the 1fanwang/aip97-failure-details branch from e08f021 to 77b67ad Compare September 22, 2026 18:50
@potiuk potiuk added the closed because of open PR limit Closed as a one-time step of introducing the open pull request limit label Sep 25, 2026
@potiuk

potiuk commented Sep 25, 2026

Copy link
Copy Markdown
Member

Hello @1fanwang - thank you for your contributions to Apache Airflow!

The Airflow community has introduced a limit of 5 open pull requests at a time for contributors without write access to the repository. You currently have 34 open pull requests, so - as a one-time step of introducing the limit - we closed the ones where maintainers have not engaged yet:

These pull requests stay open because maintainers are already engaged in them - they count towards your limit:

This is not a judgement of you or of your changes. We never told contributors before that opening many pull requests at once was a problem, so there is nothing to feel bad about - and nothing is lost: your branches, commits and the review history stay where they are.

What we ask you to do is to make your first prioritization decision: choose which of the pull requests above matter most to you, and reopen them (up to 5 open at a time, including the ones still open) with the "Reopen pull request" button or gh pr reopen <PR_NUMBER> --repo apache/airflow. Reopen the ones you are ready to follow through - keep them rebased, respond to review comments and fix failing checks.

While your pull requests are waiting for review, the most valuable thing you can do is help in other ways - reviewing other contributors' pull requests, helping with issues, and taking part in the discussions on the devlist and Slack.

Why we introduced the limit, what it means for you and how to reopen or restore a pull request is explained in https://github.com/apache/airflow/blob/main/contributing-docs/32_open_pull_request_limit.rst.

Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

@potiuk potiuk closed this Sep 25, 2026
This was referenced Sep 25, 2026
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ashb ashb Awaiting requested review from ashb ashb is a code owner
@amoghrajesh amoghrajesh Awaiting requested review from amoghrajesh amoghrajesh is a code owner
@potiuk potiuk Awaiting requested review from potiuk potiuk is a code owner
@kaxil kaxil Awaiting requested review from kaxil
@jedcunningham jedcunningham Awaiting requested review from jedcunningham jedcunningham is a code owner
@hussein-awala hussein-awala Awaiting requested review from hussein-awala hussein-awala is a code owner
@jscheffl jscheffl Awaiting requested review from jscheffl jscheffl is a code owner
@ephraimbuddy ephraimbuddy Awaiting requested review from ephraimbuddy ephraimbuddy is a code owner
@pierrejeambrun pierrejeambrun Awaiting requested review from pierrejeambrun pierrejeambrun is a code owner
@rawwar rawwar Awaiting requested review from rawwar
@jason810496 jason810496 Awaiting requested review from jason810496 jason810496 is a code owner
@bugraoz93 bugraoz93 Awaiting requested review from bugraoz93 bugraoz93 is a code owner
@shubhamraj-git shubhamraj-git Awaiting requested review from shubhamraj-git shubhamraj-git is a code owner
@choo121600 choo121600 Awaiting requested review from choo121600 choo121600 is a code owner
@henry3260 henry3260 Awaiting requested review from henry3260 henry3260 is a code owner
@XD-DENG XD-DENG Awaiting requested review from XD-DENG XD-DENG is a code owner
@o-nikolas o-nikolas Awaiting requested review from o-nikolas o-nikolas is a code owner
@dheerajturaga dheerajturaga Awaiting requested review from dheerajturaga dheerajturaga is a code owner
@shahar1 shahar1 Awaiting requested review from shahar1 shahar1 will be requested when the pull request is marked ready for review shahar1 is a code owner

Assignees

@1fanwang 1fanwang

Labels

closed because of open PR limit Closed as a one-time step of introducing the open pull request limit

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@1fanwang @potiuk