Repository navigation
Conversation
5c281c2 to
94fc568
Compare
Metadata recovery should revisit only the files whose Dags are missing and make progress available one file at a time. Restore surviving unused Dag versions instead of silently skipping their missing serialized rows. Signed-off-by: 1fanwang <[email protected]>
Keep the recovery instructions readable without requiring dictionary exceptions in the documentation build. Signed-off-by: 1fanwang <[email protected]>
Wide metadata loss should not pay the persistence setup cost for each source file. Keep initial recovery visible promptly while avoiding unnecessary bundle setup and repeated metadata lookups. Signed-off-by: 1fanwang <[email protected]>
94fc568 to
ed2c9d0
Compare
potiuk
added
the
closed because of open PR limit
|
Hello @1fanwang - thank you for your contributions to Apache Airflow! The Airflow community has introduced a limit of 5 open pull requests at a time for contributors without write access to the repository. You currently have 34 open pull requests, so - as a one-time step of introducing the limit - we closed the ones where maintainers have not engaged yet:
These pull requests stay open because maintainers are already engaged in them - they count towards your limit:
This is not a judgement of you or of your changes. We never told contributors before that opening many pull requests at once was a problem, so there is nothing to feel bad about - and nothing is lost: your branches, commits and the review history stay where they are. What we ask you to do is to make your first prioritization decision: choose which of the pull requests above matter most to you, and reopen them (up to 5 open at a time, including the ones still open) with the "Reopen pull request" button or While your pull requests are waiting for review, the most valuable thing you can do is help in other ways - reviewing other contributors' pull requests, helping with issues, and taking part in the discussions on the devlist and Slack. Why we introduced the limit, what it means for you and how to reopen or restore a pull request is explained in https://github.com/apache/airflow/blob/main/contributing-docs/32_open_pull_request_limit.rst. Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting |
On upstream Airflow 3, metadata repair can leave an active Dag registered but without a serialized definition, even when its source file is still available. This public report describes missing metadata and unsuccessful attempts to recover by reserializing.
In our past incident recovery, we needed to get missing definitions back quickly while preserving usable metadata. Reparsing unrelated healthy files delayed that work. This PR addresses that recovery need in AF3 without changing Dag cleanup.
Recovery behavior
airflow dags reserialize --only-missingselects the recorded files for active Dags without serialized metadata. It queries the selected bundles together, initializes only bundles with work, and parses each selected file once.--bundle-namelimits the scope.The first file containing Dags is committed immediately. Later writes use batches of up to 32 files per bundle to reduce repeated queries and commits. Existing serialized definitions are not cleared; normal Dag processing still owns removals.
The shared writer also recreates a missing serialized row when its unused Dag version survives. Previously, that path returned without restoring the row. The fix applies to normal Dag processing as well as the CLI.
This repairs current definitions for active, registered Dags with usable source locations. Stale or unregistered Dags still need normal discovery. It does not repair source or schema failures or reconstruct historical versions.
Benchmark
The same test recovered 64 generated one-task Dags against real SQLite, with three alternating samples of the per-file revision and the optimized revision. These are recovery-handler timings, not production or end-to-end CLI latency.
Both made one definition visible on the first recovery commit. First-visible ranges overlapped; this sample does not show an improvement in first-availability latency.
A separate native-CLI case restores two missing definitions among 70 generated Dags by parsing one file instead of 68, while preserving the existing serialized rows.
Benchmark setup, commands, and raw samples
The fixture first parses and stores the Dags normally, then removes the selected serialized definitions. The timing includes its read-after-commit visibility checks and excludes process startup, argument parsing, and database initialization. First-visible ranges were 0.226 to 0.446 seconds before and 0.240 to 0.425 seconds after.
The baseline checkout is
38cba6be05168f807130916fbedf8ff7b0cdb698; the optimized checkout ised2c9d0857e95b603f984375e387df99bccc578b. Copy this PR'sairflow-core/tests/unit/cli/commands/test_dag_reserialize.pyinto the baseline checkout so both run the same observer. Runtime source stays at the pinned revisions.Run this command from each checkout. The three sample pairs used before/after, after/before, then before/after order:
The baseline restores all 64 definitions but fails the new
[1, 33, 64]progress assertion because it commits once per file. The optimized revision passes. These are the recorded samples, with revision and sample labels added by the benchmark driver:{"recovery_commits":64,"first_committed_dags":1,"first_visible_seconds":0.446,"recovery_seconds":2.917,"variant":"before","sample":1} {"recovery_commits":3,"first_committed_dags":1,"first_visible_seconds":0.24,"recovery_seconds":0.717,"variant":"after","sample":1} {"recovery_commits":3,"first_committed_dags":1,"first_visible_seconds":0.425,"recovery_seconds":1.721,"variant":"after","sample":2} {"recovery_commits":64,"first_committed_dags":1,"first_visible_seconds":0.226,"recovery_seconds":1.631,"variant":"before","sample":2} {"recovery_commits":64,"first_committed_dags":1,"first_visible_seconds":0.267,"recovery_seconds":1.913,"variant":"before","sample":3} {"recovery_commits":3,"first_committed_dags":1,"first_visible_seconds":0.345,"recovery_seconds":0.768,"variant":"after","sample":3}The native-CLI file-selection case produced:
Coverage also includes bundle filtering, relocated paths, ZIP and single-file bundles, import failures within a batch, compressed metadata, and the surviving-version recovery case. A database failure can interrupt the current batch; previously committed recovery remains available.
Was generative AI tooling used to co-author this PR?
Generated-by: GitHub Copilot CLI (GPT-6 Astra) following the guidelines
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.