Add pin_symlink_on_refresh option to LocalDagBundle for atomically-swapped (git-sync) checkouts - #70003

Thanks for the review @ephraimbuddy — you're right, LocalDagBundle already covers the git-sync pattern the same way the Airflow 2 dags folder did, so a new bundle class was unnecessary.

I reworked the PR as you suggested: the GitSyncDagBundle class is removed, and the symlink pinning is now an opt-in pin_symlink_on_refresh flag on LocalDagBundle (default False, so no behavior change for existing users).

On your note about workers: correct — each component (dag processor, worker, triggerer) resolves the symlink when it initializes the bundle, so pinning gives per-process consistency (one parse round or one task run), not global consistency. I documented that scope, plus the git-sync --stale-worktree-timeout recommendation, in the dag-bundles docs.

PTAL when you have a chance.

Maybe I don't follow, but why wouldn't you want to jump to the newer commit right when you can? I don't see a point in finishing the round on the old version, just to reparse it all on the new version?

@jedcunningham Good question — pinning doesn't delay picking up new commits: the very next refresh() repoints to the new target, so the update cadence is unchanged. What it prevents is a swap landing mid-round, where one parse round (or one task run) reads files from two different commits — e.g. a DAG file from the new commit importing a shared helper still resolved from the old one. For parsing that means transient import errors that mark DAGs broken and vanish on the next round; for a task run it's a real, non-self-healing failure. We hit both in production with a ~3,600-DAG repo, where long parse rounds make the window wide.

Another way to look at it: GitDagBundle also only moves its working tree forward at refresh() — nobody expects it to jump to a newer commit halfway through a parse. This flag just gives the git-sync/symlink deployment style the same refresh-boundary semantics; today it's the odd one out, with code changing underneath a round at arbitrary times.

For context: we ran this exact pattern (a central git-sync deployment + shared RWX PVC + LocalDagBundle) over that repo — stock GitDagBundle would have added ~15s of per-pod clone cold-start to every single task, and shallow/sparse cloning shaves that down but every pod still pays a network fetch on startup. Note this is also what the official helm chart deploys: the default dagBundleConfigList is a LocalDagBundle over the dags folder, and dags.gitSync syncs that folder underneath it — so every chart user with gitSync enabled is on this pattern today, with reads tearing across commits at arbitrary times.

@jedcunningham On losing the versioning benefits — you're right that GitDagBundle brings DagVersioning (each commit becomes a DagVersion, and task runs are pinned to the version they were queued with), and that's a real benefit. For deployments that want commit-pinned execution it's the right choice, and nothing here argues against it.

The thing is, git-sync + LocalDagBundle is a separate, already-shipped pattern — it's what the Helm chart deploys by default when dags.gitSync is enabled (the default bundle is a LocalDagBundle over the dags folder, with git-sync swapping the checkout underneath it). Deployments on it have already traded per-commit versioning for centralized sync and no per-pod clone cold-start on large repos. This PR doesn't push anyone toward that trade or away from GitDagBundle; it just fixes a correctness bug — mid-round cross-commit reads — that today silently affects everyone already on that pattern, including default-Helm gitSync users.

So "use GitDagBundle to keep versioning" is fair advice, but it doesn't help the deployments that are on git-sync right now and currently hit silent cross-commit tearing — telling them not to use git-sync isn't a fix for the bug they're hitting. The two are orthogonal: this opt-in flag (default False) makes the git-sync path correct; it doesn't make it preferable.

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.