GitDagBundle: an interrupted clone can leave untracked working-tree files that block every later checkout #72922

Apache Airflow Provider(s)

git

Versions of Apache Airflow Providers

apache-airflow-providers-git 0.4.x (code path unchanged on main)

Apache Airflow version

3.3.0

Operating System

Linux (containerized)

Deployment

Other 3rd-party Helm chart

Deployment details

Long-lived worker pods (a worker Deployment rather than one pod per task), with the bundle storage path left at its default under /tmp, so it is pod-local and lives as long as the pod.

What happened

A task was marked failed from the UI while its worker was cloning the DAG bundle's tracking repository. The clone started at 06:04:11.472800 and the supervisor sent SIGTERM at 06:04:11.787280, about 315 ms in:

{"timestamp":"2026-09-01T06:04:11.775140Z","level":"error","event":"Server indicated the task shouldn't be running anymore","detail":{"reason":"not_running","message":"TI is no longer in the running state and task should terminate","current_state":"failed"},"status_code":409,"logger":"supervisor"}
{"timestamp":"2026-09-01T06:04:11.787280Z","level":"info","event":"Process exited","pid":16,"exit_code":-15,"signal_sent":"SIGTERM","logger":"supervisor"}

From then on every task that landed on that worker failed at parse time:

Top level error source=task loc=task_runner.py:2347 GitCommandError: Cmd('git') failed due to: exit code(1)
  cmdline: git checkout master
  stderr: 'error: The following untracked working tree files would be overwritten by checkout:
        .agents/instructions/airflow-patterns.md
        .agents/instructions/column_naming_conventions.md
        .agents/instructions/data-modeling-conventions.md
        <...>
Aborting'
    File .../airflow/sdk/execution_time/task_runner.py, line 1005 in parse
    File .../airflow/providers/git/bundles/git.py, line 244 in initialize
    File .../airflow/providers/git/bundles/git.py, line 215 in _initialize

Thirty task instances ran on that pod over the following 23 minutes and all thirty failed. Sibling pods in the same ReplicaSet were unaffected, one of them completed 1,180 tasks the same day, and it was the only worker in three weeks with a 100% failure rate. The killed task was the first task that pod ever ran, so it was poisoned on its first bundle initialization and never served a successful task. Recovery only came when the workers rolled to a new ReplicaSet 20 minutes later.

The leftover directory here is different from the one in #72759. There the working tree is empty and the refs are missing, so the checkout fails with pathspec ... did not match any file(s) known to git. Here git clone had got far enough to write files into the working tree but not far enough to record them in the index, so those files are untracked relative to the incomplete index and the checkout refuses to overwrite them.

Why it never self-heals is the same as in #72759:

1. _is_pruned_worktree() treats "no .git" as a completed version, and a .git is present, so it returns False.
2. _local_repo_has_version() returns False for the truncated repo.
3. _clone_repo_if_required() only clones when repo_path does not exist. It does exist, so the clone is skipped and the truncated repository is reopened.
4. self.repo.git.checkout(self.tracking_ref) then fails. It sits outside both the try/except and the @retry / shutil.rmtree block inside _clone_repo_if_required(), so nothing cleans the directory up and nothing retries.

Two things make this worse than a one-off task failure. The failure is raised in task_runner.parse(), before the DAG is parsed, so task-level on_failure_callback and on_retry_callback never run and the failures are invisible to DAG-level alerting. And on a long-lived worker the pod-local bundle directory survives for the pod's lifetime, so a transient fault becomes permanent. On KubernetesExecutor, where every task gets a fresh pod, the same bug is invisible.

Which of the open fixes cover this

#72787 does. Moving the checkout inside _clone_repo_if_required()'s try means the GitCommandError hits the existing except (InvalidGitRepositoryError, GitCommandError) handler, shutil.rmtree(repo_path) runs, and the single retry produces a clean clone.

#71535 does not. Fetching the missing tracking refs before the checkout does nothing about untracked files in the working tree, so git checkout master still aborts with the error above. So this state is a concrete reason to prefer the broader recovery in #72787 over the narrow fetch, rather than a matter of taste.

What is still uncovered by either

Both fixes recover from the broken directory after the fact. Neither stops it being created. Repo.clone_from still writes straight into repo_path, so any interruption can still leave a directory there that the next initialization has to detect and discard. #72759 already noted one residual case this leaves open: a process killed in the narrow window before git clone writes the remote configuration leaves a repository with no origin, where self.repo.remotes.origin raises AttributeError rather than anything the recovery path catches.

Cloning into a staging directory next to the final path and renaming it into place once the clone is complete removes the whole class. rename is atomic, so repo_path either does not exist or holds a finished clone, and an interrupted clone leaves only an orphaned staging directory that the next attempt discards before it starts.

What you think should happen instead

An interrupted clone should not be able to poison a bundle directory at all, and if a poisoned directory does exist (from an earlier provider version, or from anything else that damaged it), initialization should discard and re-clone it rather than failing forever.

How to reproduce

The end state reproduces in seconds without racing a real signal:

git clone --no-checkout <bare-repo> tracking_repo   # valid .git, empty index and working tree
# copy the repository's tracked files into tracking_repo/ so they are untracked
git -C tracking_repo checkout master
# error: The following untracked working tree files would be overwritten by checkout: ...
# Aborting

Point a GitDagBundle at that directory and every initialize() fails with the traceback above.

For the crash-consistency half, patch Repo.clone_from to write a .git skeleton into its target and then raise, and check whether repo_path exists afterwards. On main it does.

Anything else

Happy to take the crash-consistency change; I have it working with tests. The recovery half is already covered by #72787 and I would rather that landed than have a competing PR for it.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct