Skip to content

Fetch missing Git bundle tracking refs - #71535

Open
CAOShurong wants to merge 1 commit into
apache:mainfrom
CAOShurong:codex/airflow-git-bundle-fetch-missing-ref
Open

CAOShurong wants to merge 1 commit into
apache:mainfrom
CAOShurong:codex/airflow-git-bundle-fetch-missing-ref

Conversation

@CAOShurong

Copy link
Copy Markdown

Promoting a SHA-pinned Git Dag bundle can reuse a working clone whose object database predates the new commit, even though the bundle's local bare mirror has already fetched it. The subsequent checkout then fails with reference is not a tree unless local bundle storage is cleared.

Fetch the working clone from its local bare origin only when the configured tracking_ref cannot already be resolved, then continue with the existing checkout flow. This keeps the normal branch and tag paths unchanged and avoids an unconditional extra fetch.

The regression test now promotes a second commit into the same local bundle storage and verifies both the resolved version and the newly added file.

closes: #71388

Testing:

  • pytest providers/git/tests/unit/git/bundles/test_git.py::TestGitDagBundle::test_tracking_ref_commit_sha_promote_without_clearing_storage -xvs (fails on main, passes with this change)
  • pytest providers/git/tests/unit/git/bundles/test_git.py -q -s --disable-warnings (105 passed)
  • focused test after installing the built provider wheel (1 passed)
  • ruff check and ruff format --check for both changed files
  • mypy providers/git/src/airflow/providers/git/bundles/git.py
  • uv build --project providers/git --wheel
  • prek run --from-ref b2ee721a2cd30181b2759c00f359d115e2859a8c --stage pre-commit
  • prek run mypy-providers --from-ref b2ee721a2cd30181b2759c00f359d115e2859a8c --stage manual

Was generative AI tooling used to co-author this PR?
  • Yes (OpenAI Codex)

Generated-by: OpenAI Codex following the guidelines


Drafted-by: OpenAI Codex (no human review before posting)

Promoting a SHA-pinned Dag bundle can reuse a working clone whose object database predates the new commit. Reusing that storage should not require operators to clear it manually.
@boring-cyborg

boring-cyborg Bot commented Aug 13, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: [email protected]
    Slack: https://s.apache.org/airflow-slack

@8silvergun

Copy link
Copy Markdown
Contributor

Following up from #72759, which describes a different way to reach the same failing checkout: if a process is killed while a bundle version is being cloned, versions/<sha>/ is left with a .git that has no references and an empty working tree, and it is reused on every later initialization.

Your change looks like it recovers that state too, for a reason it was not written for:

  • _has_version(repo, tracking_ref) cannot resolve tracking_ref in a repository with no refs, so it returns False and the fetch runs.
  • git clone writes the remote before it transfers objects, so a leftover can still have remote.origin.url configured, pointing at the bundle's healthy local bare mirror. The fetch is local and repopulates refs/remotes/origin/*, after which the checkout resolves.

If you agree, would you like a regression test for that path added to this PR? It initializes a versioned bundle with a named branch such as main as tracking_ref, removes all loose and packed refs from versions/<sha>/.git, leaves HEAD pointing at a nonexistent branch, empties the working tree, verifies that origin remains configured, then initializes the same version again and asserts that both the expected version and the Dag files are restored. It fails on main and should pass with this change.

I have it written against the current TestGitDagBundle fixtures and can push it to your branch or post it here as a patch, whichever you prefer. Happy to leave it out if you would rather keep this PR scoped to #71388.

@CAOShurong

Copy link
Copy Markdown
Author

Thanks for the detailed follow-up — I agree the same recovery path covers the scenario in #72759, for the reasons you laid out (no resolvable refs -> fetch runs, and the leftover clone still has origin pointing at the local bare mirror).

A regression test for that path would be a valuable addition. Please post it here as a patch and I will apply it to this branch and update the PR description to reference #72759 alongside #71388.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GitDagBundle: promoting a SHA-pinned tracking_ref can fail with 'reference is not a tree' if local storage isn't cleared

2 participants