Skip to content

Commit ddf3c7a

Browse files
authored
Breeze: fail fast when building provider sdists from a git worktree (#65771)
* Breeze: fail fast when building provider sdists from a git worktree flit's --use-vcs silently produces incomplete sdists when run from a `git worktree add ...` directory, because flit.vcs.identify_vcs() checks `(p / ".git").is_dir()` and in a worktree `.git` is a file (gitdir: pointer). flit then falls back to a minimal sdist that omits docs/, tests/, provider.yaml and other tracked files, and the resulting packages fail reproducibility checks against released sdists on dist.apache.org — with no warning. `breeze release-management prepare-provider-distributions` now exits 1 with a clear explanation and a workaround (use a plain checkout, or pass `--distribution-format wheel` — wheels are unaffected) when it detects it is running from a worktree and would build sdists. Wheels and providers using hatchling with explicit sdist includes are not affected, so only `sdist` and `both` formats trigger the check. * Breeze: detect Docker-mounted worktrees in provider-sdist check When Breeze runs prepare-provider-distributions from a git worktree, the worktree's .git file carries an absolute gitdir: pointer to the main repo's .git/worktrees/<name> directory. Only the worktree folder is bind-mounted into Breeze's Docker container, so that pointer target is unreachable from inside the build. flit's VCS detection then either fails or silently produces an incomplete sdist. Expand check_flit_worktree_compatibility to parse .git and branch on the failure mode: read error, unexpected format, missing gitdir target (Docker mount case), or healthy host-side worktree. Tests cover all paths. * Link upstream flit issue/PR + tracking issue at the workaround site Add references to pypa/flit#798 and pypa/flit#799 in the docstring of check_flit_worktree_compatibility, and the Airflow tracking issue #65772 at both the helper and its call site. CLAUDE.md requires the tracking-issue URL to appear at the workaround site in the code so the follow-up work is discoverable from any grep or code review.
1 parent 77fa348 commit ddf3c7a

3 files changed

Lines changed: 241 additions & 1 deletion

File tree

‎dev/breeze/src/airflow_breeze/commands/release_management_commands.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -112,6 +112,7 @@
112112
PrepareReleasePackageTagExistException,
113113
PrepareReleasePackageWrongSetupException,
114114
build_provider_distribution,
115+
check_flit_worktree_compatibility,
115116
cleanup_build_remnants,
116117
get_packages_list_to_act_on,
117118
move_built_distributions_and_cleanup,
@@ -1157,6 +1158,11 @@ def prepare_provider_distributions(
11571158
version_suffix: str,
11581159
):
11591160
perform_environment_checks()
1161+
# Workaround for pypa/flit#798 (fix PR pypa/flit#799) plus the Breeze
1162+
# Docker-mount case. Remove this call and the helper it invokes once
1163+
# the conditions in the tracking issue are met:
1164+
# https://github.com/apache/airflow/issues/65772
1165+
check_flit_worktree_compatibility(distribution_format)
11601166
fix_ownership_using_docker()
11611167
cleanup_python_generated_files()
11621168
console_print("\n[info]Cleaning generated _api folders from docs directories")

‎dev/breeze/src/airflow_breeze/prepare_providers/provider_distributions.py‎

Lines changed: 108 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,11 +33,118 @@
3333
get_removed_provider_ids,
3434
tag_exists_for_provider,
3535
)
36-
from airflow_breeze.utils.path_utils import AIRFLOW_DIST_PATH
36+
from airflow_breeze.utils.path_utils import AIRFLOW_DIST_PATH, AIRFLOW_ROOT_PATH
3737
from airflow_breeze.utils.run_utils import run_command
3838
from airflow_breeze.utils.version_utils import is_local_package_version
3939

4040

41+
def check_flit_worktree_compatibility(distribution_format: str) -> None:
42+
"""Refuse to build provider sdists when VCS detection will silently break.
43+
44+
Provider sdists are built with ``flit --use-vcs``. flit identifies the VCS
45+
via ``flit.vcs.identify_vcs``, which checks ``(p / ".git").is_dir()``.
46+
That check fails in two related scenarios, both of which produce a
47+
broken sdist with no warning from flit:
48+
49+
1. **Plain git worktree on the host.** Inside a ``git worktree add ...``
50+
directory ``.git`` is a regular file containing a ``gitdir: <path>``
51+
pointer, not a directory. ``is_dir()`` returns False, ``identify_vcs``
52+
returns ``None``, and flit falls back to a non-VCS sdist that omits
53+
tracked files like ``docs/``, ``tests/`` and ``provider.yaml``. The
54+
resulting packages fail reproducibility checks against the released
55+
sdists on dist.apache.org.
56+
2. **Breeze Docker builds with a worktree mounted.** When the build runs
57+
inside Breeze's container, only the worktree directory is
58+
bind-mounted. The absolute ``gitdir:`` path baked into ``.git``
59+
points somewhere on the host (``<main_repo>/.git/worktrees/<name>``)
60+
that does not exist inside the container, so even if flit recognised
61+
worktrees, ``git ls-files`` would fail or misbehave.
62+
63+
Wheels are built from ``pyproject.toml`` metadata and are unaffected, so
64+
this check is a no-op for ``--distribution-format wheel``. Providers
65+
that use ``hatchling`` with an explicit
66+
``[tool.hatch.build.targets.sdist]`` ``include`` list are also
67+
unaffected.
68+
69+
Upstream flit tracking:
70+
71+
- Issue: https://github.com/pypa/flit/issues/798
72+
- Fix PR: https://github.com/pypa/flit/pull/799
73+
74+
Airflow-side follow-up (remove this workaround when both conditions in
75+
the tracking issue are met):
76+
https://github.com/apache/airflow/issues/65772
77+
"""
78+
if distribution_format == "wheel":
79+
return
80+
81+
git_marker = AIRFLOW_ROOT_PATH / ".git"
82+
if not git_marker.is_file():
83+
# Regular .git directory (or missing entirely) — flit's VCS detection
84+
# works correctly, nothing to do.
85+
return
86+
87+
# .git is a file — typical in a git worktree (or submodule). Parse the
88+
# pointer so we can give a precise error for each failure mode, and so
89+
# we detect the Breeze-in-Docker case where the target is unreachable.
90+
try:
91+
content = git_marker.read_text(encoding="utf-8").strip()
92+
except OSError as err:
93+
console_print(
94+
f"\n[error]Cannot read {git_marker}: {err}.[/]\n"
95+
"[info]Run this command from a plain `git checkout`, or pass "
96+
"`--distribution-format wheel` (wheels are unaffected).[/]\n"
97+
)
98+
sys.exit(1)
99+
100+
if not content.startswith("gitdir:"):
101+
console_print(
102+
f"\n[error]Unexpected `.git` file format at {git_marker}.[/]\n\n"
103+
f"[warning]Expected a `gitdir: <path>` pointer (git worktree or "
104+
f"submodule), got:\n {content!r}[/]\n\n"
105+
"[info]Run this command from a plain `git checkout`, or pass "
106+
"`--distribution-format wheel` (wheels are unaffected).[/]\n"
107+
)
108+
sys.exit(1)
109+
110+
gitdir = Path(content[len("gitdir:") :].strip())
111+
if not gitdir.exists():
112+
console_print(
113+
"\n[error]Cannot build provider sdists: git worktree pointer "
114+
"targets a missing gitdir.[/]\n\n"
115+
f"[warning]The `.git` file at `{git_marker}` points to "
116+
f"`{gitdir}`, but that directory does not exist from here.[/]\n\n"
117+
"[info]This typically happens when Breeze runs the build inside "
118+
"Docker: only the worktree directory is bind-mounted into the "
119+
"container, so the main repo's `.git/worktrees/<name>` folder "
120+
"that the pointer references is not reachable. flit then either "
121+
"fails outright or silently produces an incomplete sdist that "
122+
"omits tracked files.[/]\n\n"
123+
"[info]Run this command from a plain `git checkout` (not a "
124+
"`git worktree add ...` directory), or pass "
125+
"`--distribution-format wheel` — wheels don't need VCS metadata "
126+
"and are unaffected.[/]\n"
127+
)
128+
sys.exit(1)
129+
130+
# Healthy worktree on the host. flit's `.is_dir()` check still misses it,
131+
# so we refuse sdist builds here too.
132+
console_print(
133+
"\n[error]Cannot build provider sdists from a git worktree.[/]\n\n"
134+
"[warning]flit's `--use-vcs` does not recognise git worktrees "
135+
"(flit.vcs.identify_vcs uses `.git.is_dir()`, but in a worktree "
136+
"`.git` is a file). flit silently falls back to a minimal sdist "
137+
"that omits docs/, tests/, provider.yaml and other tracked files, "
138+
"so the resulting packages fail reproducibility checks against "
139+
"the released sdists on dist.apache.org.[/]\n\n"
140+
"[info]Run this command from a plain `git checkout` (not a "
141+
"`git worktree add ...` directory). If you only need wheels, pass "
142+
"`--distribution-format wheel` — wheels are built from "
143+
"`pyproject.toml` metadata and are not affected by this bug.[/]\n"
144+
)
145+
sys.exit(1)
146+
147+
41148
class PrepareReleasePackageTagExistException(Exception):
42149
"""Tag already exist for the package."""
43150

Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
from __future__ import annotations
18+
19+
import re
20+
from pathlib import Path
21+
from unittest.mock import patch
22+
23+
import pytest
24+
25+
from airflow_breeze.prepare_providers import provider_distributions
26+
from airflow_breeze.prepare_providers.provider_distributions import (
27+
check_flit_worktree_compatibility,
28+
)
29+
30+
# Rich output is wrapped to the terminal width and peppered with ANSI escapes;
31+
# strip both when asserting on message contents.
32+
_ANSI_RE = re.compile(r"\x1b\[[0-9;]*m")
33+
34+
35+
def _plain(out: str, err: str) -> str:
36+
return " ".join(_ANSI_RE.sub("", out + err).split())
37+
38+
39+
@pytest.fixture
40+
def healthy_worktree_root(tmp_path: Path) -> Path:
41+
"""A worktree whose `gitdir:` pointer targets a real directory."""
42+
gitdir = tmp_path / "main_repo" / ".git" / "worktrees" / "feature"
43+
gitdir.mkdir(parents=True)
44+
worktree = tmp_path / "worktree"
45+
worktree.mkdir()
46+
(worktree / ".git").write_text(f"gitdir: {gitdir}\n")
47+
return worktree
48+
49+
50+
@pytest.fixture
51+
def broken_worktree_root(tmp_path: Path) -> Path:
52+
"""A worktree whose `gitdir:` pointer targets a path that does not exist.
53+
54+
This is the Breeze-in-Docker shape: the worktree directory is mounted
55+
into the container but the absolute `gitdir:` path is not.
56+
"""
57+
worktree = tmp_path / "worktree"
58+
worktree.mkdir()
59+
(worktree / ".git").write_text("gitdir: /nonexistent/main/.git/worktrees/feature\n")
60+
return worktree
61+
62+
63+
@pytest.fixture
64+
def malformed_git_file_root(tmp_path: Path) -> Path:
65+
"""`.git` is a file but the content is not a `gitdir:` pointer."""
66+
(tmp_path / ".git").write_text("this is not a git pointer\n")
67+
return tmp_path
68+
69+
70+
@pytest.fixture
71+
def plain_checkout_root(tmp_path: Path) -> Path:
72+
(tmp_path / ".git").mkdir()
73+
return tmp_path
74+
75+
76+
@pytest.mark.parametrize("dist_format", ["sdist", "both"])
77+
def test_healthy_worktree_blocks_sdist_build(healthy_worktree_root: Path, dist_format: str) -> None:
78+
with patch.object(provider_distributions, "AIRFLOW_ROOT_PATH", healthy_worktree_root):
79+
with pytest.raises(SystemExit) as exc_info:
80+
check_flit_worktree_compatibility(dist_format)
81+
assert exc_info.value.code == 1
82+
83+
84+
def test_healthy_worktree_allows_wheel_build(healthy_worktree_root: Path) -> None:
85+
with patch.object(provider_distributions, "AIRFLOW_ROOT_PATH", healthy_worktree_root):
86+
check_flit_worktree_compatibility("wheel")
87+
88+
89+
@pytest.mark.parametrize("dist_format", ["sdist", "both"])
90+
def test_broken_gitdir_pointer_exits_with_docker_hint(
91+
broken_worktree_root: Path, capsys: pytest.CaptureFixture[str], dist_format: str
92+
) -> None:
93+
with patch.object(provider_distributions, "AIRFLOW_ROOT_PATH", broken_worktree_root):
94+
with pytest.raises(SystemExit) as exc_info:
95+
check_flit_worktree_compatibility(dist_format)
96+
assert exc_info.value.code == 1
97+
# The Docker-specific message must surface — the broken pointer is the
98+
# signal we use to tell the user that this is the container-mount case.
99+
captured = capsys.readouterr()
100+
output = _plain(captured.out, captured.err)
101+
assert "Docker" in output
102+
assert "does not exist" in output
103+
104+
105+
def test_broken_gitdir_still_allows_wheel_build(broken_worktree_root: Path) -> None:
106+
with patch.object(provider_distributions, "AIRFLOW_ROOT_PATH", broken_worktree_root):
107+
check_flit_worktree_compatibility("wheel")
108+
109+
110+
@pytest.mark.parametrize("dist_format", ["sdist", "both"])
111+
def test_malformed_git_file_exits(malformed_git_file_root: Path, dist_format: str) -> None:
112+
with patch.object(provider_distributions, "AIRFLOW_ROOT_PATH", malformed_git_file_root):
113+
with pytest.raises(SystemExit) as exc_info:
114+
check_flit_worktree_compatibility(dist_format)
115+
assert exc_info.value.code == 1
116+
117+
118+
def test_malformed_git_file_allows_wheel_build(malformed_git_file_root: Path) -> None:
119+
with patch.object(provider_distributions, "AIRFLOW_ROOT_PATH", malformed_git_file_root):
120+
check_flit_worktree_compatibility("wheel")
121+
122+
123+
def test_plain_checkout_allows_all_formats(plain_checkout_root: Path) -> None:
124+
with patch.object(provider_distributions, "AIRFLOW_ROOT_PATH", plain_checkout_root):
125+
check_flit_worktree_compatibility("sdist")
126+
check_flit_worktree_compatibility("both")
127+
check_flit_worktree_compatibility("wheel")

0 commit comments

Comments
 (0)