Skip to content

Commit 7a02983

Browse files
anandgupta42clauderalphstodomingo
authored
fix(ci): validate cached dbt e2e environments; pin the uv venv to the system Python (#1258)
* fix(ci): validate cached dbt e2e environments; pin the uv venv to the system Python The dbt-tools E2E job on main fails with ENOENT executing test/.dbt-resolve-envs/uv/.venv/bin/dbt: the restored cache holds a uv venv whose bin/python links to a uv-managed interpreter outside the cached directory, absent on a fresh runner. setup-resolve.sh trusted the .done marker and skipped setup, and actions/cache never re-saves on a hit, so the broken environment came back on every run. - setup-resolve.sh: a cached environment counts only if its dbt --version runs (with a timeout); otherwise it is rebuilt. Applied to every scenario. - uv venv is created with --python "$REAL_PYTHON" so it links to an interpreter that exists on every runner. - ci.yml: cache key bumped to -v2 so the broken v1 cache is not restored. Closes #1257 Co-Authored-By: Claude Fable 5.1 <[email protected]> * ci: run the dbt-tools E2E job on pull requests that touch packages/dbt-tools The job ran only on push to main, so a PR fixing its environment setup could not prove the fix before merging. The `dbt-tools` change filter already existed for exactly this; the job now honours it, keeping the 3-minute cost to PRs that change dbt-tools. Co-Authored-By: Claude Fable 5.1 <[email protected]> * ci(e2e): review fixes — gtimeout fallback, setup-python interpreter, read-only token - setup-resolve.sh: `with_timeout` falls back to Homebrew's `gtimeout` on macOS, so the cached-env check is bounded there too (cubic). - setup-resolve.sh: `find_real_python` honours `DBT_E2E_PYTHON`; the workflow sets it to the interpreter actions/setup-python installed, so the scenario venvs no longer build on the runner image's /usr/bin/python3 while the workflow believes it chose 3.11 (CodeRabbit). - ci.yml: the E2E job now runs pull-request code, so it gets `permissions: contents: read` and `persist-credentials: false`, matching the tracker-leaks job (CodeRabbit). Co-Authored-By: Claude Fable 5.1 <[email protected]> * fix(ci): rebuild stale dbt version venvs; stop the versions suite skipping - setup-versions.sh rebuilds a cached venv whose dbt does not run - key both caches on the setup-python version - versions test fails on a missing requested version, and points dbt-cli at each venv - job timeout 20 min Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_0174HUYiaceApqHP7EKMNgnb * ci: keep the dbt-tools E2E job at 10 min The full run takes about 4 min in CI on a cache miss (tests 128 s), so the original limit fits. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_0174HUYiaceApqHP7EKMNgnb * fix(ci): trust a cached dbt venv only if it runs and reports its version - setup-versions.sh requires `dbt --version` to exit 0 and report the requested minor; otherwise it rebuilds - the versions guard compares the version each dbt reports, not dir names Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_0174HUYiaceApqHP7EKMNgnb --------- Co-authored-by: Claude Fable 5.1 <[email protected]> Co-authored-by: Ralph Sto. Domingo <[email protected]>
1 parent 82ef0be commit 7a02983

4 files changed

Lines changed: 110 additions & 20 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 27 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -637,24 +637,34 @@ jobs:
637637
# altimate_change end
638638

639639
# ---------------------------------------------------------------------------
640-
# dbt-tools E2E — slow (~3 min), only on push to main.
640+
# dbt-tools E2E — slow (~4 min): on push to main, and on pull requests that
641+
# touch packages/dbt-tools (the `dbt-tools` change filter above). Without the
642+
# PR path a fix to this job's own environment setup could only be proven
643+
# after merging.
641644
# Tests dbt CLI fallbacks against real dbt versions (1.8, 1.10, 1.11) and
642645
# real Python environments (venv, uv, system).
643646
# ---------------------------------------------------------------------------
644647
dbt-tools-e2e:
645648
name: "dbt-tools E2E"
646649
needs: changes
647-
if: github.event_name == 'push'
650+
if: github.event_name == 'push' || needs.changes.outputs.dbt-tools == 'true'
648651
runs-on: ubuntu-latest
649652
timeout-minutes: 10
653+
# Runs pull-request code (setup scripts, tests), so it gets the same
654+
# read-only token scope and no persisted credentials as tracker-leaks.
655+
permissions:
656+
contents: read
650657
steps:
651658
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
659+
with:
660+
persist-credentials: false
652661

653662
- uses: oven-sh/setup-bun@ecf28ddc73e819eb6fa29df6b34ef8921c743461 # v2
654663
with:
655664
bun-version: "1.3.14"
656665

657666
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
667+
id: python
658668
with:
659669
python-version: "3.11"
660670

@@ -668,21 +678,34 @@ jobs:
668678
uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4
669679
with:
670680
path: packages/dbt-tools/test/.dbt-venvs
671-
key: dbt-venvs-${{ runner.os }}-1.8-1.10-1.11
681+
# Both caches hold venvs linked to the setup-python interpreter, which
682+
# is not cached. Keying on its exact version starts a fresh cache when
683+
# the runner's 3.11 patch release moves, instead of restoring venvs
684+
# that no longer run.
685+
key: dbt-venvs-${{ runner.os }}-py${{ steps.python.outputs.python-version }}-1.8-1.10-1.11
672686

673687
- name: Cache Python env scenarios
674688
uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4
675689
with:
676690
path: packages/dbt-tools/test/.dbt-resolve-envs
677-
key: dbt-resolve-envs-${{ runner.os }}-v1
691+
# v2: the v1 cache held a uv venv linked to a uv-managed Python that
692+
# fresh runners lack (bin/dbt present, exec ENOENT). setup-resolve.sh
693+
# now validates cached envs and pins uv to the system interpreter.
694+
key: dbt-resolve-envs-${{ runner.os }}-py${{ steps.python.outputs.python-version }}-v2
678695

679696
- name: Set up dbt versions
680697
run: ./test/e2e/setup-versions.sh 1.8 1.10 1.11
681698
working-directory: packages/dbt-tools
699+
env:
700+
DBT_E2E_PYTHON: ${{ steps.python.outputs.python-path }}
682701

683702
- name: Set up Python env scenarios
684703
run: ./test/e2e/setup-resolve.sh venv uv system
685704
working-directory: packages/dbt-tools
705+
env:
706+
# Build the scenario venvs on the interpreter setup-python installed,
707+
# not the image's /usr/bin/python3; setup-resolve.sh honours this.
708+
DBT_E2E_PYTHON: ${{ steps.python.outputs.python-path }}
686709

687710
- name: Run dbt-tools E2E tests
688711
run: bun run test:e2e

‎packages/dbt-tools/test/e2e/dbt-versions.test.ts‎

Lines changed: 21 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,9 @@
1212
* If no dbt is available at all, the entire suite is skipped.
1313
*
1414
* Environment variables:
15-
* DBT_E2E_VERSIONS — comma-separated list of versions to test (e.g. "1.8,1.9")
15+
* DBT_E2E_VERSIONS — comma-separated list of versions to test (e.g. "1.8,1.9");
16+
* a listed version without a working dbt of that version
17+
* fails the run
1618
* DBT_E2E_SKIP — set to "1" to skip e2e tests entirely
1719
*/
1820

@@ -29,6 +31,7 @@ import { tmpdir } from "os"
2931
const FIXTURE_DIR = resolve(import.meta.dir, "../fixture")
3032
const VENVS_DIR = resolve(import.meta.dir, "../.dbt-venvs")
3133
const SKIP = process.env.DBT_E2E_SKIP === "1"
34+
const REQUESTED = process.env.DBT_E2E_VERSIONS?.split(",").map((v) => v.trim())
3235

3336
/** Timeout for dbt commands (seed + build can be slow on first run) */
3437
const DBT_TIMEOUT = 120_000
@@ -51,13 +54,12 @@ interface DbtVersion {
5154
}
5255

5356
function discoverVersions(): DbtVersion[] {
54-
const filterVersions = process.env.DBT_E2E_VERSIONS?.split(",").map((v) => v.trim())
5557
const versions: DbtVersion[] = []
5658

5759
// Check venvs
5860
if (existsSync(VENVS_DIR)) {
5961
for (const entry of readdirSync(VENVS_DIR)) {
60-
if (filterVersions && !filterVersions.includes(entry)) continue
62+
if (REQUESTED && !REQUESTED.includes(entry)) continue
6163
const dbtPath = join(VENVS_DIR, entry, "bin", "dbt")
6264
const pythonPath = join(VENVS_DIR, entry, "bin", "python")
6365
if (!existsSync(dbtPath)) continue
@@ -98,6 +100,16 @@ if (!HAS_DBT && !SKIP) {
98100
)
99101
}
100102

103+
// Asking for versions (CI does) makes a missing one a failure, not a skip: a
104+
// stale venv cache once skipped every test below on every run without a sign.
105+
describe.skipIf(SKIP || !REQUESTED)("requested dbt versions", () => {
106+
test("every version in DBT_E2E_VERSIONS has a dbt that runs and reports it", () => {
107+
// Compare what each dbt reports (1.11.0-b3 → 1.11), not its directory name.
108+
const reported = VERSIONS.map((v) => v.full.split(".").slice(0, 2).join("."))
109+
expect(reported.sort()).toEqual([...(REQUESTED ?? [])].sort())
110+
})
111+
})
112+
101113
// ---------------------------------------------------------------------------
102114
// Helpers
103115
// ---------------------------------------------------------------------------
@@ -205,10 +217,15 @@ describe.skipIf(!HAS_DBT)("altimate-dbt e2e", () => {
205217
describe(`dbt ${version.label} (${version.full})`, () => {
206218
let workDir: string
207219

208-
beforeAll(() => {
220+
beforeAll(async () => {
209221
console.log(`\n→ Setting up dbt ${version.full} project...`)
210222
workDir = setupProject(version)
211223
console.log(` Project: ${workDir}`)
224+
// dbt-cli.ts caches the dbt it resolves on first use. configure()
225+
// clears that cache and points it at this version's venv; without it
226+
// every version after the first runs the first version's dbt.
227+
const { configure } = await import("../../src/dbt-cli")
228+
configure({ pythonPath: version.pythonPath, projectRoot: workDir })
212229
}, DBT_TIMEOUT * 2)
213230

214231
afterAll(() => {

‎packages/dbt-tools/test/e2e/setup-resolve.sh‎

Lines changed: 41 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -33,8 +33,39 @@ ok() { echo " ✓ $1"; }
3333
skip() { echo " ⊘ $1 — skipped ($2)"; }
3434
fail() { echo " ✗ $1 — $2"; }
3535

36+
# A restored cache can hold an environment whose interpreter no longer exists:
37+
# CI caches only this directory, and a venv's `bin/python` is a symlink (or a
38+
# launcher shebang) into an interpreter outside it. uv in particular links to
39+
# its own managed Python under ~/.local/share/uv, which a fresh runner lacks,
40+
# so `bin/dbt` is present on disk yet fails with ENOENT when executed. The
41+
# `.done` marker alone therefore proves nothing; a cached environment counts
42+
# only if its dbt actually runs.
43+
with_timeout() {
44+
# GNU timeout on Linux; Homebrew coreutils installs it as gtimeout on macOS.
45+
# Without either, the check still runs, just unbounded.
46+
if has timeout; then timeout "$TIMEOUT" "$@"
47+
elif has gtimeout; then gtimeout "$TIMEOUT" "$@"
48+
else "$@"; fi
49+
}
50+
env_ok() {
51+
[ -x "$1" ] && with_timeout "$1" --version >/dev/null 2>&1
52+
}
53+
# Usage: cached_or_rebuild <scenario> <dbt binary>; returns 0 when the cached
54+
# environment is usable (caller returns), 1 when it must be rebuilt.
55+
cached_or_rebuild() {
56+
local dir="$ENVS_DIR/$1"
57+
[ -f "$dir/.done" ] || return 1
58+
if env_ok "$2"; then ok "$1 (cached)"; return 0; fi
59+
echo " ↻ $1 cache is stale (dbt does not run) — rebuilding..."
60+
return 1
61+
}
62+
3663
# Find a real (non-shim) python3 for venv creation
3764
find_real_python() {
65+
# An explicit interpreter wins. CI sets DBT_E2E_PYTHON to the interpreter
66+
# actions/setup-python installed, so the venvs build on the Python the
67+
# workflow chose rather than whatever the runner image happens to ship.
68+
if [ -n "${DBT_E2E_PYTHON:-}" ] && [ -x "$DBT_E2E_PYTHON" ]; then echo "$DBT_E2E_PYTHON"; return; fi
3869
# Try pyenv's actual python first
3970
if has pyenv; then
4071
local p
@@ -55,7 +86,7 @@ echo "Using Python: $REAL_PYTHON ($($REAL_PYTHON --version 2>&1))"
5586

5687
setup_venv() {
5788
local dir="$ENVS_DIR/venv"
58-
if [ -f "$dir/.done" ]; then ok "venv (cached)"; return; fi
89+
cached_or_rebuild venv "$dir/bin/dbt" && return
5990
rm -rf "$dir"
6091
echo " → Setting up venv..."
6192
"$REAL_PYTHON" -m venv "$dir"
@@ -68,12 +99,14 @@ setup_venv() {
6899
setup_uv() {
69100
local dir="$ENVS_DIR/uv"
70101
if ! has uv; then skip "uv" "uv not installed"; return; fi
71-
if [ -f "$dir/.done" ]; then ok "uv (cached)"; return; fi
102+
cached_or_rebuild uv "$dir/.venv/bin/dbt" && return
72103
rm -rf "$dir"
73104
echo " → Setting up uv..."
74105
mkdir -p "$dir"
75-
# uv project mode: create .venv in dir
76-
uv venv "$dir/.venv" --quiet
106+
# uv project mode: create .venv in dir. Pin the interpreter to the system
107+
# Python so the venv survives a cache restore on a fresh runner; left to
108+
# itself uv links a managed Python that lives outside the cached directory.
109+
uv venv "$dir/.venv" --python "$REAL_PYTHON" --quiet
77110
uv pip install --quiet --python "$dir/.venv/bin/python" "$DBT_SPEC"
78111
touch "$dir/.done"
79112
ok "uv ($("$dir/.venv/bin/dbt" --version 2>&1 | grep -oE 'installed:\s+\S+' | head -1))"
@@ -82,7 +115,7 @@ setup_uv() {
82115
setup_pipx() {
83116
local dir="$ENVS_DIR/pipx"
84117
if ! has pipx; then skip "pipx" "pipx not installed"; return; fi
85-
if [ -f "$dir/.done" ]; then ok "pipx (cached)"; return; fi
118+
cached_or_rebuild pipx "$dir/bin/dbt" && return
86119
rm -rf "$dir"
87120
echo " → Setting up pipx..."
88121
mkdir -p "$dir/bin" "$dir/venvs"
@@ -100,7 +133,7 @@ setup_pipx() {
100133
setup_conda() {
101134
local dir="$ENVS_DIR/conda"
102135
if ! has conda; then skip "conda" "conda not installed"; return; fi
103-
if [ -f "$dir/.done" ]; then ok "conda (cached)"; return; fi
136+
cached_or_rebuild conda "$dir/bin/dbt" && return
104137
rm -rf "$dir"
105138
echo " → Setting up conda..."
106139
conda create -y -p "$dir" python=3.11 --quiet 2>/dev/null
@@ -117,7 +150,7 @@ setup_conda() {
117150
setup_poetry() {
118151
local dir="$ENVS_DIR/poetry"
119152
if ! has poetry; then skip "poetry" "poetry not installed"; return; fi
120-
if [ -f "$dir/.done" ]; then ok "poetry (cached)"; return; fi
153+
cached_or_rebuild poetry "$dir/.venv/bin/dbt" && return
121154
rm -rf "$dir"
122155
echo " → Setting up poetry (in-project venv)..."
123156
mkdir -p "$dir"
@@ -153,7 +186,7 @@ setup_pyenv_venv() {
153186
# Simulates a pyenv user who creates a venv with their pyenv-managed python
154187
local dir="$ENVS_DIR/pyenv-venv"
155188
if ! has pyenv; then skip "pyenv-venv" "pyenv not installed"; return; fi
156-
if [ -f "$dir/.done" ]; then ok "pyenv-venv (cached)"; return; fi
189+
cached_or_rebuild pyenv-venv "$dir/bin/dbt" && return
157190
rm -rf "$dir"
158191
echo " → Setting up pyenv + venv..."
159192
local pyenv_python

‎packages/dbt-tools/test/e2e/setup-versions.sh‎

Lines changed: 21 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,19 @@ get_pip_flags() {
4747
esac
4848
}
4949

50+
# CI passes the interpreter actions/setup-python installed; locally the first
51+
# python3 on PATH is used.
52+
PYTHON="${DBT_E2E_PYTHON:-python3}"
53+
54+
# Prints e.g. `core=1.8.7`, or nothing when `dbt --version` fails. A restored
55+
# cache can hold a venv whose bin/python links to an interpreter that no longer
56+
# exists, so bin/dbt being on disk proves nothing.
57+
installed_version() {
58+
local out
59+
out=$("$1" --version 2>&1) || return 0
60+
echo "$out" | grep -oE 'installed: [0-9.a-z]+' | head -1 | sed 's/installed: /core=/' || true
61+
}
62+
5063
mkdir -p "$VENVS_DIR"
5164

5265
for ver in "${VERSIONS[@]}"; do
@@ -60,13 +73,17 @@ for ver in "${VERSIONS[@]}"; do
6073
fi
6174

6275
if [ -f "$venv_dir/bin/dbt" ]; then
63-
existing=$("$venv_dir/bin/dbt" --version 2>&1 | grep -oE 'installed: [0-9.a-z]+' | head -1 | sed 's/installed: /core=/' | head -1 || echo "unknown")
64-
echo "✓ dbt $ver already installed ($existing) at $venv_dir"
65-
continue
76+
existing=$(installed_version "$venv_dir/bin/dbt")
77+
if [[ "$existing" == "core=$ver."* ]]; then
78+
echo "✓ dbt $ver already installed ($existing) at $venv_dir"
79+
continue
80+
fi
81+
echo "↻ dbt $ver cache is stale (found ${existing:-no dbt that runs}) — rebuilding..."
82+
rm -rf "$venv_dir"
6683
fi
6784

6885
echo "→ Installing dbt $ver..."
69-
python3 -m venv "$venv_dir"
86+
"$PYTHON" -m venv "$venv_dir"
7087
"$venv_dir/bin/pip" install --quiet --upgrade pip
7188
# shellcheck disable=SC2086
7289
"$venv_dir/bin/pip" install --quiet $extra_flags $install_spec

0 commit comments

Comments
 (0)