Skip to content

Commit bbe5f68

Browse files
chiruu12chiruu12
committed
Live tests: fresh resolution, harden conftest
1 parent 21ac18c commit bbe5f68

3 files changed

Lines changed: 44 additions & 26 deletions

File tree

‎.github/workflows/integrations-live.yml‎

Lines changed: 15 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -78,22 +78,28 @@ jobs:
7878
with:
7979
python-version: "3.12"
8080

81-
- name: Install (one framework extra in isolation)
82-
run: uv sync --extra dev --extra ${{ matrix.extra }}
81+
- name: Install (fresh per-framework resolution — matches `pip install "unplug-ai[extra]"`)
82+
# Deliberately NOT `uv sync` from the unified lock: that lock co-resolves
83+
# every extra together and pins some frameworks down to old, broken
84+
# releases (e.g. semantic-kernel 1.15, which imports `Url` from
85+
# `pydantic.networks` — gone in Pydantic v2). A user never installs all
86+
# frameworks at once; they run `pip install "unplug-ai[langgraph]"`, which
87+
# resolves that one framework freshly. We mirror that here.
88+
run: |
89+
uv venv
90+
uv pip install -e ".[dev,${{ matrix.extra }}]"
8391
8492
- name: Live integration test
8593
run: |
8694
set +e
87-
uv run pytest -q -m requires_integrations ${{ matrix.test }}
95+
.venv/bin/python -m pytest -q -m requires_integrations ${{ matrix.test }}
8896
rc=$?
8997
set -e
90-
# Exit 5 = pytest collected nothing because the module self-skipped:
91-
# the framework installed but is unimportable under our pinned deps
92-
# (e.g. an upstream release incompatible with our Pydantic v2). That is
93-
# an upstream conflict, not an Unplug regression — surface it loudly but
94-
# don't block. Real test failures (exit 1) still fail the job.
98+
# Exit 5 = the module self-skipped: the framework is unimportable even
99+
# under a fresh resolution (a genuine upstream break). Surface it loudly
100+
# but don't block the release. Real test failures (exit 1) still fail.
95101
if [ "$rc" -eq 5 ]; then
96-
echo "::warning title=Live integration skipped::${{ matrix.extra }} installed but its live tests were skipped (framework unimportable under current deps). Tolerated; not blocking."
102+
echo "::warning title=Live integration skipped::${{ matrix.extra }} installed but its live tests were skipped (framework unimportable under a fresh resolution). Tolerated; not blocking."
97103
exit 0
98104
fi
99105
exit $rc

‎sdk/integrations/TESTING.md‎

Lines changed: 21 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -115,31 +115,36 @@ These run in the dedicated **`Integrations (live)`** workflow
115115
nightly (06:00 UTC), and via manual dispatch — keeping the everyday PR gate fast while
116116
still catching framework drift.
117117

118-
Run one framework locally (installs just that extra):
118+
**Fresh resolution, on purpose.** Each leg runs `uv pip install -e ".[dev,<extra>]"`, *not*
119+
`uv sync`. The unified `uv.lock` co-resolves every extra together and pins some frameworks to
120+
old releases (e.g. `semantic-kernel` 1.15, which imports `Url` from `pydantic.networks` —
121+
removed in Pydantic v2). No user installs every framework at once; they run
122+
`pip install "unplug-ai[semantic-kernel]"`, which resolves that one framework freshly (→ 1.36,
123+
which works). We mirror the user path. The `-e` (editable) keeps the source tree on `sys.path`
124+
so test-only assets like `configs/ml_validation.json` resolve.
119125

120-
```bash
121-
cd sdk
122-
uv sync --extra dev --extra langgraph
123-
uv run pytest -q -m requires_integrations tests/optional/live/test_langgraph_live.py
124-
```
125-
126-
Run every live test you have frameworks for (installs all agent SDKs):
126+
Run one framework locally the same way CI does (latest compatible version of that extra):
127127

128128
```bash
129-
uv sync --extra dev --extra integrations
130-
uv run pytest -q -m requires_integrations tests/optional/live/
129+
cd sdk
130+
uv venv /tmp/unplug-lg && uv pip install --python /tmp/unplug-lg -e ".[dev,langgraph]"
131+
/tmp/unplug-lg/bin/python -m pytest -q -m requires_integrations tests/optional/live/test_langgraph_live.py
131132
```
132133

133134
> Live tests never call an LLM. They build the smallest real framework object and assert the
134135
> Unplug guard's decision (block on injection / destructive tool, allow benign), so they stay
135136
> hermetic and need no API keys.
136137
137-
**Tolerated skips.** Each leg installs its framework, so a test that runs and fails is a real
138-
regression. But some optional frameworks ship releases that are unimportable under our pinned
139-
core deps (e.g. a `semantic-kernel` build that imports `Url` from `pydantic.networks`, removed
140-
in Pydantic v2). When that happens the module `importorskip`s, pytest exits `5` ("no tests
141-
ran"), and the job emits a non-blocking `::warning::` rather than failing — that is an upstream
142-
conflict, not an Unplug bug. Genuine test failures (exit `1`) still fail the job.
138+
**Verified compatible (2026-06-25)** against latest releases: LangGraph 1.2.6, Agno 2.6.19,
139+
CrewAI 1.14.7, AutoGen-AgentChat 0.7.5, LlamaIndex-core 0.14.23, Pydantic AI 2.0.0,
140+
Semantic Kernel 1.36.0. (Most adapters are framework-agnostic — they wrap `AgentHooks` and
141+
never import the framework — so they are robust across major version bumps.)
142+
143+
**Tolerated skips (safety net).** Each leg installs its framework, so a test that runs and
144+
fails is a real regression. If a future upstream release becomes genuinely unimportable even
145+
under a fresh resolution, the module `importorskip`s, pytest exits `5` ("no tests ran"), and
146+
the job emits a non-blocking `::warning::` rather than failing. Genuine failures (exit `1`)
147+
still fail the job.
143148

144149
## CI markers
145150

‎sdk/tests/conftest.py‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,14 @@ def pytest_configure(config: object) -> None:
2727

2828

2929
def pytest_collection_modifyitems(config: pytest.Config, items: list[pytest.Item]) -> None:
30-
if resolve_validation_checkpoint(require_weights=True) is None:
30+
try:
31+
weights_available = resolve_validation_checkpoint(require_weights=True) is not None
32+
except FileNotFoundError:
33+
# No ML validation manifest on disk (e.g. running the suite against a
34+
# non-editable/wheel install where configs/ are not packaged). Treat as
35+
# "no weights" and skip ML-gated tests instead of crashing collection.
36+
weights_available = False
37+
if not weights_available:
3138
skip_ml = pytest.mark.skip(reason="ML checkpoint weights not available")
3239
for item in items:
3340
if "requires_ml_weights" in item.keywords:

0 commit comments

Comments
 (0)