Skip to content

Commit 25997cb

Browse files
authored
Merge pull request NVIDIA#161 from opendatahub-io/test/python-ocp-e2e/gmenher
CARRY: test(odh): run supported Python gateway e2e on OpenShift
2 parents 2b8cdff + 9b803f7 commit 25997cb

25 files changed

Lines changed: 1905 additions & 54 deletions
Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
name: ODH Python Gate Checks
2+
3+
on:
4+
pull_request:
5+
branches: [main, master]
6+
paths:
7+
- ".github/workflows/odh-python-gates.yml"
8+
- "e2e/python/**"
9+
- "e2e/rust/tests/odh/tier_plan.py"
10+
- "e2e/rust/tests/odh/tiers.toml"
11+
- "mise.toml"
12+
- "mise.lock"
13+
- "odh/python-gates.toml"
14+
- "odh/requirements.in"
15+
- "odh/requirements.txt"
16+
- "odh/scripts/**"
17+
- "proto/**"
18+
- "python/**"
19+
- "pyproject.toml"
20+
- "tasks/python.toml"
21+
- "tasks/test-odh.toml"
22+
- "uv.lock"
23+
workflow_dispatch:
24+
25+
jobs:
26+
test:
27+
name: ODH Python gate tooling
28+
runs-on: ubuntu-latest
29+
timeout-minutes: 15
30+
permissions:
31+
contents: read
32+
steps:
33+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
34+
with:
35+
persist-credentials: false
36+
37+
# This check uses Python and uv only. Avoid installing the complete
38+
# repository toolchain (Rust, Docker helpers, Helm, etc.), which is
39+
# unrelated to the gate tooling and can time out on hosted runners.
40+
- uses: jdx/mise-action@e6a8b3978addb5a52f2b4cd9d91eafa7f0ab959d # v4.2.0
41+
with:
42+
install: false
43+
experimental: true
44+
45+
- name: Install Python tools
46+
run: mise install --locked python uv
47+
48+
- name: Test ODH Python gate tooling
49+
run: mise run --skip-tools test:odh-python-gates

‎deploy/konflux/e2e-odh/README.md‎

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -9,11 +9,13 @@ by sandbox lifecycle tests, and Git for repository workloads.
99

1010
At runtime, the image entrypoint reads the fork-local tier plan, deploys each
1111
phase's gateway mode through the shared Quay deployment script, runs the
12-
phase-specific nextest selection from the precompiled archive, and tears down
13-
that deployment before moving to the next phase. Local `mise run e2e:odh*`
14-
tasks use the same plan, entrypoint, and deployment path. Tier 2, `odh`, and
15-
`full` run shared mode followed by managed-workspace mode; each logical tier
16-
runs image provenance once in the shared phase. The entrypoint preserves
12+
phase's configured runners (the precompiled nextest selection and, when
13+
selected, the Python gateway E2E), and tears down that deployment before moving
14+
to the next phase. Local `mise run e2e:odh*` tasks use the same plan,
15+
entrypoint, and deployment path. Tier 2, `odh`, and `full` run shared mode
16+
followed by managed-workspace mode; Tier 2's shared phase also runs the Python
17+
gateway E2E. Each logical tier runs image provenance once in the shared phase.
18+
The entrypoint preserves
1719
cleanup on failures and signals, continues after a test-phase failure, and
1820
merges the ordered JUnit files into one `e2e-odh-<tier>-xunit.xml` for Jenkins
1921
discovery plus `e2e-odh-<tier>.html`. Phase reports remain available as
@@ -47,8 +49,9 @@ take precedence. See [runtime image selection](../../../e2e/rust/tests/odh/READM
4749
for selection rules, permissions, and fallback behavior.
4850

4951
The image also carries a fully prefetched Python 3.11 test environment and
50-
generated protobuf bindings, so later named Python test modes can execute
51-
without runtime dependency downloads.
52+
generated protobuf bindings. The `python` entrypoint mode uses that environment
53+
to run the supported upstream Python gateway E2E on OpenShift without runtime
54+
dependency downloads.
5255

5356
Konflux onboarding must use the `linux-m2xlarge/*` builders and retain the
5457
three Hermeto prefetch inputs: Cargo at `.` and `e2e/rust`, plus the locked

‎e2e/rust/tests/odh/AGENTS.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,10 @@ a rule and the README disagree, fix the drift rather than guessing.
8282
- Add tests by creating a `.rs` file under the right tier and a `mod` line in
8383
that tier's `mod.rs`, and map upstream binaries/filters in `tiers.toml`. Do
8484
not add new one-off `mise` tasks for individual test lanes.
85+
- The sole narrow exception is `e2e:odh:python`: it is the plan-backed,
86+
full upstream Python gateway-E2E suite, not an individual test lane. It
87+
must continue to use the same tier-plan resolver and shared entrypoint as
88+
every other ODH task.
8589
- Keep the standard tier entry points
8690
(`e2e:odh:smoke|tier1|tier2|tier3`, `e2e:odh`, and `e2e:odh:full`)
8791
stable. The OpenShift AI Shift-Left pipeline consumes those tiers as

‎e2e/rust/tests/odh/README.md‎

Lines changed: 28 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -203,10 +203,34 @@ context you happen to have active elsewhere. This means:
203203
| `mise run e2e:odh:full` | All upstream e2e + e2e-kubernetes + ODH tests (see caveat below), including image provenance |
204204
| `mise run e2e:odh:smoke` | Smoke tier: mapped upstream tests + ODH `smoke::` (includes image provenance) |
205205
| `mise run e2e:odh:tier1` | Tier 1: mapped upstream tests + ODH `tier1::` + image provenance |
206-
| `mise run e2e:odh:tier2` | Tier 2: mapped upstream tests + ODH `tier2::` + image provenance |
206+
| `mise run e2e:odh:tier2` | Tier 2: mapped upstream tests + ODH `tier2::` + supported Python gateway E2E + image provenance |
207207
| `mise run e2e:odh:tier3` | Tier 3: mapped upstream tests + ODH `tier3::` + image provenance |
208+
| `mise run e2e:odh:python` | Supported upstream Python gateway E2E in the shared OpenShift mode |
208209
| `cargo nextest run --config-file "$PWD/.config/nextest.toml" --tool-config-file "odh:$PWD/e2e/rust/tests/odh/nextest.toml" --profile e2e-odh --manifest-path e2e/rust/Cargo.toml --features e2e-odh --test odh -E 'test(=module::test_name)'` | A single ODH test function |
209210

211+
### Python gateway E2E
212+
213+
`e2e:odh:python` reuses the public ODH entrypoint: it deploys the shared
214+
OpenShift gateway, imports the example provider profiles, runs upstream
215+
`e2e/python`, and tears the deployment down. It writes the phase-scoped report
216+
`e2e-odh-python-shared-python.xml` and merged `e2e-odh-python-xunit.xml` plus HTML
217+
companion in `results/`. `odh/python-gates.toml` contains the exact tests that cannot run
218+
on the current OpenShift lane (host-only fixtures, unrestricted egress, the
219+
missing Python workload image, and the pending upstream deletion-timeout
220+
decision). It does not run the Rust image-provenance check. Gate entries are
221+
collection-validated so an upstream rename cannot
222+
silently leave a stale exclusion. The phase also fails when the JUnit report
223+
shows no executed test case, so a suite that skips every collected case cannot
224+
report success. The Python tests also run as part of the tier2 shared phase,
225+
riding the same gateway deployment.
226+
227+
The public `mise` tasks that include the Python runner (`e2e:odh:python` and
228+
`e2e:odh:tier2`) provision `python:dev` first. If invoking
229+
`odh/scripts/e2e-odh-entrypoint.sh` directly for one of those plan-backed
230+
tiers, prepare that environment yourself with `mise run python:dev`; this is
231+
intentional so the entrypoint fails before deploying when the test interpreter
232+
is unavailable.
233+
210234
Example, running the Smoke tier against a real cluster:
211235

212236
```bash
@@ -306,10 +330,10 @@ mode. Phase JUnit files are named `e2e-odh-<tier>-<phase>.xml`; the entrypoint
306330
merges them into `e2e-odh-<tier>-xunit.xml` for Jenkins discovery and
307331
`e2e-odh-<tier>.html` in `results/`.
308332

309-
### Every `e2e:odh*` task runs the image provenance check
333+
### Image provenance runs in every nextest-backed tier
310334

311335
`smoke::image_provenance::test_sandbox_gateway_supervisor_images` runs in the
312-
shared phase of every logical tier. An empty phase fails instead of passing
336+
shared phase of every nextest-backed tier. An empty phase fails instead of passing
313337
with an empty report.
314338

315339
## Image provenance verification
@@ -413,7 +437,7 @@ locally with:
413437

414438
The local build tags the image as `odh-openshell-e2e`.
415439

416-
The image accepts `smoke`, `tier1`, `tier2`, `tier3`, `odh`, or `full`
440+
The image accepts `smoke`, `tier1`, `tier2`, `tier3`, `odh`, `full`, or `python`
417441
as its first argument; `smoke` is the default. The Shift-Left job mounts
418442
cluster credentials and a writable report directory, then passes environment
419443
variables through `--env-file`:

‎e2e/rust/tests/odh/tier_plan.py‎

Lines changed: 42 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,9 @@
1616
GLOBAL_EXCLUSIONS_KEY = "global_test_exclusions"
1717
_BINARY = re.compile(r"[A-Za-z0-9_-]+\Z")
1818
_FILTER = re.compile(r"[A-Za-z0-9_:]+\Z")
19+
# The runners a phase may execute, in order. "nextest" runs the Rust test
20+
# selection; "python" runs the upstream Python gateway E2E suite.
21+
_RUNNERS = frozenset({"nextest", "python"})
1922

2023

2124
def _validate_mode(mode_name: str, mode: dict) -> dict:
@@ -76,6 +79,7 @@ def load_plan(tier: str, plan_path: Path = PLAN_FILE) -> dict:
7679
if name in modes and name != base_mode
7780
)
7881
names: set[str] = set()
82+
report_paths: set[str] = set()
7983
resolved = []
8084
for phase_name, phase_config in phase_tables:
8185
phase = phase_config
@@ -87,6 +91,20 @@ def load_plan(tier: str, plan_path: Path = PLAN_FILE) -> dict:
8791
names.add(name)
8892
if not isinstance(mode_name, str) or mode_name not in modes:
8993
raise ValueError(f"unknown mode {mode_name!r} in tier {tier!r}")
94+
runners = phase.get("runners", ["nextest"])
95+
if not isinstance(runners, list) or not runners or any(
96+
not isinstance(runner, str) or runner not in _RUNNERS for runner in runners
97+
):
98+
raise ValueError(f"invalid runner in phase {name!r}; known: {', '.join(sorted(_RUNNERS))}")
99+
if len(set(runners)) != len(runners):
100+
raise ValueError(f"duplicate runner in phase {name!r}")
101+
for runner in runners:
102+
report_path = f"{name}{'-python' if runner == 'python' else ''}"
103+
if report_path in report_paths:
104+
raise ValueError(
105+
f"duplicate derived report path for phase {name!r}: {report_path!r}"
106+
)
107+
report_paths.add(report_path)
90108
binaries = phase.get("upstream_tests", [])
91109
exclusions = phase.get("upstream_test_exclusions", {})
92110
excluded_binaries = phase.get("excluded_binaries", [])
@@ -115,11 +133,31 @@ def load_plan(tier: str, plan_path: Path = PLAN_FILE) -> dict:
115133
for flag in ("all_binaries", "all_odh_tests", "include_image_provenance"):
116134
if flag in phase and not isinstance(phase[flag], bool):
117135
raise ValueError(f"phase {name!r} {flag} must be a boolean")
118-
if not binaries and not odh_filter and not phase.get("all_binaries") and not phase.get("all_odh_tests") and not phase.get("odh_only") and not phase.get("include_image_provenance"):
136+
has_nextest_selection = (
137+
binaries
138+
or exclusions
139+
or excluded_binaries
140+
or odh_filter
141+
or excluded_odh_filters
142+
or any(
143+
phase.get(flag, False)
144+
for flag in (
145+
"all_binaries",
146+
"all_odh_tests",
147+
"odh_only",
148+
"include_image_provenance",
149+
)
150+
)
151+
)
152+
if "nextest" not in runners and has_nextest_selection:
153+
raise ValueError(
154+
f"phase {name!r} declares a nextest selection but runners omit nextest"
155+
)
156+
if "nextest" in runners and not binaries and not odh_filter and not phase.get("all_binaries") and not phase.get("all_odh_tests") and not phase.get("odh_only") and not phase.get("include_image_provenance"):
119157
raise ValueError(f"phase {name!r} selects no tests")
120158
if "odh_only" in phase and not isinstance(phase["odh_only"], bool):
121159
raise ValueError(f"phase {name!r} odh_only must be a boolean")
122-
resolved.append({**phase, "name": name, "mode": mode_name, "upstream_tests": binaries,
160+
resolved.append({**phase, "name": name, "mode": mode_name, "runners": runners, "upstream_tests": binaries,
123161
"upstream_test_exclusions": exclusions, "odh_filter": odh_filter,
124162
"global_test_exclusions": global_exclusions,
125163
"all_odh_tests": phase.get("all_odh_tests", phase.get("odh_only", False)),
@@ -147,6 +185,8 @@ def load_mode(mode_name: str, plan_path: Path = PLAN_FILE) -> dict:
147185

148186

149187
def nextest_filter(phase: dict) -> str:
188+
if "nextest" not in phase.get("runners", ["nextest"]):
189+
raise ValueError(f"phase {phase['name']!r} runs no nextest selection")
150190
terms = []
151191
odh_filter = phase["odh_filter"]
152192
if odh_filter:

‎e2e/rust/tests/odh/tiers.toml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,10 @@ description = "Medium/low priority positive tests"
4545
upstream_tests = ["workspace_lifecycle"]
4646
odh_filter = "tier2::"
4747
excluded_odh_filters = ["tier2::managed_identity::"]
48+
# The Python gateway suite rides tier2's shared deployment: the DevTestOps
49+
# CI only runs smoke through tier3, so the Python lane would never execute
50+
# in CI unless it shares a tier.
51+
runners = ["nextest", "python"]
4852

4953
[tier2.managed]
5054
description = "Managed-workspace lifecycle tests"
@@ -56,6 +60,13 @@ description = "Negative and destructive tests"
5660
upstream_tests = []
5761
odh_filter = "tier3::"
5862

63+
[python]
64+
description = "Supported upstream Python gateway E2E on OpenShift"
65+
# Single-phase tier over the shared gateway mode; the python runner writes
66+
# the e2e-odh-python-shared-python.xml phase report. Gate the unsupported tests
67+
# through odh/python-gates.toml (validated strictly before deployment).
68+
runners = ["python"]
69+
5970
[odh]
6071
description = "All ODH-specific tests"
6172
odh_only = true

‎e2e/rust/tests/workspace_namespace_managed.rs‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -227,9 +227,10 @@ async fn managed_creates_namespace_with_labels() {
227227
"openshell"
228228
);
229229
assert_eq!(policy["spec"]["ingress"][0]["ports"][0]["port"], 2222);
230+
let gateway_namespace = std::env::var("NAMESPACE").unwrap_or_else(|_| "openshell".into());
230231
assert_eq!(
231232
policy["spec"]["ingress"][0]["from"][0]["namespaceSelector"]["matchLabels"]["kubernetes.io/metadata.name"],
232-
"openshell"
233+
gateway_namespace
233234
);
234235
assert!(
235236
policy["spec"]["ingress"][0]["from"][0]["podSelector"]["matchLabels"]

‎mise.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -74,7 +74,7 @@ DOCKER_BUILDKIT = "1"
7474

7575
[vars]
7676
# Python paths to include in formatting/linting
77-
python_paths = "python/ tasks/scripts/*.py e2e/mcp-conformance/*.py deploy/sbom/*.py deploy/konflux/e2e-odh/*.py"
77+
python_paths = "python/ tasks/scripts/*.py e2e/mcp-conformance/*.py deploy/sbom/*.py deploy/konflux/e2e-odh/*.py odh/scripts/*.py"
7878

7979
[task_config]
8080
includes = ["tasks/*.toml"]

‎odh/python-gates.toml‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
# Upstream Python gateway E2E that cannot yet run on the downstream OpenShift
5+
# lane. Each entry is an exact upstream node ID (without parametrization), so
6+
# an upstream rename or removal fails the entrypoint's collection preflight.
7+
# OIDC tests retain their own upstream skip conditions and are owned by the
8+
# separate downstream Keycloak/OIDC lane.
9+
10+
[[gate]]
11+
category = "docker-host"
12+
reason = "requires a fake inference endpoint on the test host, which sandbox pods cannot reach through host.openshell.internal"
13+
tests = [
14+
"e2e/python/test_sandbox_providers.py::test_imported_openai_profile_allows_native_endpoint_with_attached_provider",
15+
"e2e/python/test_sandbox_providers.py::test_imported_anthropic_profile_allows_native_endpoint_with_attached_provider",
16+
]
17+
18+
[[gate]]
19+
category = "egress"
20+
reason = "requires sandbox egress to PyPI or GitHub, which downstream CI clusters do not guarantee"
21+
tests = [
22+
"e2e/python/test_sandbox_venv.py::test_pip_install_in_sandbox",
23+
"e2e/python/test_sandbox_venv.py::test_uv_pip_install_in_sandbox",
24+
"e2e/python/test_sandbox_venv.py::test_uv_run_with_ephemeral_dependency",
25+
"e2e/python/test_sandbox_providers.py::test_github_provider_allows_https_git_clone",
26+
]
27+
28+
[[gate]]
29+
category = "workload-image"
30+
reason = "requires Python or /sandbox/.venv in the sandbox; the current OpenShift workload is Ubuntu, while upstream builds e2e/python/Dockerfile.workload locally"
31+
tests = [
32+
"e2e/python/test_sandbox_exec_python.py::test_exec_python_accepts_bound_methods",
33+
"e2e/python/test_sandbox_exec_python.py::test_exec_python_surfaces_callable_errors",
34+
"e2e/python/test_sandbox_landlock.py::test_landlock_blocks_write_to_read_only_path",
35+
"e2e/python/test_sandbox_landlock.py::test_landlock_allows_write_to_read_write_path",
36+
"e2e/python/test_sandbox_landlock.py::test_landlock_allows_read_on_read_only_path",
37+
"e2e/python/test_sandbox_landlock.py::test_landlock_blocks_access_outside_policy",
38+
"e2e/python/test_sandbox_api.py::test_sandbox_api_crud_and_exec",
39+
"e2e/python/test_sandbox_policy.py::test_policy_applies_to_exec_commands",
40+
"e2e/python/test_sandbox_policy.py::test_transparent_tcp_policy_denies_unauthorized_connections",
41+
"e2e/python/test_sandbox_providers.py::test_provider_credentials_available_as_env_vars",
42+
"e2e/python/test_sandbox_providers.py::test_endpointless_profile_credentials_fail_closed_without_policy_binding",
43+
"e2e/python/test_sandbox_providers.py::test_nvidia_provider_injects_nvidia_api_key_env_var",
44+
"e2e/python/test_sandbox_providers.py::test_endpointless_profile_credentials_use_explicit_policy_binding",
45+
"e2e/python/test_sandbox_providers.py::test_attach_detach_updates_credentials_for_later_exec_launches",
46+
"e2e/python/test_sandbox_venv.py::test_sandbox_venv_in_path",
47+
"e2e/python/test_sandbox_providers.py::test_google_metadata_sdk_discovery",
48+
]
49+
50+
[[gate]]
51+
category = "workload-fixture"
52+
reason = "upstream Python E2E builds and selects e2e/python/Dockerfile.workload; this OpenShift lane uses the product-default Ubuntu workload, so these no-image sandbox tests never reach Ready"
53+
tests = [
54+
"e2e/python/test_exec_admission.py::test_exec_request_timeout_keeps_launch_unresolved",
55+
"e2e/python/test_sandbox_api.py::test_mutation_replay_preserves_sandbox_lifecycle_and_replacement",
56+
]
57+
58+
[[gate]]
59+
category = "lifecycle-timeout"
60+
reason = "the shared Python fixture's 30-second delete RPC deadline is shorter than the Kubernetes driver's bounded Pod-termination wait; an upstream fixture-timeout correction is being validated before these tests are re-enabled"
61+
tests = [
62+
"e2e/python/test_exec_admission.py::test_exec_request_id_never_relaunches_or_replays_output",
63+
"e2e/python/test_exec_admission.py::test_exec_client_cancellation_does_not_clear_launch_admission",
64+
"e2e/python/test_policy_validation.py::test_update_policy_rejects_immutable_fields",
65+
"e2e/python/test_sandbox_api.py::test_sandbox_interactive_exec_honors_tty",
66+
"e2e/python/test_sandbox_api.py::test_interactive_exec_drains_output_after_request_eof",
67+
"e2e/python/test_sandbox_landlock.py::test_landlock_blocks_user_owned_path_outside_policy",
68+
"e2e/python/test_sandbox_providers.py::test_credentials_not_in_persisted_spec_environment",
69+
]

‎odh/requirements.in‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,4 +12,6 @@ grpcio-tools>=1.60
1212
mypy-protobuf>=5.1
1313
httpx>=0.27
1414
protobuf>=4.25
15-
pytest>=8.0
15+
pytest>=9.0.3
16+
pytest-asyncio>=0.23
17+
pytest-xdist>=3.0

0 commit comments

Comments
 (0)