Repository navigation
Commit 50a3237
authored
test(features): run feature tests against environment services, starting with Keycloak (#4891)
* feat(testkit): pass extra environment variables to the CLI
Add Cli::run_with_env and Context::cli_with_env so tests can hand
credentials to the CLI by name instead of on its command line.
Signed-off-by: Matthew Grossman <[email protected]>
* feat(tmachine): add an Ubuntu Docker environment that runs Keycloak
Environment setup can now take inputs, like installers do, and hashes
them only when present so existing cached disks stay valid.
ubuntu-docker-rootful-keycloak runs the Docker setup plus a Keycloak
playbook that starts the container with a restart policy, since setup is
cached as a disk, and writes how to reach it as OPENSHELL_TEST_* lines in
/etc/openshell-test/env.d/keycloak.env.
Signed-off-by: Matthew Grossman <[email protected]>
* feat(test-runner): pass the variables an environment publishes to tests
After boot, read every /etc/openshell-test/env.d/*.env file, accept only
OPENSHELL_TEST_* names, and add the variables to the testkit-ready and
nextest commands through guest_env. Errors name the key, never the value.
The runner doesn't know which services an environment runs; suites read
the variables, so they also run against a deployment that sets them.
Signed-off-by: Matthew Grossman <[email protected]>
* test(features): port Keycloak provider refresh to testkit
Add a keycloak features crate with a suite-local KeycloakContext fixture
that reads OPENSHELL_TEST_KEYCLOAK_*, waits for the realm to answer, and
provides password grants, revocation and the token endpoint. Port the
provider refresh test onto it. Align the features lockfile with the root
and add it to the alignment check. The old provider-refresh crate stays
until the required lanes switch.
Signed-off-by: Matthew Grossman <[email protected]>
* ci(tests): run the Keycloak features suite in E2E Suites
Add a matrix row for features/keycloak on ubuntu-docker-rootful-keycloak,
include the Keycloak realm in the tmachine disk cache key, and document
environments that provide services in TESTING.md.
Signed-off-by: Matthew Grossman <[email protected]>
* docs(testing): describe how to add a feature suite
Signed-off-by: Matthew Grossman <[email protected]>
* refactor(tmachine): publish test variables through a shared task file
Signed-off-by: Matthew Grossman <[email protected]>
* fix(test-runner): keep published test variables out of SSH commands
The runner passed the variables in env.d on the nextest command line, so
an SSH timeout or launch failure printed their values, which can be
credentials, in the error context. The runner now only checks the files;
the guest shell exports them before starting nextest, so only the gateway
name and CLI path appear in the command.
Lines are read literally on both sides: no trimming, and only empty lines
and lines starting with # are skipped.
Signed-off-by: Matthew Grossman <[email protected]>
* fix(features): fit Keycloak readiness in the per-test timeout
The fixture waited up to three minutes for Keycloak inside the test, but
the runner's nextest profile kills a test after two. Wait up to 90 seconds
so a slow Keycloak fails with a readiness error instead.
Signed-off-by: Matthew Grossman <[email protected]>
* test(test-runner): run the export script test with shell builtins only
The Nix build runs the runner's tests in a sandbox without env on PATH,
so the test failed there and the runner didn't build. Print the exported
variables with printf instead.
Signed-off-by: Matthew Grossman <[email protected]>
* refactor(features): name the Keycloak suite after what it tests
The suite was named after the service it needs rather than what it
tests. Rename it to keycloak-provider-refresh and move the Keycloak
fixture into a support crate at tests/support/keycloak, so other suites
that need an OIDC issuer can depend on it. TESTING.md now asks for a
descriptive suite name instead of prescribing one. Lint and format the
new crates in CI and mise.
Signed-off-by: Matthew Grossman <[email protected]>
* fix(features): bound each Keycloak request in the fixture
poll_until checks its deadline only between attempts, and curl had no
time limit, so a Keycloak that accepts connections but stalls could hold
one attempt until nextest killed the test. Give each request 10 seconds.
Signed-off-by: Matthew Grossman <[email protected]>
* ci(tests): key the tmachine disk cache by environment and installer
Both E2E suites rows computed the same cache key. Only the first job to
save it got its disks in, and actions/cache never saves after an exact
hit, so the other row rebuilt its setup and install layers on every run.
Signed-off-by: Matthew Grossman <[email protected]>
* fix(testkit): keep the gateway selection over caller CLI env
run_with_env applied the caller's variables last, so one named like the
gateway, endpoint or workspace variable could silently point the CLI at
another gateway. Apply them first so testkit's selection wins.
Signed-off-by: Matthew Grossman <[email protected]>
* fix(features): leave the test body room after Keycloak readiness
poll_until checks its deadline only between attempts, so a 90 second
wait could run for about 102 seconds with the request time limit and the
poll interval, leaving the test body under 20 seconds of the 2 minute
budget. Wait 75 seconds and spell out the margin.
Signed-off-by: Matthew Grossman <[email protected]>
* ci(tests): include config.nix in the tmachine disk cache key
tmachine's setup hash covers each environment's playbook list and setup
inputs, which config.nix declares. Changing only config.nix left the
cache key unchanged, so the rebuilt layers were never saved again.
Signed-off-by: Matthew Grossman <[email protected]>
* docs(testing): add the lint step for new feature suites
CI formats and lints feature suites one package at a time, so a suite
added by following TESTING.md got no fmt or clippy coverage. Also say
what test_env actually guarantees about published values.
Signed-off-by: Matthew Grossman <[email protected]>
* test(features): report Keycloak suite cleanup failures
The provider and profile cleanup discarded both errors and failed exit
codes, so leftovers went unnoticed. Print them without failing the test.
Signed-off-by: Matthew Grossman <[email protected]>
---------
Signed-off-by: Matthew Grossman <[email protected]>1 parent 73ae222 commit 50a3237
24 files changed
Lines changed: 5758 additions & 446 deletions
File tree
- .github/workflows
- tasks
- scripts
- tests
- runner/src
- suites/features
- keycloak-provider-refresh
- tests
- support
- keycloak
- src
- testkit/testkit/src
- tmachine
- ansible
- playbooks
- roles/keycloak/tasks
- src
- qemu
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
234 | 234 | | |
235 | 235 | | |
236 | 236 | | |
| 237 | + | |
| 238 | + | |
237 | 239 | | |
238 | 240 | | |
239 | 241 | | |
| |||
296 | 298 | | |
297 | 299 | | |
298 | 300 | | |
| 301 | + | |
| 302 | + | |
299 | 303 | | |
300 | 304 | | |
301 | 305 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
50 | 50 | | |
51 | 51 | | |
52 | 52 | | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
53 | 58 | | |
54 | 59 | | |
55 | 60 | | |
| |||
77 | 82 | | |
78 | 83 | | |
79 | 84 | | |
80 | | - | |
81 | | - | |
82 | | - | |
83 | | - | |
84 | | - | |
85 | | - | |
86 | | - | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
87 | 96 | | |
88 | 97 | | |
89 | 98 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
230 | 230 | | |
231 | 231 | | |
232 | 232 | | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
233 | 290 | | |
234 | 291 | | |
235 | 292 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
40 | 40 | | |
41 | 41 | | |
42 | 42 | | |
| 43 | + | |
| 44 | + | |
43 | 45 | | |
44 | 46 | | |
45 | 47 | | |
| |||
56 | 58 | | |
57 | 59 | | |
58 | 60 | | |
| 61 | + | |
| 62 | + | |
59 | 63 | | |
60 | 64 | | |
61 | 65 | | |
| |||
71 | 75 | | |
72 | 76 | | |
73 | 77 | | |
| 78 | + | |
| 79 | + | |
74 | 80 | | |
75 | 81 | | |
76 | 82 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
| 38 | + | |
38 | 39 | | |
39 | 40 | | |
| 41 | + | |
40 | 42 | | |
41 | 43 | | |
42 | 44 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
10 | | - | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
11 | 13 | | |
12 | 14 | | |
13 | 15 | | |
| |||
19 | 21 | | |
20 | 22 | | |
21 | 23 | | |
| 24 | + | |
22 | 25 | | |
23 | 26 | | |
24 | 27 | | |
| |||
225 | 228 | | |
226 | 229 | | |
227 | 230 | | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
228 | 234 | | |
229 | 235 | | |
230 | 236 | | |
| |||
289 | 295 | | |
290 | 296 | | |
291 | 297 | | |
292 | | - | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
293 | 301 | | |
294 | | - | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
295 | 307 | | |
296 | 308 | | |
297 | 309 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| 20 | + | |
20 | 21 | | |
21 | 22 | | |
22 | 23 | | |
| |||
0 commit comments