Skip to content

Commit b33017a

Browse files
authored
ci(tui): run the TUI tests with cargo nextest (#31)
* ci(tui): run the TUI tests with cargo nextest The vendored tests were written for cargo nextest (one process per test), and some say in a comment that they are reliable only under it. TUI CI ran cargo test, where all tests of a binary share one process. - scripts/tui_baseline.toml: `[test] runner = "nextest"`; `--runner {cargo,nextest}` on the command line takes the other one. The libtest path stays as it was. - scripts/tui_baseline.py: `parse_nextest_output` reads the Summary line (nextest counts the ignored tests and the tests removed by --skip together as "skipped") and the recap of the failed tests (the live part has the captured output of a test, so it is not read); `suite_command` builds the command line; the report names the runner; a missing cargo-nextest stops the run at once, with a hint. - stream(): after the timeout, SIGTERM to the process group, and SIGKILL 20 s later. nextest puts each test in a process group of its own, so a SIGKILL to nextest left a hanging test alive (measured on a demo crate); a nextest that gets a SIGTERM stops its tests. - tui-ci.yml: install cargo-nextest 0.9.148 (taiki-e/install-action) in the test job; the job is now called "cargo nextest (Linux x86_64)". - 21 new unit tests. The fixtures are real output of nextest 0.9.148. * docs(tui): say in AGENTS.md and the tui_envspy README that TUI CI runs nextest * docs: record the nextest change and the r18 round - Plan (status, section 10 index, revision log), execution log (new section 10.21), CHANGELOG, baselines and baseline history: the numbers of the first nextest run on the whole tree (TUI CI run 39359879537: 8,490 passed, 0 failed, 22 skipped), the test time (about 130 s with nextest, about 65 s with cargo test, same code), and the dry run of release.yml for PR #29. - Correct the r17 notes: the 7 / 4 / 3 compiles of pi-tools-api in the cargo test job were not caused by the protoc path (PR #30 shows it), and the runner never ran doctests, so nextest loses nothing there. - Remove a duplicated "verification" heading from the r17 text.
1 parent d2182d9 commit b33017a

11 files changed

Lines changed: 573 additions & 62 deletions

File tree

‎.github/workflows/tui-ci.yml‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -87,7 +87,7 @@ jobs:
8787
path: tui-baseline/
8888

8989
test:
90-
name: cargo test (Linux x86_64)
90+
name: cargo nextest (Linux x86_64)
9191
runs-on: ubuntu-24.04
9292
timeout-minutes: 150
9393
steps:
@@ -116,6 +116,14 @@ jobs:
116116
shared-key: tui-linux-x86_64-test
117117
save-if: ${{ github.ref == 'refs/heads/main' }}
118118

119+
# The runner is `[test] runner` in scripts/tui_baseline.toml: cargo-nextest, one process per
120+
# test (the vendored tests were written for it). The version is pinned because
121+
# scripts/tui_baseline.py reads its output; change it together with the parser and its tests.
122+
- name: Install cargo-nextest
123+
uses: taiki-e/install-action@v2
124+
with:
125+
126+
119127
# The pager's client against the real Python agent (tui/crates/codegen/pi-pager/tests/
120128
# python_agent_e2e.rs, plan 0.3): the tests need a Python that has the agent installed, and
121129
# fail without one (PI_E2E_REQUIRED) instead of skipping, so a CI that lost it is not green.
@@ -125,8 +133,8 @@ jobs:
125133
echo "PI_E2E_PYTHON=$(command -v python)" >> "$GITHUB_ENV"
126134
echo "PI_E2E_REQUIRED=1" >> "$GITHUB_ENV"
127135
128-
# Report-only until `[test] enforce = true` in scripts/tui_baseline.toml (after the first
129-
# Linux run has been triaged); the per-suite table lands in the job summary either way.
136+
# What blocks (`[test] enforce`, the per-suite floors) is in scripts/tui_baseline.toml; the
137+
# per-suite table lands in the job summary either way.
130138
- name: Test suites
131139
run: python scripts/tui_baseline.py test --out tui-baseline
132140

‎AGENTS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ AgentMessage[] → transform_context() → convert_to_llm() → LangChain BaseMe
5050

5151
## Development
5252

53-
Python monorepo (`pi-agent-core`, `pi-agent-harness`, `pi-agent-cli`) + Rust TUI workspace `tui/` (vendored grok-build fork, binary `zypi`). Packages installed editable; tested via `pytest`. TUI: `cd tui && cargo check -p pi-pager-bin` (WSL; Cargo reads `CARGO_TARGET_DIR` from WSL env exclusively). Rust tests run through `python scripts/tui_baseline.py test` (manifest `scripts/tui_baseline.toml`, numbers in `docs/baselines/tui.md`); it clears `PI_HOME` and friends, which a WSL login shell may export pointing at the real home — do not run test binaries or `zypi` without clearing them. The run does not isolate `HOME` (plan D.9): some tests write into the real `~/.pi-python` and `/tmp`, so check both after a local run. Build on the Linux file system, not `/mnt/<drive>`. With a built `zypi`, `scripts/tui_pty/` drives it under a pseudo-terminal on Linux (smoke test, exit matrix, `-p` shutdown, the matrix of flags that resume a session, a probe of which permission modes the agent honours, a network audit (it shows that `zypi` itself opens no connection to a non-local address during a turn), a probe of the status line; README there). Two tools read the TUI tree: `python scripts/tui_callgraph.py keep --out docs/baselines/tui-support.md` regenerates the list of what the support crates still use (it reads source text only; the list is an estimate, `cargo check` has the last word), and `python scripts/tui_notices.py check` (needs cargo) says whether `tui/THIRD-PARTY-NOTICES` still matches `Cargo.lock`; run it after a dependency change, then `update --write`. The pager's ACP client is also run against the real Python agent without a terminal: `tui/crates/codegen/pi-pager/tests/python_agent_e2e.rs`, which needs `PI_E2E_PYTHON` (a Python with `pi_agent_cli` installed; TUI CI sets it and `PI_E2E_REQUIRED=1`, locally an unset variable makes the tests skip with a note). The tests of one Rust test binary run as threads of one process (CI: `cargo test`, 4 threads, a fixed order), and they share the environment variables and every `static`; the vendored tests were written for `cargo nextest` (one process per test), so their locks are partial. A test that sets such state can break another test in some runs only. `scripts/tui_envspy/` finds these tests (an `LD_PRELOAD` spy for environment variables and a shuffled many-thread stress script; README there). Every test that writes or reads one variable must hold the one lock that the other tests of that variable hold. A test that changes a process-wide `static` that other tests read without a lock (the theme, the colour level, the minimal-mode flag, the bidi latch) runs its body in a process of its own: `crate::test_util::in_own_process(|| { … })` in `pi-pager`.
53+
Python monorepo (`pi-agent-core`, `pi-agent-harness`, `pi-agent-cli`) + Rust TUI workspace `tui/` (vendored grok-build fork, binary `zypi`). Packages installed editable; tested via `pytest`. TUI: `cd tui && cargo check -p pi-pager-bin` (WSL; Cargo reads `CARGO_TARGET_DIR` from WSL env exclusively). Rust tests run through `python scripts/tui_baseline.py test` (manifest `scripts/tui_baseline.toml`, numbers in `docs/baselines/tui.md`); it runs `cargo nextest` (one process per test, as the vendored tests expect; install `cargo-nextest`, or pass `--runner cargo` for libtest), and it clears `PI_HOME` and friends, which a WSL login shell may export pointing at the real home — do not run test binaries or `zypi` without clearing them. The run does not isolate `HOME` (plan D.9): some tests write into the real `~/.pi-python` and `/tmp`, so check both after a local run. Build on the Linux file system, not `/mnt/<drive>`. With a built `zypi`, `scripts/tui_pty/` drives it under a pseudo-terminal on Linux (smoke test, exit matrix, `-p` shutdown, the matrix of flags that resume a session, a probe of which permission modes the agent honours, a network audit (it shows that `zypi` itself opens no connection to a non-local address during a turn), a probe of the status line; README there). Two tools read the TUI tree: `python scripts/tui_callgraph.py keep --out docs/baselines/tui-support.md` regenerates the list of what the support crates still use (it reads source text only; the list is an estimate, `cargo check` has the last word), and `python scripts/tui_notices.py check` (needs cargo) says whether `tui/THIRD-PARTY-NOTICES` still matches `Cargo.lock`; run it after a dependency change, then `update --write`. The pager's ACP client is also run against the real Python agent without a terminal: `tui/crates/codegen/pi-pager/tests/python_agent_e2e.rs`, which needs `PI_E2E_PYTHON` (a Python with `pi_agent_cli` installed; TUI CI sets it and `PI_E2E_REQUIRED=1`, locally an unset variable makes the tests skip with a note). Under `cargo test`, the tests of one Rust test binary run as threads of one process (4 threads, a fixed order), and they share the environment variables and every `static`; the vendored tests were written for `cargo nextest` (one process per test), so their locks are partial. A test that sets such state can break another test in some runs only. TUI CI and `scripts/tui_baseline.py` run nextest now; a local `cargo test` still shows these races. `scripts/tui_envspy/` finds these tests (an `LD_PRELOAD` spy for environment variables and a shuffled many-thread stress script; README there). Every test that writes or reads one variable must hold the one lock that the other tests of that variable hold. A test that changes a process-wide `static` that other tests read without a lock (the theme, the colour level, the minimal-mode flag, the bidi latch) runs its body in a process of its own: `crate::test_util::in_own_process(|| { … })` in `pi-pager`.
5454

5555
### Virtual environments (uv)
5656

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
1616
- **`zypi --continue`, `--resume` and `--session-id` ask the agent which session is meant (TUI)**: `zypi` used to look for the session in the session files of the old Rust agent. The Python agent writes another layout, so `--resume <title>` found no session, an unknown id failed only after the UI had started, a used `--session-id` left the UI open with `Session creation failed: Invalid params`, and `--continue` took the session that was created last (the first of the **Known issues** of 0.5.0). Now `zypi` starts the agent first and then asks it over ACP, with `session/list` (page after page) and `session/load`. It reads no session file for this. A session that does not exist ends the run before the event loop starts: the terminal is restored, the agent is stopped, the message is printed, and the exit status is 1. **Behaviour changes**: `--continue` and `--resume` without a value take the session of this directory with the latest activity (the time of its last write), not the one that was created last. `--resume <id>` finds the id in every directory and loads the session in the directory it belongs to. `--resume <title>` matches the title, which is the first message of the session, among the sessions of this directory, without regard to letter case and to spaces at the ends; two sessions with the same title are an error that lists their ids and last activity. `--session-id <id>` reads every page of the session list and refuses an id that any session has (`Session ID <id> is already in use.`, exit status 1); that costs one pass through the list (`scripts/session_list_bench.py` prints it as `walk all`), and an agent that offers no `session/list` skips the check and refuses a used id itself. The lookup stops after 30 s, the limit of the connect (`GROK_CONNECT_UI_TIMEOUT_SECS`). A session in which nobody typed has no file now, so `--continue` cannot take it (see the next entry). **Not solved, and left as it is**: the agent lists its session directory again for each page, so reading all pages costs pages × directory size, which grows faster than the number of sessions. `scripts/session_list_bench.py` (median of 3, Linux / Windows): `--session-id` takes 1.3 s / 1.8 s at 1,000 sessions and 28 s / 34 s at 5,000, which is over the 30 s limit on Windows; `--continue` takes 0.6 s / 0.8 s at 1,000 and 10 s / 12 s at 5,000. The owner decided that this is fine below 1,000 sessions (plan decision 18). Tests: 25 unit tests of the lookup against a scripted agent, 4 of the time limit and of a cancel, 3 end-to-end tests against the real agent, and `scripts/tui_pty/resume_matrix.py --strict` (13 of 13 pass; cases 5 and 11 were known gaps, and case 8 now needs exit status 1).
1717
- **The agent writes the file of a session with its first message (`pi-agent-cli`, `pi-agent-harness`)**: the agent used to write the file of a session when it answered `session/new`. A run in which nobody typed a message left a file with the title `(no messages)`. That file stayed in the `/resume` list, and `zypi --continue` took it when it was the latest session of the directory. Now `session/new` makes the session in memory only. The file is written with the first message of the user, in one write: the header, the entries that were made before the message (for example a model change), and the message. A session in which nobody typed has no file. `session/list` does not show it, and `session/load` and `session/resume` answer `resource_not_found` for it. A client that has the session open can keep using it. **Behaviour changes**: `zypi` with no flag starts no session until the first message; `zypi --session-id <id>` and `/new` start one at once, with no file. An id that a client chose (`_meta.sessionId`) stays reserved while its session is open in the agent, so a second `session/new` with that id is refused (`invalid_params`) although no file has it. If another agent process saves a session with that id first, the first prompt of the session is refused (`a session with this id already exists`). This narrows the gap between two agent processes. It does not close it. Files that earlier versions wrote keep the title `(no messages)`. **For library users**: `JsonlSessionRepo.create({"deferred": True})` returns a `DeferredJsonlSessionStorage`, and its `written` property says whether the file exists. `create` without `deferred`, and `fork`, write at once as before. Tests: 10 for the storage (what is written and in which order, a failed write and its retry, two first messages at once), 6 new ones for the agent (the ACP contract and the chosen ids), and the older tests that expect a session in the list now send a message first. `scripts/tui_pty/resume_matrix.py --strict` has 16 cases (13 before). Two of the three new ones fail against an agent that writes at once: `--session-id` and `/new`, each left with no message. The third types a message after `/new`.
1818
- **A resumed session no longer takes the sandbox profile it was created with (TUI)**: the old Rust agent saved the sandbox profile in the files of a session. `zypi --continue` and `--resume` read it back before the sandbox was applied, so a resumed session kept its sandbox, and `--sandbox X` on a session that was saved with `Y` was refused (`cannot resume this session under sandbox profile ...`). The Python agent never saved a profile, so for its sessions nothing changes. The profile now comes from `--sandbox`, from `GROK_SANDBOX` and from the configuration, as it does for a new session. The code that read it is gone (decision 2 of the plan).
19+
- **`TUI CI` and `scripts/tui_baseline.py` run the Rust tests with `cargo nextest` (TUI)**: `cargo nextest` runs each test in a process of its own. `cargo test` runs all tests of a test binary as threads of one process, where they share the environment variables, the working directory and every `static`. The vendored tests were written for nextest, and some of them are reliable only under it. So the failures of this class cannot occur in CI now, also for a test that someone writes later. `scripts/tui_baseline.toml` has `[test] runner = "nextest"`, and the option `--runner cargo` takes libtest back. The runner reads the output of nextest (the `Summary` line, and the list of the tests that did not pass), so CI installs a pinned version (`[email protected]`, with `taiki-e/install-action`); the job is now named `cargo nextest (Linux x86_64)`. Three differences to know. nextest counts the `#[ignore]` tests and the tests that a `--skip` filter removes together as skipped (22 for the whole list, where `cargo test` printed 21 ignored). nextest does not run doctests, and these suites never ran them (the runner selects explicit `--lib`, `--test` and `--bin` targets). A suite that hangs is stopped with SIGTERM first and with SIGKILL after 20 s: nextest puts each test in a process group of its own, so SIGKILL alone left the hanging test running. 21 new `pytest` tests cover the parser, the commands, the choice of the runner and the stop signals. **CI result**: the first run of the whole tree under nextest passed: 8,490 passed, 0 failed, 22 skipped, and every suite equals its reference number (the 14 tests of `python_agent_e2e` ran too). The tests take about 130 s (the `Summary` times of nextest, added up), where `cargo test` took about 65 s for the same code. The job took 36 min 17 s, against 33 min 56 s for a `cargo test` job that started 4 minutes earlier. The time of the job varies from 17 to 34 min between runs of `cargo test`, so the added test time (about 65 s) is the cost that this change causes.
1920

2021
### Removed
2122

0 commit comments

Comments
 (0)