Skip to content

Epic: Migrate legacy bash E2E into the Vitest E2E system #5098

New issue
New issue
Closed
2 of 2 issues completed
#5361
#5357
Closed
Epic: Migrate legacy bash E2E into the Vitest E2E system#5098
2 of 2 issues completed
#5361
#5357
Assignees
cv
Labels
area: e2eEnd-to-end tests, nightly failures, or validation infrastructuresprint 6Sprint 6

Description

@jyaunches
jyaunches
opened on Jun 10, 2026

Goal

Migrate every direct legacy bash E2E entry point under test/e2e/test-*.sh into the typed Vitest E2E system, then delete or retire the legacy shell entry points as cleanup.

The end state is:

  • E2E tests are TypeScript/Vitest tests.
  • Real system boundaries are still tested when they are the contract: Vitest may invoke bash, bash -lc, installer scripts, host commands, sudo-gated commands, process signals, /proc probes, Docker/OpenShell commands, and sandbox exec.
  • Shared setup, clients, shell probes, process supervision, artifacts, cleanup, and redaction live in Vitest fixtures/support code.
  • Legacy bash scripts are not kept as a second durable E2E suite.

Core Principle

Do not classify tests as "keep bash" just because they need shell behavior.

If a contract needs shell, process, host, platform, installer, or sandbox behavior, preserve that boundary from a typed Vitest test:

  • installer fidelity: run bash install.sh ... from Vitest;
  • shell profile behavior: run bash -lc or explicitly source profile files from Vitest;
  • /proc and kernel/process probes: run the probe in the same host or sandbox context from Vitest;
  • signal handling: use Node process control or host shell commands from Vitest;
  • privileged/platform checks: use guarded host exec from Vitest and skip with evidence when unavailable;
  • sandbox context: use typed sandbox clients / openshell sandbox exec wrappers from Vitest;
  • full user journeys: keep them as full-flow Vitest tests instead of splitting away their integration value.

Definition of Done for a Conversion

A legacy script is considered converted when all of the following are true:

  1. Equivalent typed Vitest coverage exists under the E2E Vitest tree or an appropriate focused Vitest test.
  2. The new test preserves the same user-visible contract, including shell/system boundaries where those boundaries matter.
  3. The test has deterministic artifacts, cleanup, timeout behavior, and secret redaction.
  4. The PR explains the contract mapping from the legacy shell script to the new Vitest coverage.
  5. The replacement test is wired into .github/workflows/e2e-vitest-scenarios.yaml in the conversion PR so it can be dispatched on the same runner class as the legacy lane.
  6. Any legacy shell workflow references that must be removed are removed, replaced, or explicitly deferred to Phase 11 in the same PR.

Shell Deletion Deferral and Governance

Default conversion PRs should defer legacy shell script deletion unless the PR is explicitly scoped to retire the script and its workflow lane. A script not being deleted is not by itself a reason to leave the conversion unchecked.

Deletion/quarantine of converted shell scripts is intentionally deferred to Phase 11 cleanup so parallel conversion PRs can land with less workflow and allowlist churn. During conversion phases:

  • do not add new legacy test/e2e/test-*.sh entry points;
  • do not expand or casually rewrite existing legacy shell scripts;
  • do not add new nightly/regression workflow wiring that runs legacy shell scripts unless maintainers intentionally approve it;
  • every conversion PR must add the replacement Vitest execution path to .github/workflows/e2e-vitest-scenarios.yaml before it is considered converted;
  • free-standing live/process Vitest replacements must get a discrete dispatchable job in e2e-vitest-scenarios.yaml unless they are registered as a typed live scenario in the existing matrix;
  • do not rely on e2e-scenarios-all / empty scenarios dispatch as proof for a newly added free-standing test unless the PR also wires that job into e2e-vitest-scenarios.yaml;
  • move legacy shell workflow lanes to Vitest only when a conversion PR intentionally replaces that execution path; otherwise leave stable shell wiring for Phase 11 cleanup.

Repository governance exists to enforce this freeze:

  • test/e2e-script-workflow.test.ts freezes the top-level legacy shell script allowlist;
  • the same test freezes scheduled nightly legacy shell wiring and verifies referenced files still exist;
  • the test is included in the cli Vitest project and runs through PR cli-test-shards / main cli-test-shards.

If a PR intentionally changes the legacy shell set or nightly shell wiring, it must update the allowlist/contract test with an explicit migration rationale.

Parallelization Model

The remaining work should be run as parallel tracks, not a single linear phase ladder.

  • Phase 1 anchor PRs can start in parallel across tracks.
  • Within a track, dependent migrations should wait until that track's anchor PR has established the helper/test shape, but they do not need to wait for anchors in other tracks.
  • Prefer one owner per anchor to avoid competing helper designs.
  • Avoid broad shared-fixture or workflow refactors inside individual conversions.
  • Use local helpers first. Promote helpers to shared fixtures only when 3+ migrations clearly need the same boundary.
  • Minimize workflow YAML contention, but do not skip required Vitest wiring: each migration still needs its own smallest safe e2e-vitest-scenarios.yaml dispatch path.
  • Use unique sandbox names, ports, artifact directories, and workflow job names in parallel PRs.

Phase Tracker

Legend: checked means equivalent Vitest conversion coverage has landed. Line counts are sizing hints from the legacy script inventory.

Phase 0 — Converted / already covered

Phase 1 — Parallel anchor PRs

Start these in parallel. Each anchor should establish the smallest useful local/helper shape for its track, then dependent scripts in that track can fan out.

Phase 2 — Independent quick wins / no anchor dependency

These should not need a large shared helper or anchor PR. Convert opportunistically in parallel.

Phase 3 — Onboarding and baseline dependents

Start after the onboarding anchor shape exists. These may share onboard/run/resume/repair helpers but can be separate PRs.

Phase 4 — Sandbox and gateway dependents

Start after the sandbox/gateway anchor shape exists. These can run in parallel with inference, messaging, Hermes, security, rebuild, and platform tracks.

Phase 5 — Inference and provider dependents

Start after the inference/provider anchor shape exists. Mock-provider and live-provider slices can be split across owners.

Phase 6 — Messaging, channels, and pairing dependents

Start after the messaging anchor shape exists. Avoid making test-messaging-providers.sh the first PR; split provider/channel slices where possible.

Phase 7 — Hermes dependents

Start after the Hermes anchor shape exists. Keep Hermes-specific helpers separate from OpenClaw helpers unless repeated boundaries prove otherwise.

Phase 8 — Security and policy dependents

Start after the security/policy anchor shape exists, except the independent quick wins already listed in Phase 2.

Phase 9 — Rebuild, state, and runtime dependents

Start after the rebuild/state anchor shape exists. Runtime-only scripts that stay local can proceed independently if they do not need rebuild helpers.

Phase 10 — Platform and resource-constrained dependents

These are valid parallel work, but they need explicit runner/resource planning and should not block non-platform tracks.

Phase 11 — Review, refactor, and shell retirement cleanup

Phase 11 cleanup is being handled by PR #5756. It retires the legacy shell workflow/action lanes, removes converted top-level test/e2e/test-*.sh entrypoints, consolidates the surviving E2E target/live surface, promotes the repeated shell quoting helper, and adds retired-entrypoint workflow boundary tests. Check these boxes after #5756 lands.

Run this after enough anchor/dependent migrations have landed to see real duplication patterns.

  • Review all migrated tests for repeated local helpers used by 3+ migrations.
  • Promote repeated helper patterns into shared Vitest fixture/support locations with focused tests.
  • Remove redundant local helper copies after shared helpers exist.
  • Delete or quarantine converted legacy shell scripts that remain present only for transition.
  • Remove stale shell workflow wiring and update existing workflow/allowlist contract tests.
  • Confirm no durable legacy bash suite remains beyond intentionally documented platform/manual exceptions.

Tracking Guidance

Use this issue for migration order, parallelization guidance, and track ownership. Use individual PRs for exact assertion mapping, validation evidence, e2e-vitest-scenarios.yaml wiring, same-runner dispatch evidence, and script deletion/retirement notes. When a PR lands equivalent Vitest coverage with a dispatchable Vitest workflow path, link it here and mark the item complete even if deletion is deferred to Phase 11 cleanup.

Activity

  1. jyaunches
    added
    area: e2eEnd-to-end tests, nightly failures, or validation infrastructure
    on Jun 10, 2026
  2. jyaunches
    added a commit that references this issue on Jun 10, 2026

    Initial cutover plan placeholder

    3570edf
  3. cv
    added 4 commits that reference this issue on Jun 10, 2026

    Initial cutover plan placeholder

    55c4998

    ci(e2e): route scenario advisor to Vitest workflow (#5105)

    56cbecf

    test(e2e): typed-shell-runner cutover (parity → retirement) (#5106)

    82499f8

    test(e2e): retire docker-unreachable gateway script (#5119)

    44376ec
  4. jyaunches
    added 9 commits that reference this issue on Jun 10, 2026

    test(e2e): add SandboxClient.get and SandboxClient.exists helpers

    8705dbb

    ci(e2e): add onboard-resume-vitest job to e2e-vitest-scenarios

    49755fa

    test(e2e): add typed live test for onboard-resume regression (#446)

    75d5b2c

    test(e2e): add SandboxClient.upload + execShell for live test fixtures

    3f7c2b5

    test(e2e): add openclaw-tui-chat-correlation free-standing live test …

    fd0f7bd

    test(e2e): pass framework env to onboard-resume spawns to fix ENOENT

    debb4b2

    test(e2e): pass framework env to sandbox.exists() in onboard-resume

    3827a96

    fix(e2e): SandboxClient.exec must use `-n <name>` flag form, not posi…

    bf1a9a2

    test(e2e): mark test-onboard-resume.sh as covered after typed migration

    4577fc5

  5. 119 remaining items

  6. jyaunches
    added a commit that references this issue on Jun 25, 2026

    test(e2e): restore messaging bash parity (#5806)

    858be9a
  7. cv
    added a commit that references this issue on Jun 26, 2026

    test(e2e): close platform and workflow parity decisions (#5823)

    38964bd
  8. cv
    added a sub-issue on Jun 28, 2026

  9. cv commented on Jun 28, 2026

    @cv
    cv
    on Jun 28, 2026
    Collaborator

    Phase 11 cutover acceptance is now tracked in #5919, linked as a sub-issue of this epic.

    #5919 is the only cutover tracker. It operates at old workflow-job variant granularity, requires reviewed Equivalent/Subsumed/Moved/Retired dispositions plus paired run evidence, and explicitly forbids adding a migration ledger or parity tracker to the repository.

    The implementation path is intentionally incremental: missing Vitest variants, scheduled observability, and current-state ratchets land as small PRs on main while the old nightly still exists. #5756 then merges those changes and remains the terminal legacy deletion/rename PR. This epic and #5919 close together only after the final cutover and burn-in gates pass.

  10. github-actions
    assigned
    cv
    on Jun 28, 2026
  11. cv
    added a commit that references this issue on Jun 30, 2026

    test(e2e): retire legacy shell lanes (#5756)

    8120223

  12. jyaunches commented on Jul 9, 2026

    @jyaunches
    jyaunches
    on Jul 9, 2026
    ContributorAuthor

    Closing as completed: both sub-issues (#5331 and #5919) are closed, and #5919 was the terminal cutover tracker for the legacy-to-Vitest E2E migration.

  13. jyaunches
    closed this as completedon Jul 9, 2026
  14. Hadar301
    added 7 commits that reference this issue on Jul 12, 2026

    test(e2e): restore onboard rebuild lifecycle parity

    4f09559

    test(e2e): restore inference helper parity (NVIDIA#5805)

    3c4fd32

    test(e2e): restore recovery kimi scope parity (NVIDIA#5808)

    699eb85

    test(e2e): restore messaging bash parity (NVIDIA#5807)

    cabbc51

    test(e2e): restore messaging bash parity (NVIDIA#5806)

    ac54d16

    test(e2e): close platform and workflow parity decisions (NVIDIA#5823)

    5b8efe2

    test(e2e): retire legacy shell lanes (NVIDIA#5756)

    2054f52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area: e2eEnd-to-end tests, nightly failures, or validation infrastructuresprint 6Sprint 6

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions