Skip to content

test: select-e2e's full-suite asserts pass on any selection larger than five suites #3652

Closed

Description

Summary

Three tests in hack/select-e2e_test.bats assert "the full suite was selected" as [ "$(echo "$output" | wc -w)" -gt 5 ] (lines 67, 76, 117 at the current main). A word count above five is a proxy for "full suite": it passes on any selection of six or more suites, so a regression that drops the selection from the full set to, say, half of it stays green in all three tests. The comment above the first one states the intent plainly ("Full suite means more than 5 suites"), which makes the gap easy to inherit as fact.

Why it matters

These are exactly the tests guarding the escalation paths (shared library change, e2e harness change), where the failure mode worth catching is partial escalation: a selector bug that picks some suites but not all of them. The current assert cannot distinguish "all" from "more than five", so the guard is real against total failure and blind against the partial one.

Fix shape

Assert equality with the full set instead of a threshold: derive the expected list the same way the script's full-suite branch does (or pin the known suite list with the fixture sources the tests already construct) and compare with the actual selection. The fixtures are already synthetic and small, so the exact expected output is knowable in each test.

Activity

  1. added
    kind/bugCategorizes issue or PR as related to a bug
    area/ciIssues or PRs related to CI workflows, GitHub Actions, automation
    area/testingIssues or PRs related to testing (e2e, bats, unit tests)
    on Aug 8, 2026
  2. lexfrei commented on Sep 23, 2026

    @lexfrei
    ContributorAuthor

    Fixed by #3670. The escalation tests compare against the full suite list now instead of checking for more than five suites.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/ciIssues or PRs related to CI workflows, GitHub Actions, automationarea/testingIssues or PRs related to testing (e2e, bats, unit tests)kind/bugCategorizes issue or PR as related to a bugtriage/not-reproducibleIndicates an issue can not be reproduced as described

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions