Skip to content

fix(test-impact): a parse gap names only reached functions, variables, macros - #2970

Merged
DeusData merged 1 commit into
mainfrom
fix/test-impact-gap-matching
Nov 13, 2026
Merged

DeusData merged 1 commit into
mainfrom
fix/test-impact-gap-matching

Conversation

@DeusData

Copy link
Copy Markdown
Owner

What this changes

The test-impact engine reads the text of a parse gap (lines the C parser could
not turn into edges): when the text names a definition the walk reached, the
definition around the gap is reached too. Two parts of that rule made it fire
almost everywhere in this repository:

  • a gap's words were matched against the short name of every reached
    node, fields and types included, so words like idx, n, index and
    used fired gaps in unrelated code;
  • a gap outside every definition seeded its whole file.

The cascade reaches code the change cannot reach. One file-scope gap shows
the mechanism: src/foundation/compat_thread.c:109
(__attribute__((used))) fired on any reached field named used and seeded
the whole file, including cbm_mutex_init, which the memory core calls, so
the walk went on to every caller of cbm_free. In the replays below the
old rule kept 79-81% of the 10,302 mapped tests for every product change, and in
the setup without a team artifact it vetoed whole runs (RUNNER_REACHED
through the memory core).

User decision R155 (2026-11-11):

  • a gap's words match only the names of reached functions, methods,
    variables and macros
    of at least three characters;
  • a gap outside every definition seeds the definitions that start on its
    own lines
    , never its whole file.

A gap inside a function that names a reached function still reaches that
function's callers, as before.

Replays

Setup: one base for every replay, b3ec08d (main on 2026-11-12), with its
producer bundle (graph + Linux x86_64 coverage map, 0.1 days old, admitted).
Each change is the revert of a recently merged PR on top of b3ec08d
(git merge-tree + commit-tree, no refs): a revert touches the same
functions as the PR, so it reaches the same code, and every replay can share
one admissible coverage map. 13 of 16 candidates revert cleanly (0d835c5,
5b87320 and f61d7c0 conflict with later commits). Old = the engine on
main (c90b1d9); new = this branch. Tests are counted at test level: a whole
suite counts all its mapped tests.

merge (reverted) change old answer old tests new answer new tests dropped required (oracle) new misses old misses
822e7fc #2913 layout json invalid unicode selected 8,130 selected 4,580 3,550 5 0 0
6df7fcb #2906 agentty install selected 8,167 selected 4,599 3,568 145 0 0
f5d5746 #2951 idle cadence guard selected 8,144 selected 4,551 3,593 13 0 0
c1af82f #2840 watcher reindex cooldown selected 8,167 selected 4,588 3,579 59 0 0
55206a8 #2903 manifest walk NUL file selected 8,184 selected 4,584 3,600 1,892 0 0
fe55b86 #2749 argv grep find selected 8,198 selected 4,629 3,569 57 0 0
50079ed #2947 swift in-scope binding selected 8,294 selected 7,758 536 5,596 0 0
8bf489e #2905 retrofit client http calls selected 8,195 selected 7,639 556 5,626 4 * 4 *
0640103 #2950 fqn unicode collision guard (tests only) selected 2,400 selected 2,400 0 0 0 0

Of 10,302 mapped tests, the old engine kept 79-81% for every product change;
the new one keeps 44-45% for small changes and 74-75% for the two
pipeline-wide ones (the test-only change is the same either way).
With a bundle the old engine did not veto in this set; it over-selected: a
watcher change, for example, pulled in every language-server, grammar and
cypher suite.

The four PRs replayed for #2959 (merged PR as M^1..M, no bundle, so
every suite runs whole and only the run-all reason matters):

merge main's engine (#2959) this branch
50079ed #2947 RUNNER_REACHED (main -> cbm_index_supervisor_capture_build_fingerprint -> ... -> cbm_free -> check_owned -> ..., the memory-core chain) selected
edfb824 #2943 RUNNER_REACHED (same chain) selected
0d835c5 #2436 RUNNER_REACHED (same chain) RUNNER_REACHED main -> cbm_kind_in_set_free_cache
5b87320 #1128 RUNNER_REACHED (same chain) RUNNER_REACHED main -> cbm_kind_in_set_free_cache

The remaining veto on 0d835c5 and 5b87320 does not come from a gap: a
debug run of 5b87320 shows no gap seeding cbm_kind_in_set_free_cache
(both PRs edit internal/cbm/cbm.h; the seed comes from the change itself).
It is outside this change and recorded for a follow-up.

Miss audit

The oracle is independent of the engine's walk: the producer bundle of the
base commit b3ec08d (its graph and its per-test coverage map, recorded on
Linux x86_64, 10,302 tests). For each replay it collects the tests the change
really needs:

  • every test whose recorded coverage executes a changed function;
  • every test of a suite whose setup (suite:*) executes one;
  • for a changed definition that is not a mapped function (a macro, a global,
    a type, a field, a function no test executed): every test executing a
    function that uses it: by graph edge (up to 6 hops to the nearest mapped
    caller), by name in its source text (macros, globals, types), or by member
    access ->name / .name (fields).

A test the new selection drops is a miss if the oracle needs it.

Result: no miss this change introduces. 9 replays, 22,551 dropped tests in
total; the oracle requires none of them (no dropped test executes a changed
function, a suite setup that does, or a user of a changed macro, global,
type or field).

  • * 8bf489e: 4 tests are flagged for both engines, only through the
    oracle's coarse type rule. The change removes the field
    CBMDefinition.client_route_path. Every use of that field is in a function
    the change itself edits, and those tests are selected. The 4 tests only use
    CBMDefinition elsewhere, so they cannot see the removed field.
  • 6df7fcb: the one product change the map cannot vouch for is
    src/main.c:print_help (the product binary, not the coverage runner). The
    tests that check the help text (suites cli, daemon_bootstrap) run whole
    in the new selection.
  • 0640103 (a test-only change) selects the same 2,400 tests with both
    engines.

Four more clean reverts (edfb824, d40ebf6, ff812bf, 14aeda1) were
stopped without an answer after 21-27 minutes (main's engine) and 6-14
minutes (this branch), to keep the replay disk bounded: both engines sat in
the snapshot build, before any gap matching (cbm_pipeline_pass_calls on
the 35k-line tests/test_pipeline.c or the 22k-line tests/test_cli.c, or a
full rebuild after a Python change). That is a separate, pre-existing cost
(the same would run into select-tests' 30-minute budget in CI), recorded for
a follow-up. edfb824 is covered by the no-bundle replay above.

What the narrower rule gives up (accepted by R155)

  • A gap that hides a use of a field or a type no longer reaches its
    enclosing definition. A changed field still reaches every reader the parser
    did see (USAGE edges) and every test whose coverage executes them.
  • A gap that hides a call to a definition named with fewer than three
    characters
    no longer fires. In this repository that is the function
    sw (iris_export_xml.c) and the macros Q, U, I, S, CH and c.
  • A gap outside every definition reaches only the definitions that start
    on its lines. 217 of the 585 C-family gap ranges in the graph lie outside
    every definition (44 files); before, each of them could seed its whole
    file.

CI scope

  • What changes in CI: only how much of the test suite PR CI selects. No
    workflow, job, runner or gate changes. Less over-reach means fewer suites
    and tests on the test legs (x86 runs the selection, arm64/macOS/Windows its
    portable view, per fix(test-impact): started roles, fork PRs, base tooling, all test legs #2959).
  • When it takes effect: select-tests runs the merge base's engine
    (fix(test-impact): started roles, fork PRs, base tooling, all test legs #2959), so this lands for PRs whose merge base contains it. This PR's own
    run uses main's engine, and a change to src/mcp/test_impact* runs
    everything by rule anyway.
  • Gating: unchanged. select-tests stays outside ci-ok; the test legs
    gate as before, on what they are given to run.
  • Risk: a test that is no longer selected because a gap stops firing.
    The miss audit above checks every dropped test against the recorded
    coverage of the changed code.
  • Cost: no new jobs, no new steps.
  • Off switches: unchanged (TEST_IMPACT_MODE=shadow, or the 'gate'
    default in pr.yml); reverting this commit restores the old matching.

Proof

  • test_impact_engine_parse_gaps_name_only_reached_functions: on main's
    engine it fails with all three over-reaches (the file-scope gap's file, the
    field word used, the two-letter global nn); with this change it passes.
  • test_impact_engine_parse_gaps_reach_what_they_name (a gap naming a
    reached function inside a function still selects its callers' tests)
    passes before and after.
  • ASan + UBSan, macOS arm64: test_impact_engine 23/23; test_impact,
    test_impact_artifact, test_impact_runner_filter, runner_isolation 99/99.
  • scripts/test.sh --contracts-only (every contract step, including
    tests/test_test_impact_selection.sh and tests/test_lane_wiring_contract.sh)
    passes; memory-core linter passes; clang-format (Homebrew LLVM) clean on the
    touched hunks. cppcheck 2.20 is left to the hosted lint job.

…, macros

A partial parse leaves lines the graph has no edges from, so the engine
reads a gap's text: when it names a definition the walk reached, the
definition around the gap is reached too. The match took the short name of
every reached node, fields and types included, and a gap outside every
definition seeded its whole file. In this repository that cascaded: one
replay fired 312 distinct gaps, most on words like `idx`, `n`, `index` and
`used`. The file-scope gap at src/foundation/compat_thread.c:109
(`__attribute__((used))`) fired on any reached field named `used` and seeded
the whole file, including cbm_mutex_init, which the memory core calls, so
nearly every change reached every role handler and the runner's main
(RUNNER_REACHED) through cbm_free.

User decision R155 (2026-11-11): a gap's words match only the names of
reached functions, methods, variables and macros of at least three
characters, the kinds of definition a hidden call or read can name; and a gap
outside every definition seeds the definitions that start on its own lines,
never its whole file.

Test: test_impact_engine_parse_gaps_name_only_reached_functions. Before the
fix a changed field `used` and a changed two-letter global `nn` fire gaps in
unrelated helpers, and a file-scope gap naming a reached function seeds the
whole file; the existing function-name gap test still passes.

Signed-off-by: Martin Vogel <[email protected]>
@DeusData
DeusData merged commit d68f491 into main Nov 13, 2026
27 of 29 checks passed
DeusData added a commit that referenced this pull request Nov 13, 2026
Brings in #2974 (incl. the jsonnet scanner patch) and #2970. One conflict:
scripts/vendored-checksums.txt, where #2974 and the vba vendoring both
rewrite the MANIFEST.md line. Resolved by taking main's file and
regenerating it with scripts/security-vendored.sh --update; against main it
differs only in the MANIFEST.md line and the five vba lines, and the
integrity verify passes.

THIRD_PARTY.md, MANIFEST.md and tests/test_parse_coverage.c merged cleanly;
each equals main plus this branch's own change (identical patch-ids), and
test_parse_coverage.c has 76 TEST and 76 RUN_TEST entries (71 at the merge
base, +2 from main, +3 from this branch). The MANIFEST summary (164
grammars) and ABI line (9/78/77) match a recount from the tree.

Signed-off-by: Martin Vogel <[email protected]>
DeusData added a commit to ankitsusheel/codebase-memory-mcp that referenced this pull request Nov 14, 2026
Brings the branch up to main d68f491 (DeusData#2967, DeusData#2968, DeusData#2970, DeusData#2974).

src/pipeline/registry.c: main (DeusData#2968) gives registry_resolve_scoped the
calling definition's test verdict (`caller_test`), adds
cbm_registry_resolve_ref for references (USAGE, READS, WRITES) and
routes calls through a static resolve_call_lang. This branch wraps the
lookups with the VB.NET case-insensitive retry. Resolved by keeping
both: vb_resolve_folded takes `caller_test` and passes it to its own
lookup, and the retry hooks cbm_registry_resolve_lang (false),
cbm_registry_resolve_ref and resolve_call_lang (the caller's verdict),
exactly as main's first lookup in each. Without the hook in
cbm_registry_resolve_ref, VB.NET references would have stopped folding.

internal/cbm/vendored/grammars/MANIFEST.md: main added the sql row next
to the vbnet row this branch edits; both rows kept.

scripts/vendored-checksums.txt: regenerated with
scripts/security-vendored.sh --update; only the MANIFEST.md line
differs from main.

Adapted with the merge: the VB.NET registry test passes `false` as the
new caller_test argument of cbm_registry_resolve_call_scoped (a
production caller).

Signed-off-by: Martin Vogel <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant