Skip to content

fix(registry): rank test-file definitions as test for production callers - #2968

Merged
DeusData merged 2 commits into
mainfrom
fix/registry-rank-test-file-defs
Nov 13, 2026
Merged

DeusData merged 2 commits into
mainfrom
fix/registry-rank-test-file-defs

Conversation

@DeusData

Copy link
Copy Markdown
Owner

What regressed

#2427 (merged as b3ec08d) ranks registry candidates by the extractor's stored
test verdict instead of the old QN substring scan. That verdict is incomplete
for everything but callables: the extractor ORs the file's test-ness into
functions, methods and the module only (internal/cbm/extract_defs.c), so a
variable, field, class, struct or interface defined in a test file keeps
is_test=false. The QN scan used to hide this, because Java and Scala QNs spell
src.test and test class names.

Since #2427 those symbols rank as production code. A name-only match
(suffix_match) from production code then picks them whenever they share more
of the caller's path than the real candidate. Examples on main:

  • finagle: Future.exception(...) in production code binds to a Variable in
    TimeoutFactoryTest.scala.
  • kafka: production Scala xs.map(...) binds to a Variable map in
    LogCleanerTest.scala; production->test-file CALLS onto Variables went from
    6 to 362.
  • gardener: production calls bind to Go structs declared in _test.go files
    (75 of them).

Fix (two commits)

  1. fix(registry): rank every definition in a test file as test. The three
    registry seeders (pass_definitions.c, pass_parallel.c,
    pipeline_incremental.c) already pass extractor verdict || cbm_registry_test_double_path(path). They now also OR in
    cbm_is_test_file, the exact rule the extractor computes is_test_file
    with, on the same path (the one the node stores) and the same language, so
    every definition in a test file ranks as test, identically in full and
    incremental indexes.

  2. fix(registry): let test code rank test candidates like real code. The
    test penalty is for production callers. Test and test-double code uses its
    own helpers and mocks, so for such a caller no candidate takes the penalty
    and proximity alone ranks. The caller's verdict uses the same three rules
    (the extractor's verdict stored on the calling definition, the file rule,
    the test-double path rule) and reaches the registry from both call
    resolvers and both USAGE / READS / WRITES resolvers. Without this, kafka's
    clients tests calling new MockTime() bound TopologyTestDriver.MockTime
    in streams/test-utils/src/main instead of the clients test helper.

Scope is ranking only: the stored is_test property, TESTS-edge rules and
importance are unchanged (the test-file nodes still store is_test:false).

Tests

Three new pipeline tests beside #2427's rank probes, each run as a full index
(sequential), a fresh full index padded past MIN_FILES_FOR_PARALLEL with
CBM_WORKERS (parallel resolver), a comment-only edit on the closure-repair
route and one on the forced legacy-partial route:

  • pipeline_registry_ranks_test_file_variable_last_full_and_incremental
    (Go: a package-level variable in app/sub/parse_test.go vs the production
    lib.Parse)
  • pipeline_registry_ranks_test_file_class_last_full_and_incremental
    (Java multi-module layout: a class nested in streams/src/test/.../RunTest
    vs the production clients/.../Codec)
  • pipeline_registry_test_caller_ranks_test_helpers_full_and_incremental
    (Java: a clients test's new MockTime() vs a same-named class in a
    test-utils/src/main module)

All three are RED on their base and GREEN with their commit. For the first
two, reverting only the pass_parallel.c hunk turns the parallel leg RED and
reverting only the pipeline_incremental.c hunk turns the closure-repair leg
RED; for the third, ignoring the caller's verdict in the scorer turns it RED,
and so does dropping it at the parallel call site alone.

Proof

Production-source edges whose target is in a test file (cbm_is_test_file on
both ends). Columns: before #2427 (cf337a6) / main (b3ec08d) / this PR.

corpus CALLS onto Variables onto Structs/Classes USAGE WRITES
apache/kafka 4.0.0 (985bc995) 849 / 1023 / 750 6 / 362 / 6 13 / 15 / 13 5936 / 8254 / 5832 163 / 218 / 121
gardener (58d1ab91) 125 / 123 / 52 1 / 1 / 1 75 / 75 / 4 837 / 730 / 572 323 / 254 / 160
twitter/finagle (ca472deb) 281 / 505 / 273 107 / 355 / 107 4 / 9 / 4 1568 / 2436 / 1541 59 / 86 / 58

finagle with the Scala import work of #2360 (583e71b) on top: production->test
CALLS 566 -> 300, onto Variables 391 -> 107, onto Classes 4 -> 1.

  • fix(registry): score a candidate on the extractor's test verdict, not its name #2427's gains survive: every production edge fix(registry): score a candidate on the extractor's test verdict, not its name #2427 moved from a test file to
    production stays on production (gardener USAGE 125, WRITES 51, CALLS 1;
    kafka CALLS 59, USAGE 45, WRITES 13; finagle USAGE 17, CALLS 7).
  • The second commit leaves every production-caller CALLS / USAGE / WRITES edge
    of kafka identical to the first commit, and gardener's production CALLS too.
  • Test callers: kafka's new MockTime() calls bind the clients helper again
    (487 constructor and 17 class calls); gardener's mock packages bind their own
    symbols (USAGE into test-double files 4959 -> 5132, WRITES 61 -> 137).
    Test-caller CALLS that differ from main are mostly name-only member guesses
    either way (24 judged: 7 better, 6 worse, 10 wrong both ways, 1
    equivalent); kafka TESTS edges 64603 on main, 57518 here.
  • Determinism: two runs of each corpus are edge-identical.
  • Full vs incremental (25 seeded files, seed 168): finagle identical; gardener
    CALLS/USAGE/WRITES identical; kafka differs only for callers inside the
    edited files, exactly as main does.

Out of scope

  • The stored is_test verdict and TESTS edges (unchanged by design).
  • Name-only member-call guesses with an unknown receiver type (tp.topic()
    binding the project's one topic), which produce most of the remaining wrong
    edges in either direction. A receiver gate for them was measured and is not
    part of this PR.

#2427 (b3ec08d) ranks registry candidates by the extractor's stored
test verdict instead of the old QN substring scan. The extractor gives a
definition its file's verdict only for functions, methods and the module
(extract_defs.c), so a variable, field, class, struct or interface in a
test file keeps is_test=false. The QN scan had masked that: Java and
Scala QNs spell src.test and test class names. Since #2427 such a symbol
ranks as production code, and a name-only match (suffix_match) from
production code picks it whenever it shares more of the caller's path
than the real candidate. finagle's `Future.exception` bound to a Variable
in TimeoutFactoryTest; Go structs in _test.go files won the same way.

The three registry seeders (pass_definitions.c, pass_parallel.c,
pipeline_incremental.c) now also apply cbm_is_test_file, the rule the
extractor computes is_test_file with, on the same path and language, so
every definition in a test file ranks as test, in full and incremental
indexes alike. Ranking only: the stored is_test property, TESTS edges
and importance keep their own rules.

Proof, production-source edges into test files (cbm_is_test_file on both
ends), pre-#2427 cf337a6 / main b3ec08d / this change:
  kafka 4.0.0 985bc995  CALLS 849 / 1023 / 750 (onto Variables 6/362/6)
                        USAGE 5936 / 8254 / 5832, WRITES 163 / 218 / 121
  gardener 58d1ab91     CALLS 125 / 123 / 52 (onto Structs 75/75/4)
                        USAGE 837 / 730 / 572, WRITES 323 / 254 / 160
  finagle ca472deb      CALLS 281 / 505 / 273, USAGE 1568 / 2436 / 1541
  finagle with #2360    CALLS 566 -> 300 (Variables 391 -> 107)
Every edge #2427 moved from a test file to production stays there
(gardener USAGE 125, WRITES 51, CALLS 1; kafka CALLS 59, USAGE 45,
WRITES 13; finagle USAGE 17, CALLS 7). Only suffix_match CALLS move. Two
runs are identical, and an incremental re-index of 25 seeded files
matches the full index for every caller outside the edited files.

On kafka the change also restores about 6,800 suffix_match CALLS from
test files, with their TESTS edges, that main had lost because an
unmarked test-file field won the ranking and the CALLS guard refused it;
they are the edges the index had before #2427.

Signed-off-by: Martin Vogel <[email protected]>
The registry ranks a test candidate below real code. For a reference
made by test or test-double code that is the wrong way round: tests and
mocks use their own helpers. Since every definition in a test file ranks
as test (previous commit), kafka's clients tests calling `new MockTime()`
bound TopologyTestDriver.MockTime in streams/test-utils/src/main, a
test-support module no rule marks, instead of the clients test helper
they import.

The calling definition's own verdict (cbm_pipeline_caller_is_test: the
extractor's verdict stored on its node, cbm_is_test_file and
cbm_registry_test_double_path on its file) now reaches the registry from
both call resolvers and both USAGE / READS / WRITES resolvers. For a
test or test-double caller no candidate takes the test penalty, so
proximity alone ranks; production references keep the full rule. The
per-file resolve cache records the bit, since a test and a production
function of one file can ask for the same name.

Proof:
- Production-caller CALLS/USAGE/WRITES on kafka are edge-for-edge equal
  to the previous commit; gardener production CALLS too. The
  production->test-file tables and the #2427 gains are unchanged.
- kafka `new MockTime()` from tests: 487 constructor and 17 class calls
  bind the clients helper again (previous commit: 654 went to
  TopologyTestDriver.MockTime).
- gardener mock packages bind their own symbols: USAGE into test-double
  files 4959 -> 5132, WRITES 61 -> 137.
- Test-caller CALLS that differ from main (16,373 on kafka) are mostly
  name-only member guesses either way; 24 judged (seed 162): 7 better,
  6 worse, 10 wrong both ways, 1 equivalent. kafka TESTS edges: 64,603
  on main, 57,518 here.
- Two runs identical; incremental matches full for every caller outside
  the edited files.

Signed-off-by: Martin Vogel <[email protected]>
@DeusData
DeusData merged commit a0c7f28 into main Nov 13, 2026
28 checks passed
DeusData added a commit to ankitsusheel/codebase-memory-mcp that referenced this pull request Nov 14, 2026
Brings the branch up to main d68f491 (DeusData#2967, DeusData#2968, DeusData#2970, DeusData#2974).

src/pipeline/registry.c: main (DeusData#2968) gives registry_resolve_scoped the
calling definition's test verdict (`caller_test`), adds
cbm_registry_resolve_ref for references (USAGE, READS, WRITES) and
routes calls through a static resolve_call_lang. This branch wraps the
lookups with the VB.NET case-insensitive retry. Resolved by keeping
both: vb_resolve_folded takes `caller_test` and passes it to its own
lookup, and the retry hooks cbm_registry_resolve_lang (false),
cbm_registry_resolve_ref and resolve_call_lang (the caller's verdict),
exactly as main's first lookup in each. Without the hook in
cbm_registry_resolve_ref, VB.NET references would have stopped folding.

internal/cbm/vendored/grammars/MANIFEST.md: main added the sql row next
to the vbnet row this branch edits; both rows kept.

scripts/vendored-checksums.txt: regenerated with
scripts/security-vendored.sh --update; only the MANIFEST.md line
differs from main.

Adapted with the merge: the VB.NET registry test passes `false` as the
new caller_test argument of cbm_registry_resolve_call_scoped (a
production caller).

Signed-off-by: Martin Vogel <[email protected]>
DeusData added a commit to ankitsusheel/codebase-memory-mcp that referenced this pull request Nov 14, 2026
Since DeusData#2968, USAGE, READS and WRITES resolve through
cbm_registry_resolve_ref, and the main merge gave it the VB.NET retry.
No test reached that hook: dropping it would have silently returned
VB.NET references to case-sensitive lookups. A mixed-case reference now
resolves from production code and from test code alike.

Signed-off-by: Martin Vogel <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant