Skip to content

fix(pipeline): size definition properties for every field - #2687

Merged
DeusData merged 3 commits into
mainfrom
fix/def-props-buffer-every-field
Oct 18, 2026
Merged

DeusData merged 3 commits into
mainfrom
fix/def-props-buffer-every-field

Conversation

@DeusData

Copy link
Copy Markdown
Owner

What

build_def_props (in pass_definitions.c and its twin in pass_parallel.c) appends a definition's fields atomically: a field that does not fit is dropped whole. The buffer was sized only for the docstring, variants and variant_key, and every other field shared the remaining 2 KB.

On wide definitions (long signatures, decorator lists, parameter lists), that left no room for the three fields appended last:

  • fp: the MinHash fingerprint;
  • sp: the structural profile;
  • bt: the body tokens.

They were dropped silently. Such definitions got no SIMILAR_TO edge and fewer SEMANTICALLY_RELATED edges. Any newly added property pushed more of them out; #1692's route_path / route_method dropped fp on 12 long-docstring jellyfin controller actions and cost 34 SIMILAR_TO edges.

pd_props_buf / pp_props_buf (twins, kept in lockstep) now count:

  • the fixed metrics prefix;
  • the whole serialized form of every field build_def_props appends;
  • the reserve that the fp / sp / bt room checks use.

Definitions that fit keep the stack buffer. The field order is unchanged, so definitions that already fit produce byte-identical properties.

Consumers of these properties are pass_similarity, pass_semantic_edges and semantic search; MCP's search_graph already filters fp / sp / bt out of agent output. None of them depends on the fields being absent, so this is not an index format change.

Evidence

Corpus run against current main (354ccec), all PHP and C# corpus repos (PHP 27 par / 26 seq, C# 29 par / 29 seq; the heavy roslyn/runtime pair excluded):

language SIMILAR_TO (identical on par and seq) SEMANTICALLY_RELATED par seq
PHP +964 / -276 (net +688) +583 / -709 +502 / -532
C# +239 / -49 (net +190) +1,174 / -1,180 +1,244 / -1,218
  • No other edge type moves on either venue.
  • 324 of the 325 removed SIMILAR_TO edges are displacements under the per-node edge cap: one endpoint gained a better-ranked neighbour once more definitions carried a fingerprint. The remaining one fits the LSH candidate cap.
  • SEMANTICALLY_RELATED is TF-IDF with corpus-wide document frequencies plus a threshold. Recovered body tokens shift IDF across the repo, so some near-threshold pairs cross either way.

Test: parallel_wide_definition_keeps_fp_sp_bt_f14, both resolvers.

  • A Python function with 50 annotated parameters loses fp, sp and bt without the fix (props 1,904 B) and keeps all three with it (2,600 B).
  • A control with the same body, small, has all three either way (1,026 B, unchanged).

Suites run locally: parallel, pipeline (+ conventions / frozen / semantic-manifest), semantic, incremental, store_search: 819 passed. cppcheck 2.20 (the CI version) and clang-format are clean on the changed files.

A definition's properties buffer was sized for its docstring, variants and
variant_key only; every other field had to share the remaining 2 KB. Fields
that do not fit are dropped whole, so a long signature, decorator list or
parameter list silently squeezed out the MinHash fingerprint (fp), the
structural profile (sp) and the body tokens (bt) appended behind them. Those
definitions got no SIMILAR_TO edge and fewer SEMANTICALLY_RELATED edges, and
any newly added property (route_path, route_method) pushed more of them out.

pd_props_buf and pp_props_buf (twins, kept in lockstep) now count the fixed
metrics prefix plus the whole serialized form of every field
build_def_props appends, including the reserve the fp/sp/bt room checks
use. Definitions that fit keep the stack buffer.

Signed-off-by: Martin Vogel <[email protected]>
@DeusData
DeusData merged commit 1a7839d into main Oct 18, 2026
29 checks passed
@DeusData

Copy link
Copy Markdown
Owner Author

Post-merge audit note (clean builds of main before and after this merge, 1346ac4e -> 1a7839db).

This PR was measured on PHP and C#, where only SIMILAR_TO moved. It also changes Java route handlers, which the PR description did not cover:

  • Before this change, the undersized definition-property buffer dropped route_method from Spring controller methods while keeping route_path (for example DeviceProfileController.getDeviceProfileInfos had route_path=/api/deviceProfileInfos and no method). ensure_one_decorator_route then fell back to ANY, so each such handler handled its real GET/POST route and an extra __route__ANY__ route.
  • With this change, route_method is kept (GET, POST), and the extra ANY handler edge is no longer created. On thingsboard: HANDLES 1,012 -> 998 (-14, none added), all 14 of them ANY duplicates of a method route the handler still handles.

So the change is a correctness fix for Java routes too. One side effect: those artifact edges had been linking callers whose HTTP method is unknown (they land on the ANY route, e.g. test helpers that take the URL as an argument) to the handler, so thingsboard loses 46 DATA_FLOWS edges that only existed through them. Matching an unknown-method caller to the method-specific handler routes of the same path is a separate gap in pass_route_nodes.c. It is tracked as a follow-up and will be handled in its own change.

Method: bisect with clean builds at every merge point between 354ccecb and 1a7839db, thingsboard indexed once per point, and the handler properties read back through query_graph.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant