fix(extract): capture complete doc comments across languages - #2459

Summary

extract_docstring() took only the single comment node immediately before a definition. On real code that loses most documentation:

• multi-line // / /// docs keep only their last line (in C# usually /// </summary>);
• docs above attributes, export statements and wrapper declarations are dropped;
• fields, variables, constants and modules never get a doc;
• Perl POD is never stored;
• every doc is cut at 500 bytes.

This PR captures the complete documentation comment for every supported language. Apart from the docs themselves, the graph is unchanged except for the side effects listed under "Graph impact".

What changes

• Comment runs: a contiguous same-style comment run is the doc. Go comment groups follow go/ast semantics: directive lines (//go:…, //line, //export, //extern, //[a-z0-9]+:…) are stripped.
• Climbing to the declaration: docs above Rust attributes, JS/TS export statements, wrapper declarations (arrow and function expressions, class-field arrows, pairs, assignments, ambient declarations) and C# top-level statements are found.
• Members: docs for fields, variables, constants and enum members, found with a memoised, linear parent walk. A naïve walk was quadratic on a C# class with 65,536 fields.
• Module docs: Go package comments and Rust //! docs land on the File node. Go has no per-file Module node (Nondeterministic Java USAGE-edge misattribution since #667 — edges attach to wrong source files on every fresh index #787).
• Perl POD: a POD section that names a sub (=head…/=item <name>) becomes its doc.
• Rust impl methods now get docs: extract_rust_impl never asked for them.
• Kotlin KDoc that the grammar folds into package_header or sibling gaps is recovered.
• C typedefs: typedef struct/enum/union {…} X; gets the comment above it.
• Doc vs plain comment: a doc comment wins over an intervening plain // comment, e.g. a Javadoc above // NOSONAR.
• Not docs: runs detached by a blank line, and runs consisting only of divider lines.
• No 500-byte cut. The properties buffer grows with the docstring, so a long doc no longer pushes other properties out.

Measured effect

Nine real-world corpora, indexed with origin/main (0f52d30) and with this branch. The metric is the share of documented items whose full doc comment is stored.

• Divider-only captures: 12,374 → 0. C# alone had 11,878 /// </summary>-style one-liners.
• File-node docs: 1,045 Go package comments and 45 Rust //! docs are now stored.
• Index time: paired back-to-back runs show C# +1.8% (3 pairs) and Go +2.8% (5 pairs, excluding one pair with a load spike). Both are within run-to-run noise on a shared machine.
• DB size: +0.3% to +1.6%. Perl is +17% because its POD was previously absent.

Graph impact

• Node sets: identical (label, name, path, lines) on all nine corpora.
• Edges: identical except SEMANTICALLY_RELATED / SIMILAR_TO, because the semantic pass now sees the full docs.
• Properties:
  • The docstring changes.
  • Optional properties (bt, sp, fp, …) that the fixed 2 KB buffer used to drop are now restored: Java 5,116 nodes, C# 663.
  • On 17 nodes across all corpora, the greedy optional-field order now keeps a different optional field than before. No property value changes anywhere.
• No index-format bump, following the convention for property-content changes (see pass_importance.c, store.c). Existing indexes refresh their docstrings when files change or on a full re-index. That is worth a line in the release notes.

Tests

tests/test_extraction.c: 13 new tests and 1 updated.

• New: Go comment group without directives, Go struct field, Go detached comment, Python comment run, Rust attribute + impl method + inner doc, TS export + arrow wrapper, C# XML-doc run + top-level function, Java Javadoc over // + field, Kotlin folded KDoc, C typedef struct, Perl POD by sub name, long comment kept whole, divider-only run.
• Updated: #1017's len <= 500 assertion becomes "the whole comment is kept".

tests/test_pipeline.c: 3 new tests — Go package comment on the File node, Rust inner doc on the File node, long docstring stored whole with signature intact.

RED on revert: with the source change reverted and the tests kept, extraction pipeline fails exactly these 17 tests, each on a behaviour assertion. With the change it is 710/710 green.

Verification

• Lint (clang-format + cppcheck, run.sh lint): green.
• Linux arm64 (ASan + LeakSan): 141/141 suites, 5,579 tests, 0 failed.
• macOS: affected suites green (extraction 406/406, pipeline 304/304, plus 51 further suites: 4,533 passed, 0 failed).
  • After rebasing onto c61b3806: rebuilt, and extraction pipeline doclinks semantic store_search lang_contract complexity re-run green.
  • The only rebase conflict was the new NestJS route block next to the docstring call.
• macOS full scripts/test.sh: stops at step 0x (version-metadata contract) on origin/main as well. ff8dd47 pinned scoop at 0.11.0, which is now the newest tag. That is unrelated to this change and tracked separately; this PR relies on PR CI for the full matrix.
• Linux amd64 (QEMU, local): red, but every failure checked is also red on origin/main.
  • A revert-check ran in the same container venue: the ASan SEGVs in index_resilience, mcp and complexity are identical on 0f52d30, with the same SQLite unixRead memcpy stack under QEMU.
  • The daemon_runtime/httpd identity failures match the known QEMU /proc/self/exe limit.
  • PR CI's native amd64 leg is the authority here.

Follow-ups (not in this PR)

• An optional line_comment property for Go trailing field comments (new surface).
• A policy for Go code-generation marker lines (// +k8s:…) inside docs.
• Filters for commented-out code and file-header comments, and for TS test-harness // @… lines.
• Strip divider lines inside otherwise real docs.
• Size the props buffer for all optional fields (removes the 17 swaps).
• The scoop version-metadata contract (step 0x).