Skip to content

Commit 80e000a

Browse files
committed
doc-links(c): Doxygen @ref and @see names become MENTIONS edges, held until their audit
A Doxygen comment in a .c or .h file (a block opening with slash-star-star or slash-star-bang, a run of /// or //! lines) names code with `@ref name` or `\ref name` (family doxygen_ref) and with the names an `@see`, `\see`, `@sa` or `\sa` paragraph lists that have the shape of a symbol (`name`, `name()`, `::name`, `Type::member`; family see). Code blocks, fences, code spans, quoted labels and the word after \c, \p, \a, \b, \e, \em hold none; a URL is the core's href family. A .h is parsed as C++, which does not change the rule; other C++ and Objective-C files are not part of the leg. C keeps no node for a function prototype, so the docs of a header's API were never stored anywhere. The doc-link driver gets an optional loose_docs hook, called after the definitions and the file's own doc: the C leg reads every Doxygen block outside function bodies that no definition took, and its references belong to the file (the File node is the source). The scope ("c1") lists the definitions with nodes (function, variable, type, enum constant, field, object-like and function-like macro) with their qualified names, their linkage (static, or a type, constant or macro of a .c file, is the file's own) and the parent of a constant or field, and the pages, groups, anchors and sections the file's docs define. doc_links_c.c resolves a name as Doxygen does, without imports: - a page, group, anchor or section name is no code entity (graph_gap) and never binds a symbol of the same name; - then the documented file's own definitions (static ones too), its fields, then what every file sees (external linkage, a header's types, constants and macros); a static definition of another file is not visible; - candidates count by qualified name (a prototype and its definition, #if variants: one node); with `()` written a function or function-like macro wins, then a definition wins over a macro; - `Type::member`: a field of the struct or union Type, or a constant of the enum Type; - one candidate binds, several are ambiguous; a word of an @see paragraph that names nothing and is a plain lowercase word is prose (Doxygen prints it as text): no row; test code binds only from test code. There is no scope_delta hook yet: a declaration or linkage change is a full run. The families are held (ships=false) until their held-out audit passes; the suite ships them through the test seam. Proof (ship build, Apple M5 Pro / 48 GB, base 7a321ca): - xxhash (redis 4f20cb489344, a clean copy), the field test's 174 audited links, 110 of which had hit a rename macro before C1: every harvested one now names the right entity (169 edges, 2 self-mentions: XXH_NAMESPACE's own doc); the other 3 are @copydoc, which this leg does not harvest. - redis: doxygen_ref 74 edges (179 mentions), see 25 (30); rows: graph_gap 50, missing 6, ambiguous 3, unparseable 1. The prose rule took the 75 rows of words like "for" and "details" out. - pico-sdk 079c6f39: doxygen_ref 488 (730), see 199 (219); rows: graph_gap 142 (groups and pages), missing 94, ambiguous 68 (one type per chip, rp2040 and rp2350), test_only_target 28 (assembly functions with only a C test version), unparseable 5. - libusb abba63ed: doxygen_ref 256 (284), see 39 (44); rows: graph_gap 62, missing 18, unparseable 2, ambiguous 2. 46 of the edges are Type::member references. - libevent d82464a2: see 176 (230); rows: missing 9. - a random sample of 24 edges (8 per corpus, pico-sdk, libusb, libevent) read against their source lines: all correct, among them a header prototype's doc linking to the definition in the .c file. Finding outside the leg: libevent's event_assign has no node (its return type and name are split by an #if block), so its references are missing. Tests (doc_mentions_c): tokens with their lines and the file-level source of a prototype's doc; C++ files out, headers in; the scope blob; every resolution rule; incremental equals full (a doc edit and a body edit repair one file, a definition made static is a full run); 60 modules give the same edges and rows with one worker and several. Each rule red with it reverted. Signed-off-by: Martin Vogel <[email protected]>
1 parent 7a321ca commit 80e000a

10 files changed

Lines changed: 1680 additions & 31 deletions

File tree

‎Makefile.cbm‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -392,6 +392,7 @@ EXTRACTION_SRCS = \
392392
$(CBM_DIR)/doclink_perl.c \
393393
$(CBM_DIR)/doclink_ts.c \
394394
$(CBM_DIR)/doclink_php.c \
395+
$(CBM_DIR)/doclink_c.c \
395396
$(CBM_DIR)/sql_values.c \
396397
$(CBM_DIR)/helpers.c \
397398
$(CBM_DIR)/cpp_branch_views.c \
@@ -524,6 +525,7 @@ PIPELINE_SRCS = \
524525
src/pipeline/doc_links_perl.c \
525526
src/pipeline/doc_links_ts.c \
526527
src/pipeline/doc_links_php.c \
528+
src/pipeline/doc_links_c.c \
527529
src/pipeline/doc_links_msbuild.c \
528530
src/pipeline/pass_route_nodes.c \
529531
src/pipeline/pass_enrichment.c \
@@ -786,7 +788,7 @@ TEST_DISCOVER_SRCS = \
786788

787789
TEST_GRAPH_BUFFER_SRCS = tests/test_graph_buffer.c
788790

789-
TEST_PIPELINE_SRCS = tests/test_registry.c tests/test_pipeline.c tests/test_pipeline_conventions.c tests/test_pipeline_frozen.c tests/test_importance.c tests/test_cross_repo.c tests/test_fqn.c tests/test_route_canon.c tests/test_path_alias.c tests/test_configlink.c tests/test_doclinks.c tests/test_doc_mentions.c tests/test_doc_links_md.c tests/test_doc_links_pdf.c tests/test_doc_links_rst.c tests/test_doc_links_adoc.c tests/test_doc_mentions_go.c tests/test_doc_mentions_jvm.c tests/test_doc_mentions_rs.c tests/test_doc_mentions_perl.c tests/test_doc_mentions_ts.c tests/test_doc_mentions_php.c tests/test_infrascan.c tests/test_worker_pool.c tests/test_parallel.c tests/test_index_resilience.c tests/test_index_format.c tests/test_call_reference_contract.c tests/repro/repro_call_scope_usages.c tests/repro/repro_call_argument_usages.c tests/repro/repro_reference_precision.c tests/repro/repro_lexical_binding_precision.c tests/repro/repro_call_argument_matrix_a.c tests/repro/repro_call_argument_matrix_b.c tests/repro/repro_call_node_behaviors.c tests/repro/repro_language_registry.c tests/repro/repro_call_node_manifest.c tests/repro/repro_lsp_ordered_signatures.c tests/repro/repro_lsp_ordered_local.c tests/repro/repro_ts_overload_return_chains.c tests/repro/repro_harness_cleanup.c tests/repro/repro_runner_filter.c
791+
TEST_PIPELINE_SRCS = tests/test_registry.c tests/test_pipeline.c tests/test_pipeline_conventions.c tests/test_pipeline_frozen.c tests/test_importance.c tests/test_cross_repo.c tests/test_fqn.c tests/test_route_canon.c tests/test_path_alias.c tests/test_configlink.c tests/test_doclinks.c tests/test_doc_mentions.c tests/test_doc_links_md.c tests/test_doc_links_pdf.c tests/test_doc_links_rst.c tests/test_doc_links_adoc.c tests/test_doc_mentions_go.c tests/test_doc_mentions_jvm.c tests/test_doc_mentions_rs.c tests/test_doc_mentions_perl.c tests/test_doc_mentions_ts.c tests/test_doc_mentions_php.c tests/test_doc_mentions_c.c tests/test_infrascan.c tests/test_worker_pool.c tests/test_parallel.c tests/test_index_resilience.c tests/test_index_format.c tests/test_call_reference_contract.c tests/repro/repro_call_scope_usages.c tests/repro/repro_call_argument_usages.c tests/repro/repro_reference_precision.c tests/repro/repro_lexical_binding_precision.c tests/repro/repro_call_argument_matrix_a.c tests/repro/repro_call_argument_matrix_b.c tests/repro/repro_call_node_behaviors.c tests/repro/repro_language_registry.c tests/repro/repro_call_node_manifest.c tests/repro/repro_lsp_ordered_signatures.c tests/repro/repro_lsp_ordered_local.c tests/repro/repro_ts_overload_return_chains.c tests/repro/repro_harness_cleanup.c tests/repro/repro_runner_filter.c
790792

791793
TEST_WATCHER_SRCS = tests/test_watcher.c
792794

‎internal/cbm/doclink.c‎

Lines changed: 63 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -152,6 +152,8 @@ typedef struct {
152152
/* A document language: the whole file is the documentation, so one scan
153153
* of its bytes replaces parse_doc, the file-level doc and the scope. */
154154
void (*scan_file)(CBMExtractCtx *ctx);
155+
void (*loose_docs)(CBMExtractCtx *ctx, const CBMDefinition *file_def,
156+
cbm_doclink_taken_fn taken);
155157
} doclink_lang_t;
156158

157159
static const doclink_lang_t DOCLINK_LANGS[] = {
@@ -234,6 +236,17 @@ static const doclink_lang_t DOCLINK_LANGS[] = {
234236
.scan_scope = cbm_doclink_ts_scan_scope,
235237
.scope_tag = CBM_DOCLINK_TS_SCOPE_TAG,
236238
.portable_scope = cbm_doclink_ts_portable_scope},
239+
/* C, and the .h files C++ parses (the hooks skip every other file) */
240+
{.lang = CBM_LANG_C,
241+
.parse_doc = cbm_doclink_c_parse_doc,
242+
.scan_scope = cbm_doclink_c_scan_scope,
243+
.scope_tag = CBM_DOCLINK_C_SCOPE_TAG,
244+
.loose_docs = cbm_doclink_c_loose_docs},
245+
{.lang = CBM_LANG_CPP,
246+
.parse_doc = cbm_doclink_c_parse_doc,
247+
.scan_scope = cbm_doclink_c_scan_scope,
248+
.scope_tag = CBM_DOCLINK_C_SCOPE_TAG,
249+
.loose_docs = cbm_doclink_c_loose_docs},
237250
{.lang = CBM_LANG_PHP,
238251
.parse_doc = cbm_doclink_php_parse_doc,
239252
.scan_scope = cbm_doclink_php_scan_scope,
@@ -359,12 +372,6 @@ static doc_line_ent_t *doc_info_of(CBMExtractCtx *ctx, const char *doc) {
359372
return NULL;
360373
}
361374

362-
/* A doc not built by doc_run_text keeps the definition's own line. */
363-
static uint32_t doc_line_of(CBMExtractCtx *ctx, const char *doc) {
364-
const doc_line_ent_t *info = doc_info_of(ctx, doc);
365-
return info ? info->line : 0;
366-
}
367-
368375
/* ── Persisted scope ─────────────────────────────────────────────── */
369376

370377
char *cbm_doclink_portable_scope(const char *scope) {
@@ -461,6 +468,55 @@ static doclink_twin_key_t *twin_keys(CBMExtractCtx *ctx, const char *label, int
461468
return keys;
462469
}
463470

471+
/* True when a doc taken from a definition, or the file's own doc, starts on
472+
* one of the lines [first, last]. */
473+
static bool doclink_taken(const CBMExtractCtx *ctx, uint32_t first, uint32_t last) {
474+
const doc_line_map_t *m = (const doc_line_map_t *)ctx->doc_lines;
475+
for (int i = 0; m && i < m->count; i++) {
476+
if (m->items[i].parsed && m->items[i].line >= first && m->items[i].line <= last) {
477+
return true;
478+
}
479+
}
480+
return false;
481+
}
482+
483+
/* The file's own doc, then the language's loose docs: their references belong
484+
* to the file. The parser gets a definition-shaped stand-in for the file; what
485+
* it pushes is marked, and the resolving half takes the File node as the
486+
* source. No qualified name for that node is computed on this side. */
487+
static void doclink_file_pass(CBMExtractCtx *ctx, const doclink_lang_t *L) {
488+
const char *file_doc = ctx->result->module_doc;
489+
bool has_doc = file_doc && file_doc[0];
490+
if (!has_doc && !L->loose_docs) {
491+
return;
492+
}
493+
CBMDocLinkArray *arr = &ctx->result->doc_links;
494+
int first = arr->count;
495+
CBMDefinition file_def = {
496+
.name = ctx->rel_path,
497+
.qualified_name = ctx->module_qn ? ctx->module_qn : "",
498+
.label = "File",
499+
.file_path = ctx->rel_path,
500+
.start_line = SKIP_ONE,
501+
.end_line = SKIP_ONE,
502+
.docstring = file_doc,
503+
};
504+
if (has_doc) {
505+
doc_line_ent_t *info = doc_info_of(ctx, file_doc);
506+
uint32_t doc_line = info ? info->line : 0;
507+
L->parse_doc(ctx, &file_def, file_doc, doc_line ? doc_line : file_def.start_line);
508+
if (info) {
509+
info->parsed = true;
510+
}
511+
}
512+
if (L->loose_docs) {
513+
L->loose_docs(ctx, &file_def, doclink_taken);
514+
}
515+
for (int i = first; i < arr->count; i++) {
516+
arr->items[i].flags |= CBM_DOCLINK_FLAG_FILE;
517+
}
518+
}
519+
464520
void cbm_doclinks_extract(CBMExtractCtx *ctx) {
465521
if (!ctx || !ctx->result) {
466522
return;
@@ -520,29 +576,7 @@ void cbm_doclinks_extract(CBMExtractCtx *ctx) {
520576
info->parsed = true;
521577
}
522578
}
523-
/* The file's own doc: its references belong to the file. The parser gets
524-
* a definition-shaped stand-in for the file; what it pushes is marked, and
525-
* the resolving half takes the File node as the source. No qualified name
526-
* for that node is computed on this side. */
527-
const char *file_doc = ctx->result->module_doc;
528-
if (file_doc && file_doc[0]) {
529-
CBMDocLinkArray *arr = &ctx->result->doc_links;
530-
int first = arr->count;
531-
CBMDefinition file_def = {
532-
.name = ctx->rel_path,
533-
.qualified_name = ctx->module_qn ? ctx->module_qn : "",
534-
.label = "File",
535-
.file_path = ctx->rel_path,
536-
.start_line = SKIP_ONE,
537-
.end_line = SKIP_ONE,
538-
.docstring = file_doc,
539-
};
540-
uint32_t doc_line = doc_line_of(ctx, file_doc);
541-
L->parse_doc(ctx, &file_def, file_doc, doc_line ? doc_line : file_def.start_line);
542-
for (int i = first; i < arr->count; i++) {
543-
arr->items[i].flags |= CBM_DOCLINK_FLAG_FILE;
544-
}
545-
}
579+
doclink_file_pass(ctx, L);
546580
if (L->scan_scope) {
547581
ctx->result->doc_scope = L->scan_scope(ctx);
548582
}

‎internal/cbm/doclink.h‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,10 @@
2222
* identical on full and incremental runs. Optional.
2323
* scope_tag, the blob's tag line and its persisted form (line
2424
* portable_scope numbers dropped). Optional.
25+
* loose_docs after the definitions and the file's own doc: the
26+
* file's other doc comments (C: a prototype's doc,
27+
* which the graph keeps no node for); what it pushes
28+
* belongs to the file. Optional.
2529
* scan_file instead of parse_doc, for a DOCUMENT language whose
2630
* text is the documentation (Markdown): one pass over
2731
* the file's bytes pushes every reference, with the
@@ -106,6 +110,9 @@
106110
X(CBM_DOCLINK_TS_SEE, CBM_LANG_TYPESCRIPT, "see", false, false, "MENTIONS") \
107111
/* php (doclink_php.c): PHPDoc @see and {@see}; held until their audit passes */ \
108112
X(CBM_DOCLINK_PHP_SEE, CBM_LANG_PHP, "see", false, false, "MENTIONS") \
113+
/* c (doclink_c.c): Doxygen @ref and @see in .c and .h files; held until their audit */ \
114+
X(CBM_DOCLINK_C_REF, CBM_LANG_C, "doxygen_ref", false, false, "MENTIONS") \
115+
X(CBM_DOCLINK_C_SEE, CBM_LANG_C, "see", false, false, "MENTIONS") \
109116
/* markdown (doclink_md.c): what a document's text names explicitly */ \
110117
X(CBM_DOCLINK_MD_LINK, CBM_LANG_MARKDOWN, "link", false, true, "MENTIONS") \
111118
X(CBM_DOCLINK_MD_PATH, CBM_LANG_MARKDOWN, "path", false, true, "MENTIONS") \
@@ -591,6 +598,26 @@ size_t cbm_doclink_php_qualify(char *out, size_t cap, const char *ns, size_t ns_
591598
const cbm_doclink_php_use_t *uses, int nuses, const char *ref,
592599
size_t n);
593600

601+
/* ── C (doclink_c.c) ───────────────────────────────────────────────── */
602+
603+
/* Asked by a loose_docs hook for a doc comment on lines [first, last]: true
604+
* when a definition's doc or the file's own doc there was already taken. */
605+
typedef bool (*cbm_doclink_taken_fn)(const CBMExtractCtx *ctx, uint32_t first, uint32_t last);
606+
607+
void cbm_doclink_c_parse_doc(CBMExtractCtx *ctx, const CBMDefinition *def, const char *doc,
608+
uint32_t doc_line);
609+
void cbm_doclink_c_loose_docs(CBMExtractCtx *ctx, const CBMDefinition *file_def,
610+
cbm_doclink_taken_fn taken);
611+
const char *cbm_doclink_c_scan_scope(CBMExtractCtx *ctx);
612+
613+
/* C files and the headers C++ parses share one scope tag: one resolver
614+
* indexes them. The blob holds no line: it is its own persisted form. */
615+
#define CBM_DOCLINK_C_SCOPE_TAG "c1"
616+
617+
/* The longest reference a C token keeps; a longer one is cut just past this
618+
* length and the resolver reports it as unparseable. */
619+
enum { CBM_DOCLINK_C_REF_MAX = 256 };
620+
594621
/* ── Perl POD (doclink_perl.c) ─────────────────────────────────────── */
595622

596623
void cbm_doclink_perl_parse_doc(CBMExtractCtx *ctx, const CBMDefinition *def, const char *doc,

0 commit comments

Comments
 (0)