Repository navigation
fix(swift): preserve declaration identity and compatible call edges - #2436
Conversation
…Data#2061) Overloads that share a base QN (Java/C#/C++/Kotlin/Swift/Scala/ObjC) merge into one graph node today, and that node collects the CALLS of every overload. The agreed cure is one signature-qualified callable identity for all overloading languages: callable_qn := base_qn [<tparams>] (params) [cvref] This change lands the plumbing with every language still at CBM_CALLABLE_ID_NONE, so the graph is byte-identical; each language is switched on later in its own change (with its index-format bump). - internal/cbm/callable_sig.{c,h}: the per-language mode side table (NONE|TYPED|LABELED_TYPED|LABELED|ARITY; kept out of CBMLangSpec like cbm_string_dispatch_suffixes, since ~160 positional rows would otherwise all need the member under -Wmissing-field-initializers), the shared builder cbm_callable_sig[_mode] for the def and the call-scope QN, and the inverse cbm_qn_callable_base_len[_named]. Suffix contract: no '.', '::' or '->'; whitespace-normalized; comments/annotations/attributes dropped; qualified types keep their last segment; '->' spelled '=>'; variadic '[]' (Java) / '~'; C++ top-level cv dropped, arrays decay, cvref kept; <= 200 bytes, longer ones keep whole leading entries and end in '#<fnv1a64 of the full suffix>)'. The builder is iterative (cursor walk, bounded declarator loop) and self-checks against the inverse. - The inverse is anchored to the node's bare name wherever the name is known: real QNs already end in ')' without being callables (a Java field whose name is its declarator `X = f()`, make `$(VAR)` targets), and only the anchor makes the inverse exact on them. - Leaf splitters strip a suffix first (no-op today): registry by-name key (indexed by the bare name for a suffixed callable), cbm_lsp_bare_segment, cbm_pipeline_qn_class_method_tail, pxc_last_component, pxc_qn_leaf, go_index_visit_node, collect_prod_defs, sg_qn_prefix_len. - New unused fields: CBMDefinition.qn_sig_off, CBMResolvedCall.callee_sig (relocated by result compaction), CBMRegisteredFunc.sig, CBMLSPDef.sig. The lsp_surface codec is unchanged (no format bump in this change). - get_code_snippet and trace_path gain base-QN tiers (cbm_store_find_nodes_by_qn_base) that find nothing until suffixes exist. - store: cbm_store_find_nodes_by_qn_suffix built its LIKE pattern in a 512-byte buffer (a long suffix was truncated into a pattern that matched nothing) and did not escape '_'/'%' (`my_func` also matched `myXfunc`). The pattern is now heap-built, escaped, and matched with ESCAPE '\'. Proof: full sorted node+edge+lsp_surface dumps are byte-identical before and after on 7 corpora (this repo, elasticsearch libs, Exposed, dotnet System.Linq, System.Collections, coreclr/gc, a C corpus). Refs DeusData#2061 Signed-off-by: Martin Vogel <[email protected]>
Signed-off-by: Martin Vogel <[email protected]>
Enable typed, labeled Swift callable identities on the shared DeusData#2061 plumbing. Resolve calls by compatible labels, defaulted parameters, and trailing closures; retain all compatible overload edges with candidate counts. Cover registry, pipeline, and signature behavior. Refs DeusData#2061 Signed-off-by: DavidHLP <[email protected]>
Signed-off-by: DavidHLP <[email protected]>
Force the parallel call pass on a Swift fixture and assert both label-compatible type overloads retain CALLS edges with the ambiguity count. Refs DeusData#2061 Signed-off-by: DavidHLP <[email protected]>
Signed-off-by: DavidHLP <[email protected]>
A Swift call that shares its bare name with project symbols but matches none of their overloads was handled differently by the two resolvers: the parallel path cleared the bare-name match and fell through as unresolved (keeping each resolver's empty-resolution service fallbacks), while the sequential path dropped the call outright. Skip the bare-name registry bind on the sequential path too — it would bind a wrong overload — and continue as unresolved, so count < 0 behaves exactly like any other unresolved call on each resolver. Pipeline regressions pin the global-fetch fallback (DeusData#856) on both paths: the incompatible local overload gets no wrong CALLS edge, and the URL call keeps its HTTP_CALLS edge. The route-registration fallback differs between the paths on main (sequential additionally requires a '/'-leading first argument); that pre-existing divergence is out of scope here. Refs DeusData#2061 Signed-off-by: DavidHLP <[email protected]>
Signed-off-by: DavidHLP <[email protected]>
|
Thanks for opening this — it has been seen, and it is queued. This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence. Current review status: working through a backlog. What that means for this PR, concretely:
Things that will genuinely speed it up whenever review does happen:
If this fixes a bug, a reproduction we can run is worth more than a description of the symptom. Thanks for contributing, and sorry in advance for the wait. |
Signed-off-by: DavidHLP <[email protected]>
Signed-off-by: DavidHLP <[email protected]>
3441da4 to
5538355
Compare
Signed-off-by: DavidHLP <[email protected]>
|
Thank you for the detailed status update and for clearly distinguishing the actual PR head from unpublished work and unverified results. The changed-file scope includes callable identity, resolution and persisted metadata, so we need more time for a complete review together with the parent change. This remains queued for deeper review; this comment is not an approval or a request for another routine rebase. The queue is full, and we appreciate your patience and the care taken to document the remaining work. |
Swift-scope checkpoint for DeusData#2061: capture bounded trailing-closure labels, match Swift overload candidates conservatively, and cover extraction, registry, compact/spill, serial, parallel, and incremental behavior. Validation: ASan passed on the combined WIP source only; this A-only checkpoint has not been independently verified. GCC TSan, smoke, and memory gates remain failed/open. This commit is a scope checkpoint, not acceptance. Signed-off-by: David HLP <[email protected]>
Signed-off-by: David HLP <[email protected]>
Signed-off-by: David HLP <[email protected]>
Signed-off-by: David HLP <[email protected]>
Signed-off-by: David HLP <[email protected]>
Signed-off-by: David HLP <[email protected]>
Signed-off-by: David HLP <[email protected]>
4dbfd27 to
eea3143
Compare
Signed-off-by: DavidHLP <[email protected]>
Signed-off-by: DavidHLP <[email protected]>
Signed-off-by: DavidHLP <[email protected]>
Keep extension owners within visible targets and respect literal custom SwiftPM roots, nested package boundaries and manifest invalidation. Continue compatible call chains through declared return types and bounded initializer inference. Recover conditional declaration ownership and wrapped trailing closures, and preserve serial argument keywords. Add extraction, registry and serial/parallel/incremental regressions. Signed-off-by: DavidHLP <[email protected]>
Signed-off-by: DavidHLP <[email protected]>
Signed-off-by: DavidHLP <[email protected]>
|
Thank you, @DeusData, for rerunning the census and documenting the exact call sites. The two filesystem targets in swift-nio and the generated
I also changed Alamofire’s target path from The PR head passed the default production regression checks. The local validation tree, which included the separate fixes from draft #2809, completed all 192 suites under ASan and UBSan with 10,363 passed, 0 failed, and 10 skipped. All 164 FastAPI incremental tests ran and passed. GitHub CI for the PR head passed Linux, macOS, Windows CLANG64, TSan, CodeQL, DCO, and I kept the real-repository rerun focused on the reported call sites and manifest invalidation. I haven’t rerun the complete census or the generated 60/120-directory benchmark. Thanks again for the detailed review. |
|
Thank you, @DavidHLP, for another fast and careful round, and for the clear point-by-point reply. 1a6b292 closes the three structural problems from this morning, and the conditional-compilation work goes further than we asked: it fixes the missing We re-ran the same census as this morning (Alamofire bda9ed57, swift-nio eabcc14f, helm 9b88efd6 as the non-Swift control, every run indexed from scratch), comparing 05c7ba7 to 1a6b292:
One thing went the other way: 1,976 call sites on swift-nio (103 on Alamofire) that 05c7ba7 resolved now get no edge, and for 1,362 of them (61 on Alamofire) main resolves the same target. In a sample about half of these were right before, so this is a real loss. It also shows up as test links: TESTS on swift-nio drops from 52 to 30. A few shapes we saw:
Could you:
Two smaller observations, not required for this PR: 288 Alamofire call sites such as session.request(.default) now get 6 targets (the three Endpoint overloads in TestHelpers.swift plus the three URLConvertible overloads in Session.swift), which is expected from labels alone now that those overloads have their owner. And We are also looking at a few further points and will follow up on those separately. Thank you again; the structural part is in very good shape now, and the census shows how much the conditional-compilation fix adds. |
|
Hi @DavidHLP, a quick follow-up to our comment from earlier today: to save you another round, we will make the four changes from that list ourselves on your branch (maintainer edits are enabled), and re-run the same census on the result. If you have already started on any of them, just say so here and we will build on your work instead of duplicating it. Otherwise, please hold off pushing to this branch until we post the result, or pull first, so neither of us loses commits. Thank you again for all the work on this series - the structural part is in very good shape now. |
…known A variable whose type comes only from its initializer call (`let p = loop.makePromise(of: Int.self)`) replaced the label match with a typed lookup, and a failed lookup dropped the call. On swift-nio that left 1,976 call sites without an edge that the label match had resolved. The inferred type now only narrows the label match. When the inference fails (overloads with different return types, an unknown callee or root, a cycle) the receiver stays unknown, and when the inferred type has no member with the call's name and labels the call keeps the label match. Written types (annotations, parameters, constructor calls) and non-project receivers behave as before. Inferring an initializer's own type never uses a label match, so a guess cannot become a type. A callee's own generic parameter used as its return type (`func runSALOnEventLoop<Result>(...) -> Result`) names no type. It is read from the declaration's generic clause in the qualified name and leaves the bound variable unknown. The four rows of pipeline_swift_capped_chain_returns_and_metatype_receivers that expected no edge for an unknown receiver now expect exactly one label-matched upload(_:) edge. Signed-off-by: Martin Vogel <[email protected]>
`let socket = try ServerSocket(...)` and `let reader = try await Reader(...)` left the variable untyped, so every member call on it went to a label-only guess or was dropped. The initializer is now unwrapped from `try` / `await` before its constructor type is read. Only a bare type name counts there: `try Foo.make()` returns what make returns, which the registry still infers from the call. With the type in place the existing base-class and extension walk applies: on swift-nio socket.close(), socket.bind(to:) and socket.getLocalVsockAddress() in VsockAddressTest.swift resolve to ServerSocket, BaseSocket and the BaseSocket extension in VsockAddress.swift. Signed-off-by: Martin Vogel <[email protected]>
The conditional-compilation retry always kept the masked re-parse. When the dropped branch holds an `#error(...)` line inside a type body the re-parse reads less of the file: Alamofire Protected.swift went from 3 to 14 unreadable lines and lost its Protected initializer. The re-parse is now kept only when its error regions cover no more lines than those of the first parse. Counting regions instead would reject re-parses that split one large region into a few small ones; on swift-nio that gave back 1,312 of the recovered lines. Signed-off-by: Martin Vogel <[email protected]>
pipeline_swift_generic_return_keeps_label_match_issue2061: a callee's generic parameter used as its return type, colliding with a project type of the same name, leaves the variable unknown and its call keeps the label match. pipeline_swift_try_initializer_types_receiver_issue2061: `try await Reader(...)` and `try ServerSocket(...)` type the variable; a subclass receiver reaches its base class and the base class's extension in another file, not same-labelled decoy members. swift_conditional_retry_keeps_the_better_parse_issue2061: an `#error` line from a dropped `#else` branch in a class body keeps the first parse, so the class and its initializer stay in the graph. The pipeline tests run sequential and parallel. Each test fails with its fix reverted. Signed-off-by: Martin Vogel <[email protected]>
Brings the branch up to date with main (612eee2). Two textual conflicts, both where main added its Java external-call guard (DeusData#2667) next to this branch's Swift candidate guard: - src/pipeline/pass_calls.c: the registry lookup still runs only when the Swift candidates allow it, and now passes java_external. - src/pipeline/pass_parallel.c: the registry fallback condition keeps both !java_external and count == 0. No other file conflicted. Signed-off-by: Martin Vogel <[email protected]>
|
Thank you, @DeusData . I haven’t started working on these fixes yet, so there are no changes from my side to coordinate. I appreciate you handling the fixes and rerunning the census. |
|
Hi @DeusData, Thank you again for taking this on. Following my earlier reply, I ran some verification in an isolated Docker environment, comparing I reproduced the generic-return and There was one difference in my run: These observations are specific to |
|
Thank you again, @DavidHLP, for holding the branch for us. As promised, we made the four changes from our last comment ourselves. They are on feat/issue-2061-swift-identity as separate commits on top of your 1a6b292, followed by a merge of current main. Please pull before pushing again. Thank you also for your verification notes from November 5. Your trace was spot on: a failed initializer inference ended up on the incompatible-receiver path instead of falling back to the label match, which is exactly the path the first commit below changes. And your BufferedReader.swift:145 result matches ours - see the note under commit 2.
Same census as before, 1a6b292 -> this branch (Alamofire bda9ed57, swift-nio eabcc14f, helm 9b88efd6, every run indexed from scratch):
One example from our comment still has no edge: Alamofire Source/Core/Session.swift:1389. Protected's members end up as free functions in both parses (the We are also still looking at a few further points and will follow up on those separately. Thank you again for this series and for your patience with our review rounds. |
cppcheck 2.20 (the CI lint) reports variableScope at src/pipeline/registry.c:4176: swift_candidates_for_file declared `graph` at function scope, but only the named-receiver branch reads it. Declare it inside that branch. No behaviour change. Signed-off-by: Martin Vogel <[email protected]>
A receiver typed only by its constructor call (`let x = Foo()`, also
through `try` / `await`) now narrows the label match the way a receiver
with an inferred initializer type already does: a member the type has
still wins, and when the type has no member with the call's name and
labels, the call keeps the label match instead of getting no edge. One
such site is Alamofire's Session.swift:1389, where `mutableState` is a
`Protected(...)` whose members the parser leaves as free functions.
This applies only to a type the project declares. A constructor of a
type the project does not know (DispatchSemaphore(), Task {...}) or only
extends (Data()) still gets no edge: on swift-nio and Alamofire a label
match for those receivers mostly lands on unrelated project methods,
for example `semaphore.wait()` on EventLoopFuture.wait. A name the
extractor reads as a constructor but that resolves to a project function
(`Foo.make()`) keeps the label match like an inferred call result.
Annotations and parameters still decide alone, and inferring an
initializer's type still never uses a label match. The extractor marks a
binding as `constructed` when its type comes from the initializer's
constructor call rather than from an annotation.
On swift-nio eabcc14f and Alamofire bda9ed57 this adds 947 and 166 call
edges and removes none. pipeline_swift_constructor_receiver_keeps_label_match_issue2061
(sequential and parallel) pins the fallback, a member that still wins,
and the annotated, parameter, unknown-type and extended-type cases.
Signed-off-by: Martin Vogel <[email protected]>
Brings the branch up to date with main (e2a93e4): DeusData#2205 (daemon UI setting on manual start), DeusData#2905 (Retrofit client HTTP_CALLS) and two GitHub Actions version bumps. No conflicts. Signed-off-by: Martin Vogel <[email protected]>
The previous commit kept the label match for a constructor-typed receiver only when the project declares the constructed type. After the census the maintainers decided to recover every constructor-typed receiver like one with an inferred initializer type: when the type has no member with the call's name and labels, the call keeps the label match, whether or not the project declares the type (NSLock(), DispatchSemaphore(), Data()). For constructors of types the project does not declare, that label match is often an unrelated project method (a semaphore's wait() lands on EventLoopFuture.wait on swift-nio). It is the same label match a receiver of unknown type gets; tightening it for receiver chains that pass through a property is a separate follow-up. Annotations and parameters still decide alone, and inferring an initializer's type still never uses a label match. The NSLock and Data rows of pipeline_swift_constructor_receiver_keeps_label_match_issue2061 now expect the label-matched edge. Signed-off-by: Martin Vogel <[email protected]>
|
Thank you again, @DavidHLP, for holding the branch for us while we finished this round. We added three small commits on top of aeeb271 and merged current main. Please pull before pushing again.
Same census as before, aeeb271 -> this branch (Alamofire bda9ed57, swift-nio eabcc14f, helm 9b88efd6, every run indexed from scratch):
The added edges come from the same label match that main uses, so they share its limits. For constructors of types the project does not declare, the label match is often an unrelated project method (on swift-nio a semaphore's wait() lands on EventLoopFuture.wait). The most common miss on swift-nio is We are also still looking at a few further points and will follow up on those separately. Thank you again for this series and for your patience with our review rounds. |
|
Hi @DeusData, thank you for the latest fixes. I pulled your branch updates before pushing. I added one follow-up commit, cdb7a293, to address the Ubuntu CI failure. Your constructor/inferred-receiver fallback changes remain in place. Navigation CI failure and fixThe failed Ubuntu job stopped at I reproduced this in isolated Linux Docker runs by changing only the repository directory name:
The service classifier was treating the project-identity prefix of a resolved QN as library evidence. The exact random directory name was not retained in the CI artifact, so I cannot identify its particular suffix. The fix excludes the exact project prefix when classifying graph-qualified symbols, in both sequential and parallel resolution. Raw external library identities retain their existing treatment. The existing navigation test keeps all its assertions and now deliberately includes Validation of the follow-up code: Ubuntu 24.04 x86_64 Docker, GCC 13.3, ASan/UBSan, isolated HOME/Git configuration, network disabled:
The new GitHub workflow is still running as I post this; DCO has passed. Separate Swift finding for your reviewI also found an explicit-type boundary case while independently verifying the previous head. This valid Swift example produces a false edge from the outer import Foundation
class Decoy { func wait() {} }
func identity<T>(_ value: T) -> T { value }
func declaredShadow(_ flag: Bool) {
let semaphore: DispatchSemaphore = DispatchSemaphore(value: 0)
if flag {
let semaphore = identity(DispatchSemaphore(value: 0))
semaphore.signal()
}
semaphore.wait()
}With the same fixture, The trace points to I have not rerun the full Alamofire/swift-nio/helm census on |
|
Merged - thank you so much, @DavidHLP, for this whole series, for holding the branch while we worked on it, and for tracking down the last CI failure so carefully. Your diagnosis of the Ubuntu failure was exactly right, and it was not specific to this branch: we reproduced it on main as well. The navigation fixture from pipeline_arg_url_skips_client_navigation, indexed from a directory whose name contains Thank you also for the Swift shadowing example. We confirmed it, and it comes from our own commit 8372793, not from yours: when an inner same-name binding cannot be inferred, the lookup falls back to the label match although the outer binding has a declared type. Treating that case as a conflict would also drop correct edges on Alamofire's Where this series lands: declarations keep their identity across files, extensions stay within their SwiftPM target, conditional compilation is read instead of dropped, and on swift-nio and Alamofire the typed call graph is both larger and more precise than before, with repeated runs byte-identical. We are also still looking at a few further points and will follow up on those separately. Thank you again for your persistence and care throughout this series. Merged in 0d835c5. |
|
@DavidHLP, a short follow-up to our merge note: the Swift shadowing case you found is now fixed on main by #2947 (merge 50079ed). A receiver now reads the binding that is actually in scope at the call, so your semaphore example no longer gets the false Decoy.wait() edge, while Alamofire's |
A receiver's type came from every same-name binding of the enclosing
function, wherever in the function it was declared. When an inner
binding's type could not be inferred, the receiver counted as unknown
even where only an outer, declared binding was in scope, and the call
took the label match. In
let semaphore: DispatchSemaphore = DispatchSemaphore(value: 0)
if flag {
let semaphore = identity(DispatchSemaphore(value: 0))
semaphore.signal()
}
semaphore.wait()
the outer wait() got an edge to an unrelated project wait() (DeusData#2436).
Each Swift binding now records where it is in scope: from the end of
its declaration to the end of its block, where a parameter's block is
its function. Only the bindings in scope at the call count, and of
those the innermost block's, so a local shadows a same-name local of an
enclosing block or a parameter. `var request = try request.make()`
reads the parameter on its right-hand side and the new variable after
it. Bindings of the same block that disagree still select nothing, and
fields and module-level bindings are unchanged.
pipeline_swift_in_scope_binding_issue2436 (sequential and parallel)
covers the example, the shadowed-parameter idiom of Alamofire's
encoders and two sibling blocks; it fails without the change.
Signed-off-by: Martin Vogel <[email protected]>
Brings the branch up to date with main 6d34ca1. Two resolutions: - tests/test_parallel.c (textual conflict): main added parallel_csharp_nested_type_and_global_namespace_binding and parallel_csharp_parse_damaged_namespace_is_not_global at the same spot as this branch's parallel_csharp_keyword_receiver_no_name_guess_issue2121. Resolved as main's file plus this branch's own hunk (all three tests and their RUN_TEST lines kept); the zero-context patch-id of the branch's change to the file is unchanged. - src/pipeline/registry.c (semantic conflict; merged cleanly as text but would not compile): main's cbm_registry_swift_candidates (DeusData#2436) calls receiver_chain_admits with two arguments, and this branch gives it a third, the calling file's language. The Swift caller now passes CBM_LANG_SWIFT. Behaviour is unchanged: the type-keyword branch applies to CBM_LANG_CSHARP only. Signed-off-by: Martin Vogel <[email protected]>
|
@DeusData Thanks for following up and for fixing the shadowing case in #2947. I appreciate the scope-aware approach and the care taken to preserve Alamofire’s shadowed-parameter pattern. The separate follow-up for typed property chains makes sense as well. Thank you for the detailed explanation and validation throughout this series! |
Summary
Refs #2061. Give Swift overloads declaration-qualified identities, preserve bare node names, and use the uncapped selector inside receiver-typed candidates. Keep static/class, async, and explicit return markers. Qualified types retain their last segment, and the associated-type spelling in
where T.Elementis retained pending maintainer follow-up.Package.swiftinvalidate source-only repair and trigger a full reindex.e0c7a1da, including fix(extraction): attribute Swift lifecycle calls #2395.Validation
AI assistance: This update was prepared with an AI agent acting on behalf of DavidHLP.
1a6b2923passed the default production regression guards. On Alamofirebda9ed57, the chain tails atTests/SessionTests.swift:1632andTests/DataStreamTests.swift:582each resolved to the expected method. On swift-nioeabcc14f, the NIOFS and _NIOFileSystem cross-target CALLS check returned zero.SourcetoSource/Coretriggeredincremental.force_full. The rebuilt and fresh graphs of that same modified checkout matched on node and edge rows, including properties. Earlier general incremental comparisons recordedSIMILAR_TOanddecorator_tagsdifferences separately.1a6b2923plus the remaining test-environment and baseline fixes from draft fix: isolate validation and handle empty initialization state #2809, retaining main's mimalloc 3.5.3. The defaultscripts/test.shpassed with GCC, ASan and UBSan, andUBSAN_OPTIONS=halt_on_error=1. It completed 192 suites with 10,363 passed, zero failed, and 10 skips. Theincrementalsuite ran all 164 tests using FastAPI0.99.1atdd4e78ca, with no skips. The fix: isolate validation and handle empty initialization state #2809 changes remain separate from this PR.1a6b2923passed Linux, macOS, Windows CLANG64, TSan, lint, CodeQL, DCO, shard completeness, andci-ok. PR workflow · DCO