Support overlapping first adapter loads without false recursion errors

What — Please support concurrent first requests for the same DeclaredRegistry adapter without treating a second thread as recursive loading. MolSysMT adoption in uibcdf/molsysmt#382 reproduces one successful request and one KeyError: "Recursive load of plugin 'file_pdb_gz'". Published 0.15.0 documents no concurrent-access synchronization, so this is a consumer capability requirement rather than a claim of a violated published thread-safety promise.

How — Reproduced on Linux/Python 3.14.7, provider source 69f152b, whose runtime has no diff from published tag 0.15.0 (01d83c7), with the prepared MolSysMT DeclaredRegistry(..., load_policy='adapter') adoption:

from concurrent.futures import ThreadPoolExecutor
from threading import Barrier
from molsysmt.form import _dict_modules

barrier = Barrier(2)
def request():
    barrier.wait()
    return _dict_modules['file:pdb.gz']

with ThreadPoolExecutor(max_workers=2) as executor:
    requests = [executor.submit(request) for _ in range(2)]
    adapters = [request.result() for request in requests]
    assert adapters[0] is adapters[1]

Run in a fresh interpreter so the adapter is not already cached. Overlapping requests hit the registry's shared loading marker; true same-thread recursion and independent concurrent requests are currently indistinguishable. A deterministic provider test can hold a synthetic plugin's initialization with events until both requests have started.

Why — MolSysMT's existing reader workflows include concurrent reads from read-only inputs (CIF and new gzip PDB under #385). Source adapter lookup must not introduce a new cold-start refusal for those operations. MolSysMT should not subclass the loader, maintain a second cache, silently suppress the error or require users to warm all adapters first.

Requested acceptance — Both requests return the same correctly identified module, which initializes once. Genuine recursion still fails without deadlock; failure/cause caching and retry, configuration visibility and lazy unrelated-plugin exclusion are retained. Document the exact supported concurrency scope and admit the fix in a public version before updating the consumer floor. Wider mutation/discovery or arbitrary cyclic cross-registry thread safety need not be claimed by this bounded fix.

Consumer record — devguide/pending_proposals/first_conversion_triggers_full_form_registry_loading.md in uibcdf/molsysmt. Coordination — uibcdf/molsyssuite#115. There is no demonstrated scientific corruption or wrong molecular result: the overlapping cold lookup refuses one request.

Diego requests delivery of this proposed correction to the DepDigest team, rather than direct integration by the MolSysMT contributor.

A reviewable prototype is prepared against clean source 69f152b (runtime identical to tag 0.15.0). It changes only source in a temporary copy; the provider checkout, refs and artifacts are untouched. Same development interpreter, no auxiliary environment.

Evidence: the unmodified provider fails both deterministic overlapping-load guards (strict and adapter); the true-recursion guard passes. With the patch, all 55 selected tests pass: the existing declared-registry contract suite plus three new guards. Genuine recursion still raises, and the original cache/failure/visibility/discovery cases continue passing.

Please review the synchronization choice, document its bounded scope, integrate the guards and qualify/admit a fixed public version. The prototype serializes getitem on one registry with a reentrant lock; it does not claim concurrent mutation/discovery or arbitrary cyclic calls between multiple registries. MolSysMT will retain its existing runtime registry until the provider requirement is admitted.

Proposed patch:

--- a/depdigest/core/registry.py
+++ b/depdigest/core/registry.py
@@ -3,6 +3,7 @@
 import os
 from collections.abc import Mapping, MutableMapping
 from inspect import currentframe
+from threading import RLock
 
 from .checker import check_dependency
 from .config import resolve_config
@@ -45,6 +46,7 @@
         self._discovery_mode = discovery_mode
         self._entrypoint_group = entrypoint_group
         self._load_policy = load_policy
+        self._access_lock = RLock()
         self._declarations = {}
         pairs = (
             declarations.items() if isinstance(declarations, Mapping) else declarations
@@ -103,25 +105,26 @@
         )
 
     def __getitem__(self, key):
-        if key in self._overrides:
-            return self._overrides[key]
-        if key not in self:
-            raise KeyError(key)
-        if key in self._loaded:
-            return self._loaded[key]
-        if key in self._failures:
-            raise KeyError(key) from self._failures[key]
-        frame = currentframe()
-        try:
-            caller = frame.f_back if frame is not None else None
-            location = (
-                f"{caller.f_code.co_filename}:{caller.f_lineno}"
-                if caller
-                else "<unknown>"
-            )
-        finally:
-            del frame
-        return self._load(key, key, location)
+        with self._access_lock:
+            if key in self._overrides:
+                return self._overrides[key]
+            if key not in self:
+                raise KeyError(key)
+            if key in self._loaded:
+                return self._loaded[key]
+            if key in self._failures:
+                raise KeyError(key) from self._failures[key]
+            frame = currentframe()
+            try:
+                caller = frame.f_back if frame is not None else None
+                location = (
+                    f"{caller.f_code.co_filename}:{caller.f_lineno}"
+                    if caller
+                    else "<unknown>"
+                )
+            finally:
+                del frame
+            return self._load(key, key, location)
 
     def _candidates(self):
         if self._discovery_mode == "filesystem":

Self-contained proposed guard module tests/test_concurrent_first_load.py:

"""Reject false recursion while preserving real recursive-load refusal."""

import importlib
import sys
from concurrent.futures import ThreadPoolExecutor
from threading import Event

import pytest
from depdigest import DeclaredRegistry


@pytest.fixture()
def host(tmp_path, monkeypatch):
    root = tmp_path / "concurrent_registry_host"
    plugins = root / "plugins"
    plugins.mkdir(parents=True)
    (root / "__init__.py").write_text("")
    (plugins / "__init__.py").write_text("")
    (root / "control.py").write_text(
        "from threading import Event\nentered = Event()\nrelease = Event()\ninitializations = 0\nregistry = None\n"
    )
    monkeypatch.syspath_prepend(str(tmp_path))
    control = importlib.import_module("concurrent_registry_host.control")
    yield plugins, control
    for name in tuple(sys.modules):
        if name == "concurrent_registry_host" or name.startswith(
            "concurrent_registry_host."
        ):
            sys.modules.pop(name)


@pytest.mark.parametrize("policy", ["strict", "adapter"])
def test_two_overlapping_requests_wait_for_one_initialization(host, policy):
    directory, control = host
    plugin = directory / "observed"
    plugin.mkdir()
    (plugin / "__init__.py").write_text(
        'from concurrent_registry_host import control\ncontrol.initializations += 1\ncontrol.entered.set()\nif not control.release.wait(10):\n    raise TimeoutError("test release missing")\nform_name = "Observed"\n'
    )
    entries = DeclaredRegistry(
        "concurrent_registry_host.plugins",
        directory,
        {"Observed": "observed"},
        load_policy=policy,
    )
    started, finished = Event(), Event()

    def second():
        started.set()
        try:
            return entries["Observed"]
        finally:
            finished.set()

    with ThreadPoolExecutor(max_workers=2) as pool:
        first = pool.submit(lambda: entries["Observed"])
        assert control.entered.wait(5)
        other = pool.submit(second)
        assert started.wait(5)
        try:
            # The second call must wait for the real load, not report recursion.
            assert not finished.wait(0.1)
        finally:
            control.release.set()
        first_module = first.result(timeout=5)
        second_module = other.result(timeout=5)
    assert first_module is second_module
    assert control.initializations == 1
    assert entries.loaded_keys() == ("Observed",)


def test_true_recursive_load_still_fails_without_waiting(host):
    directory, control = host
    plugin = directory / "recursive"
    plugin.mkdir()
    (plugin / "__init__.py").write_text(
        'from concurrent_registry_host import control\ncontrol.registry["Recursive"]\nform_name = "Recursive"\n'
    )
    entries = DeclaredRegistry(
        "concurrent_registry_host.plugins",
        directory,
        {"Recursive": "recursive"},
        load_policy="adapter",
    )
    control.registry = entries
    with pytest.raises(KeyError) as error:
        entries["Recursive"]
    assert "Recursive load" in str(error.value.__cause__)
    assert entries.loaded_keys() == ()

Validation command (Python 3.14, twelve workers, pytest-receptor):

python -m pytest -n 12 --receptor=llm tests/test_declared_registry.py tests/test_concurrent_first_load.py

Consumer follow-up: uibcdf/molsysmt#382. Shared adoption coordination remains uibcdf/molsyssuite#115. No provider publication or release action was performed by this handoff.

Serialize overlapping declared registry loads and prepare 0.15.1 (#39)

Integrated the reviewed per-registry RLock in c90edca. Caller capture and the existing recursion/failure machinery remain in their owning registry.

The strengthened event-controlled guards observe actual lock contention rather than a scheduling sleep. Eight overlap cases reproduce the original false-recursion failure before the change; afterward all 62 registry cases pass. They cover strict/adapter × filesystem/entry points, identical successful values, one initialization, shared original failure causes/diagnostics, cached refusal and explicit retry, current visibility and an unrelated import sentinel. Genuine recursion runs with an independent process timeout. The complete private-source suite passes 298 tests with five unavailable sibling-workspace skips; the ten new guards also pass on Python 3.11/3.12/3.13.

Preparing compatible patch 0.15.1 under this issue's public-admission acceptance. Final-candidate full matrix: https://github.com/uibcdf/depdigest/actions/runs/38450876786. Installed qualification adds a reusable isolated probe with a 30-second deadlock bound and a non-reentrant-lock mutation control. Source success is not yet installed/public admission, so #39 remains open.

The documented guarantee covers overlapping declared value requests on one registry. Requests, including different identities, serialize through initialization. Mutation/discovery/retry/refresh/configuration changes need external exclusive coordination. Initializers must not wait for another thread needing that registry; arbitrary cyclic cross-registry/thread/import interactions are outside scope. No MolSysMT floor or runtime changes, conversion/scientific parity or throughput improvement is claimed.

Retire bounded registry probe fixtures after killed-child timeouts (#39)

The final source candidate is e275703. It retains the reviewed runtime correction and additionally makes the parent own probe fixtures so a killed child cannot leak them. Positive/injected-failure/real non-reentrant-lock timeout guards exercise the owning verifier and confirm retirement. The complete private source suite now passes 300 tests with five unavailable sibling-workspace skips; Ruff, formatting, report indexes and all twenty pinned distribution routes pass. Two identified pre-correction local killed-child fixture roots were retired after verifying ownership and no active use.

Final-candidate routine CI 38452565083, suite policy 38452565665 and Conda publication governance 38452565536 succeed. Full matrix https://github.com/uibcdf/depdigest/actions/runs/38452565534 currently has eight successful Linux/Windows cells and four macOS cells queued without assigned runners (checked 2026-10-13T19:19:37Z). The earlier matrix 38450876786 is superseded: eight successful cells, four queued macOS cells cancelled when the corrected candidate was dispatched. Neither partial matrix establishes full platform qualification.

The bounded local warm-getter measurement adds approximately 0.170 microseconds per request (median baseline 0.417 vs RLock 0.587; Linux/Python 3.14.8, empty explicit configuration, seven repeats of 100000 calls). Coarse serialization is accepted for coherent initialization/provenance; this is not a MolSysMT throughput result.

0.15.1 remains a prepared staged-route candidate, not a public release. #39 stays open. Next: complete the same-candidate twelve-cell source matrix, build once in staging, validate the exact immutable installed file on all twelve cells, then tag/Release and promote those same verified bytes, independently verify the public package and hand off to MolSysMT #382 / MolSysSuite #115. No consumer floor, adoption or scientific parity claim is made. Private source/SDK/test resources remain task-owned and retained for those still-active gates; main is clean and synchronized.

Use explicit macOS 15 ARM64 runners for complete release qualification (

Align runner guard and let superseded qualification runs cancel (#39)

Record verified public 0.15.1 admission and resolve concurrent regist…

Resolved and publicly admitted in DepDigest 0.15.1: https://github.com/uibcdf/depdigest/releases/tag/0.15.1. Annotated tag/candidate 13cc73780a7cb645546ae3f396177f98b86a8792; public noarch py_0 SHA-256 2765c692de4e6aacff4197c4226ebb42ec1f0df9edb27e92f196ef1390cb1850 matches the original staging archive.

All twelve source cells (38459369859), producer evidence and twelve clean exact-file installed cells (38462313607) pass. Promotion 38462737850 preserves those same bytes; independent registry/main solver-index/download and a fresh public-only install verify version/origin, public SMonitor closure, fourteen resources, launcher and all contracts. Documentation and source-only Zenodo record 23350934 are independently verified, including downloaded ZIP size/MD5. No Conda archival or consumer scientific parity claim is made.

The owning guards reproduce eight false-recursion cases before the fix and retain real bounded recursion refusal, one initialization, identical values/causes, cache/retry, configuration visibility, selective loading and diagnostics for both policies/routes. The parent-owned installed probe also cleans after actual killed-child timeouts. The complete private source suite passes 300 tests with five unavailable sibling-workspace omissions. Superseded queues, the corrected old-label test failure and cancellation recovery remain recorded.

Concurrent value requests on one registry serialize. Mutation/discovery/retry/refresh/configuration changes require external coordination; initializers must not wait for another thread needing that registry; arbitrary cyclic cross-registry/thread/import interactions remain outside scope.

Archived owning record: devguide/completed_proposals/concurrent_declared_registry_requests.md; normalized proof: devguide/evidence/release_0.15.1_2026-10-13.json. Receiving handoffs: uibcdf/molsysmt#382 (comment) and uibcdf/molsyssuite#115 (comment). Public provider requirement is now >=0.15.1; MolSysMT owns later floor/adoption/conversion/scientific/performance qualification. Poststate main CI 38464003075, policy 38464003606 and governance 38464003598 succeed; release candidate/tag/artifact remain immutable.

Record #39 consumer handoffs and completed owned-resource retirement

Central MolSysSuite #115 reconciles public 0.15.1 at 13cc73780a7cb645546ae3f396177f98b86a8792, original py_0 digest 2765c692de4e6aacff4197c4226ebb42ec1f0df9edb27e92f196ef1390cb1850. Independently verified original twelve source cells/policy, staged producer, thirteen-job installed smoke, same-file promotion, ZIP receipts, stable tag/main index/download and sixteen original runtime file hashes. Local publisher and smoke scope are retained; no full installed pytest or consumer scientific/performance claim.

All ten registered 0.15.1 canonical copies are delivered with exact-source manual policy/Ruff controls. Their hosted conclusions are successful but step arrays empty under #118; manual administrative evidence remains explicit. MolSysMT #382 still owns receiving/parity/speed; no floor/pin/runtime or old release changed. Central immutable receiving/copy/manual evidence at a2092b7ed99e6fce338d986b351ed71b3e52749b.

Navigation Menu

Repository navigation

Support overlapping first adapter loads without false recursion errors #39

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Support overlapping first adapter loads without false recursion errors #39

Description

Activity

dprada commented on Oct 13, 2026

dprada commented on Oct 13, 2026

dprada commented on Oct 13, 2026

dprada commented on Oct 13, 2026

LMMV commented on Oct 13, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions