Skip to content

Commit 36ffca5

Browse files
committed
feat(core): the inbound pipeline plans under the lock, delivers in lanes (RMK-93)
The surgery half of the delivery-lanes amendment (roomkit-specs e0aabcc, engine in d7ecbb6). The room lock now ends at broadcast planning: _run_precommit commits the trigger through _commit_to_lane — commit, plan (router.plan against binding state consistent with the commit) and lane enqueue in one gate — and _process_broadcast is gone. Execution, the RFC §10.3 mutation trigger, delivery reporting, the ON_ERROR funnel, side effects and AFTER_BROADCAST all fire from the lane executor (LaneExecutionMixin); response events re-enter as their own commit passes under a fresh room lock, their plans queued behind the trigger. process_inbound/send_event await the full cascade, so the observable contract — the AI response is committed when the call returns — holds across the whole suite. Every committed index reaches the delivery cursor exactly once: the three commit gates (_persist_committed, _commit_indexed, _commit_to_lane) are the only paths to store.commit_event in core, a static guard test enforces it, and no-broadcast commits (BLOCKED events, injected stored-only, child-room traces, streamed segments) advance the cursor opportunistically or as lane entries. Injected events keep their bare delivery shape but ride the lane — no more channel I/O under the lock. A send_event issued from a sync hook or a tool handler short-circuits its cascade wait (deadlock-free) and hands stream consumption to a tracked background task. Broadcast telemetry spans carry their parent across the lane boundary. The Postgres schema backfills delivered_index=latest_index once, when the column is first created — verified against a legacy-schema database with data, and against a re-init. New: RoomKit(delivery_gap_timeout=30.0, delivery_claim_lock_manager=...), framework event delivery_skipped. Full suite 6654 green; PG16 gated tests (multiprocess advisory locks included) 191 green.
1 parent d7ecbb6 commit 36ffca5

17 files changed

Lines changed: 1147 additions & 481 deletions

‎CHANGELOG.md‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,79 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
77

88
## [Unreleased]
99

10+
### Changed
11+
12+
- **The room lock ends at broadcast planning — external delivery moved to
13+
per-room delivery lanes** (RFC §10.1 steps 12-14, §10.2, §13.5; the
14+
roomkit-specs amendment `e0aabcc`). Measured on the scale bench: with
15+
`channel.on_event`/`deliver` (provider round trips, AI generation) inside
16+
the room's critical section, adding workers made rooms *slower* — 74
17+
backends parked on advisory locks while 3-4 worked. Now the lock covers
18+
the pre-commit gates, the atomic commit and the *planning* of the
19+
delivery set; execution runs off the lock, ordered per room by the
20+
`Room.delivered_index` cursor (strict CAS) under a delivery claim — a
21+
derived `__delivery__:{room_id}` key on the existing lock manager. One
22+
lane per room and per process executes only the plans its process
23+
enqueued; the shared cursor forces the global order. Observable
24+
semantics preserved: `process_inbound`/`send_event` still return after
25+
the full delivery cascade (trigger set + every reentry pass it spawned),
26+
so "the AI response is committed when the call returns" holds. Response
27+
events re-enter as their own commit passes (fresh room lock) instead of
28+
being drained inside the trigger's lock tenure — a concurrent inbound
29+
MAY now commit between a trigger and its response (the RFC's explicit
30+
relaxation: index monotonicity and parent linkage, never adjacency).
31+
AFTER_BROADCAST keeps its contract (fires after the event's delivery
32+
set completes); its relative order across trigger/reentries follows
33+
execution order. A worker that commits and crashes before delivering
34+
leaves a cursor hole: the waiting lane skips it after
35+
`delivery_gap_timeout` with a `delivery_skipped` framework event —
36+
the same bounded loss as the previous crash window, now observable and
37+
without wedging the room. The Postgres store backfills
38+
`delivered_index = latest_index` once, when the column is first created
39+
(pre-lane deployments delivered under the lock, so everything stored is
40+
delivered).
41+
42+
### Added
43+
44+
- **`RoomKit(delivery_gap_timeout=30.0)`** — how long a lane waits on a
45+
cursor hole owned by an absent worker before skipping it, and
46+
**`RoomKit(delivery_claim_lock_manager=...)`** — a dedicated lock manager
47+
(own pool) for delivery claims, so claim tenures (which span provider
48+
round trips) cannot starve the room-lock pool commits depend on. New
49+
framework event: **`delivery_skipped`** (`{from_index, to_index}`).
50+
51+
- **First-class Buzz agents: owner commands + `BuzzAgent` lifecycle runner.**
52+
A RoomKit process can now honor the full lifecycle contract Buzz expects
53+
of *every* agent, however launched (the platform's remote-agents spec,
54+
layer L1):
55+
- **Owner control commands.** `BuzzRelaySource` intercepts the platform's
56+
`!shutdown` / `!cancel` / `!rotate` — kind-9, exact content, mentioning
57+
the agent — when authored by the **proven** owner: the NIP-OA auth tag's
58+
attester (Schnorr-verified by buzzkit against the agent's own pubkey),
59+
else the new `BuzzConfig.owner_pubkey`. Commands are consumed before the
60+
pipeline, so the AI can no longer answer its own stop command;
61+
`!shutdown` stops the source gracefully (or defers to the new
62+
`on_owner_command` callback). Fail-closed: no provable owner → commands
63+
stay regular messages, as does any command from a non-owner. Governed by
64+
`BuzzConfig.obey_owner_commands` (default on — a bot with an auth tag now
65+
obeys its owner; set it to `False` to keep the old answer-everything
66+
behavior).
67+
- **`BuzzAgent`** (`roomkit.providers.buzz`): the runner that owns waiting
68+
and dying — attaches the sources, installs SIGTERM/SIGINT handlers, arms
69+
an opt-in `exit_after_inactivity` bound (default off; reaper on its own
70+
timer), and exits every cause through the same graceful path
71+
(`kit.close()` → presence `offline` → sockets closed), returning a
72+
`BuzzAgentStopCause`. Intentional stops are final: the source supervisor
73+
only restarts sources that *raise*, never a clean stop.
74+
- **`BuzzConfig.from_env()`** reads the reserved identity triplet
75+
(`BUZZ_PRIVATE_KEY`/`NOSTR_PRIVATE_KEY`, `BUZZ_RELAY_URL`,
76+
`BUZZ_AUTH_TAG`), fail-closed — a RoomKit agent is launchable by the
77+
same script/unit/entrypoint as any other Buzz agent. New example:
78+
`examples/buzz_agent.py`.
79+
- The `buzz` extra floor moves to `buzzkit>=0.3.0` (adds
80+
`parse_owner_command` and the Schnorr-verified
81+
`BuzzClient.verified_owner_hex` these features are built on).
82+
1083
### Fixed
1184

1285
- **Buzz presence heartbeat no longer dies on a transient failure.**

‎src/roomkit/core/event_router.py‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -165,6 +165,8 @@ def plan(
165165
direct broadcast callers; inbound events from non-writable sources
166166
are blocked before broadcast) yields an empty delivery set.
167167
"""
168+
from roomkit.telemetry.context import get_current_span
169+
168170
if not source_binding.can_write:
169171
logger.debug(
170172
"Source %s cannot write (access=%s, muted=%s) — not broadcasting",
@@ -180,12 +182,22 @@ def plan(
180182
event = event.model_copy(update={"visibility": source_binding.visibility})
181183
# Target bindings include muted channels — they can still read.
182184
targets = self._filter_targets(event, source_binding, context.bindings)
185+
# The caller's span, carried across the lane boundary so the
186+
# broadcast span stays a child of the inbound/send_event span.
187+
parent_span_id = get_current_span()
188+
parent_span_ctx = (
189+
self._telemetry.get_span_context(parent_span_id)
190+
if self._telemetry is not None and parent_span_id is not None
191+
else None
192+
)
183193
return DeliveryPlan(
184194
event=event,
185195
source_binding=source_binding,
186196
context=context,
187197
targets=targets,
188198
exclude_delivery=exclude_delivery,
199+
parent_span_id=parent_span_id,
200+
parent_span_ctx=parent_span_ctx,
189201
)
190202

191203
async def execute_plan(self, plan: DeliveryPlan) -> BroadcastResult:
@@ -210,6 +222,14 @@ async def execute_plan(self, plan: DeliveryPlan) -> BroadcastResult:
210222

211223
telemetry = self._telemetry or NoopTelemetryProvider()
212224
session_id = (event.metadata or {}).get("voice_session_id")
225+
# Restore the planner's span context first: a lane executor runs on a
226+
# fresh contextvars context, and the broadcast span must stay a child
227+
# of the span that planned it (inline callers see their own span).
228+
parent_token = None
229+
if plan.parent_span_id is not None:
230+
parent_token = set_current_span(
231+
plan.parent_span_id, telemetry_ctx=plan.parent_span_ctx
232+
)
213233
span_id = telemetry.start_span(
214234
SpanKind.BROADCAST,
215235
"framework.broadcast",
@@ -231,6 +251,8 @@ async def execute_plan(self, plan: DeliveryPlan) -> BroadcastResult:
231251
# source binding has nothing to transcode from either way.
232252
if not targets or source_binding is None:
233253
reset_span(broadcast_token)
254+
if parent_token is not None:
255+
reset_span(parent_token)
234256
telemetry.end_span(span_id, attributes={"target_count": 0})
235257
return result
236258

@@ -478,6 +500,8 @@ async def _process_target(binding: ChannelBinding) -> None:
478500
result.observations.extend(tr.observations)
479501

480502
reset_span(broadcast_token)
503+
if parent_token is not None:
504+
reset_span(parent_token)
481505
telemetry.end_span(
482506
span_id,
483507
attributes={

‎src/roomkit/core/framework.py‎

Lines changed: 60 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,7 @@
5656
InboundLockedMixin,
5757
InboundMixin,
5858
InboundStreamingMixin,
59+
LaneExecutionMixin,
5960
MembershipMixin,
6061
RealtimeOpsMixin,
6162
RecordingMixin,
@@ -67,7 +68,6 @@
6768
from roomkit.core.transcoder import DefaultContentTranscoder
6869
from roomkit.identity.base import IdentityResolver
6970
from roomkit.models.channel import RateLimit
70-
from roomkit.models.context import RoomContext
7171
from roomkit.models.enums import (
7272
ChannelType,
7373
EventStatus,
@@ -116,6 +116,7 @@ class RoomKit(
116116
InboundIdentityMixin,
117117
InboundLockedMixin,
118118
InboundStreamingMixin,
119+
LaneExecutionMixin,
119120
RegenerateMixin,
120121
ChannelOpsMixin,
121122
RoomLifecycleMixin,
@@ -155,6 +156,8 @@ def __init__(
155156
inbound_rate_limit: RateLimit | None = None,
156157
orchestration: Orchestration | None = None,
157158
persistence_policy: PersistencePolicy | None = None,
159+
delivery_gap_timeout: float = 30.0,
160+
delivery_claim_lock_manager: RoomLockManager | None = None,
158161
) -> None:
159162
"""Initialise the RoomKit orchestrator.
160163
@@ -214,6 +217,18 @@ def __init__(
214217
``PersistencePolicy(exclude_types={...})`` to skip specific
215218
types or ``PersistencePolicy(persist_types={...})`` to
216219
whitelist.
220+
delivery_gap_timeout: Seconds a delivery lane waits on a cursor
221+
hole owned by an absent worker (one that committed events and
222+
crashed before delivering them) before skipping over it with
223+
a ``delivery_skipped`` framework event (RFC §10.2). The skip
224+
bounds the loss to the crash window; it never covers an
225+
index this process holds a plan for.
226+
delivery_claim_lock_manager: Lock manager used for the per-room
227+
delivery claims. Defaults to ``lock_manager``. Supply a
228+
separate ``PostgresAdvisoryLockManager`` (own pool) in
229+
multi-process deployments: a claim is held for the length of
230+
a delivery set — provider round trips included — and must
231+
not starve the room-lock pool that commits depend on.
217232
"""
218233
from roomkit.telemetry.base import TelemetryProvider as _TelemetryProviderCls
219234
from roomkit.telemetry.config import TelemetryConfig as _TelemetryConfigCls
@@ -246,6 +261,16 @@ def __init__(
246261
"as PostgresAdvisoryLockManager to avoid duplicate event indices.",
247262
type(self._store).__name__,
248263
)
264+
# Delivery lanes (RFC §10.2): broadcast planned under the room lock,
265+
# executed per room in index order off it. The claim defaults to the
266+
# room lock manager (derived key space); a dedicated manager keeps
267+
# long claim tenures off the commit pool in multi-process setups.
268+
from roomkit.core.lanes import LaneConfig, RoomLaneRegistry
269+
270+
self._claim_locks = delivery_claim_lock_manager or self._lock_manager
271+
self._lanes = RoomLaneRegistry(
272+
self, self._claim_locks, LaneConfig(gap_timeout=delivery_gap_timeout)
273+
)
249274
self._realtime = realtime or InMemoryRealtime()
250275
self._transcoder = DefaultContentTranscoder()
251276
self._event_handlers: list[tuple[str, FrameworkEventHandler]] = []
@@ -468,6 +493,12 @@ async def close(self) -> None:
468493
# Stop all event sources
469494
for channel_id in list(self._sources.keys()):
470495
await self.detach_source(channel_id)
496+
# Drain and seal the delivery lanes BEFORE the channels close: lane
497+
# executors call channel.on_event/deliver, and a bounded drain gives
498+
# queued deliveries their chance while the channels still accept
499+
# them. Whatever outlives the budget is aborted with its cascade
500+
# cancelled — no waiter hangs.
501+
await self._lanes.aclose()
471502
# Then close the channels. In sequence, but shielded from one another:
472503
# they close one at a time, so a close that raises would leave every
473504
# channel behind it holding its media — for a conference channel, a
@@ -500,8 +531,11 @@ async def close(self) -> None:
500531
# Close the conversation store (e.g. release a PostgreSQL pool). The
501532
# store's close() is idempotent and a no-op for a caller-owned pool.
502533
await self._store.close()
503-
# Close the lock manager (e.g. release an advisory-lock pool).
534+
# Close the lock manager (e.g. release an advisory-lock pool), and
535+
# the delivery-claim manager when the host supplied a dedicated one.
504536
await self._lock_manager.close()
537+
if self._claim_locks is not self._lock_manager:
538+
await self._claim_locks.close()
505539
# Close status bus
506540
await self._status_bus.close()
507541
# Flush telemetry (ends active spans, flushes exporter)
@@ -727,24 +761,19 @@ async def send_event(
727761
)
728762
token = set_current_span(span_id, telemetry_ctx=telemetry.get_span_context(span_id))
729763
try:
730-
# Direct injection traverses the SAME locked pipeline as inbound
731-
# (RFC §10.5): index assignment, BEFORE_BROADCAST hooks, edit/delete
732-
# handling, source write-permission gate, persistence, broadcast,
733-
# reentry drain and AFTER_BROADCAST hooks. This keeps a single
734-
# validation/hooks/indexing/persistence model across entry points.
735-
pending_async_hooks: list[tuple[HookTrigger, RoomEvent, RoomContext]] = []
736-
pending_error_hooks: list[tuple[RoomContext, Any, dict[str, Any]]] = []
737-
pending_streams: list[Any] = []
764+
# Direct injection traverses the SAME pipeline as an inbound
765+
# message (RFC §10.5): index assignment, BEFORE_BROADCAST hooks,
766+
# edit/delete handling, source write-permission gate, persistence
767+
# and broadcast planning under the room lock, then its delivery
768+
# lane, reentry passes, and AFTER_BROADCAST hooks. This keeps a
769+
# single validation/hooks/indexing/persistence model across entry
770+
# points.
771+
from roomkit.core.lanes import DeliveryCascade
772+
773+
cascade = DeliveryCascade(room_id, reentry_budget=self._max_chain_depth * 10)
738774
async with self._lock_manager.locked(room_id):
739775
context = await self._build_context(room_id)
740-
result = await self._process_locked(
741-
event,
742-
room_id,
743-
context,
744-
pending_after_broadcast_out=pending_async_hooks,
745-
pending_error_hooks_out=pending_error_hooks,
746-
pending_streams_out=pending_streams,
747-
)
776+
result = await self._process_locked(event, room_id, context, cascade)
748777
# A room that refuses events (RFC §5.1) is the one block this API
749778
# cannot report by returning: its contract is the committed event,
750779
# and handing back one marked DELIVERED for a write that never
@@ -754,14 +783,20 @@ async def send_event(
754783
raise RoomClosedError(f"Room {room_id} does not accept new events")
755784
if isinstance(result.event, RoomEvent):
756785
event = result.event
757-
# AFTER_BROADCAST/mutation and ON_ERROR run outside the room lock (RFC §10.1)
758-
await self._run_deferred_async_hooks(room_id, pending_async_hooks)
759-
await self._run_deferred_error_hooks(room_id, pending_error_hooks)
786+
# The caller observes its event's delivery-set completion (RFC
787+
# §10.1 step 18) — the wait short-circuits when this call was made
788+
# from inside the room's own lane or under its lock, where waiting
789+
# would deadlock; delivery then follows in lane order.
790+
completed = await cascade.wait()
760791
# Streaming AI responses to a directly-injected event are consumed
761-
# outside the lock, exactly like the inbound path — without this a
762-
# streaming provider's reply is generated and then silently dropped.
763-
if pending_streams:
764-
await self._process_streaming_responses(pending_streams, room_id)
792+
# outside the lane, exactly like the inbound path — without this a
793+
# streaming provider's reply is generated and then silently
794+
# dropped. A detached caller hands the consumption to a
795+
# background task once the cascade truly completes.
796+
if not completed:
797+
self._consume_streams_when_cascade_completes(cascade, room_id)
798+
elif cascade.streams:
799+
await self._process_streaming_responses(cascade.streams, room_id)
765800

766801
telemetry.end_span(span_id)
767802
except Exception as exc:

‎src/roomkit/core/lanes.py‎

Lines changed: 29 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -96,6 +96,16 @@ class DeliveryPlan:
9696
# with the execution results (they always persisted post-broadcast).
9797
hook_tasks: list[Task] = field(default_factory=list)
9898
hook_observations: list[Observation] = field(default_factory=list)
99+
# Whether the event's AFTER_BROADCAST hooks (and side-effect persistence)
100+
# fire after its set executes. False preserves the paths that never fired
101+
# them: injected events and a trigger without a source binding.
102+
fire_after_broadcast: bool = True
103+
# Trace continuity across the lane boundary: the caller's current span
104+
# (and its backend context), captured at plan time. The lane executor
105+
# runs on a fresh contextvars context, so without these the broadcast
106+
# span would be an orphan instead of a child of the inbound span.
107+
parent_span_id: str | None = None
108+
parent_span_ctx: Any = None
99109
# True on the root pass of an inbound/send_event call: emits
100110
# ``event_processed`` once the delivery set has executed.
101111
emit_processed: bool = False
@@ -169,19 +179,30 @@ def cancel(self, reason: str) -> None:
169179
self.cancelled = reason
170180
self._done.set()
171181

172-
async def wait(self) -> None:
182+
async def wait(self) -> bool:
173183
"""Wait for the cascade to complete.
174184
175-
Short-circuits when the current task is the room's own lane executor
176-
or holds the room lock: in both cases waiting would deadlock (the
177-
lane cannot progress past the caller), so the call returns with the
178-
event committed and its delivery following in lane order — the
179-
detached completion RFC §10.1 step 18 permits.
185+
Short-circuits (returning ``False``) when the current task is the
186+
room's own lane executor or holds the room lock: in both cases
187+
waiting would deadlock (the lane cannot progress past the caller),
188+
so the call returns with the event committed and its delivery
189+
following in lane order — the detached completion RFC §10.1 step 18
190+
permits. Returns ``True`` when the cascade actually completed; a
191+
detached caller must arrange for ``streams`` to be consumed later.
180192
"""
181193
if self._pending <= 0:
182-
return
194+
return True
183195
if _active_lane_room.get() == self.room_id or self.room_id in _held_rooms.get():
184-
return
196+
return False
197+
await self._done.wait()
198+
return True
199+
200+
async def wait_detached(self) -> None:
201+
"""Wait for true completion, without the short-circuit guards.
202+
203+
For the background consumer a detached caller schedules — it runs on
204+
its own clean task, where waiting cannot deadlock.
205+
"""
185206
await self._done.wait()
186207

187208

‎src/roomkit/core/mixins/__init__.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313
from roomkit.core.mixins.inbound_identity import InboundIdentityMixin
1414
from roomkit.core.mixins.inbound_locked import InboundLockedMixin
1515
from roomkit.core.mixins.inbound_streaming import InboundStreamingMixin
16+
from roomkit.core.mixins.lane_execution import LaneExecutionMixin
1617
from roomkit.core.mixins.membership import MembershipMixin
1718
from roomkit.core.mixins.realtime_ops import RealtimeOpsMixin
1819
from roomkit.core.mixins.recording import RecordingMixin
@@ -35,6 +36,7 @@
3536
"InboundLockedMixin",
3637
"InboundMixin",
3738
"InboundStreamingMixin",
39+
"LaneExecutionMixin",
3840
"MembershipMixin",
3941
"RealtimeOpsMixin",
4042
"RecordingMixin",

0 commit comments

Comments
 (0)