Add ACP access to existing Sidekick conversations #961

Story

As a person working with an external orchestration agent, I want it to prompt the existing Sidekick over ACP while I observe and steer the same work in xmd repl, so orchestration does not depend on terminal focus, pasting or Send/Enter choreography.

For example, the external agent asks Sidekick: “Have the existing Architect review this exact commit, preserve its session, and return its verdict with links to the evidence.” I see the message, Sidekick's response, its XMD programs and resulting work in the existing UI. The external caller follows the same interaction and receives its outcome.

The external caller takes the person's place as a client of Sidekick for a delegated request. Sidekick remains responsible for creating and executing its XMD programs.

External orchestration agent
          |
          | ACP prompt and session updates
          v
Existing Sidekick conversation <---- Person observes and steers
          |
          | Sidekick creates and executes XMD
          v
Existing controls and REPL entry admission
          |
          v
Results, responses and execution evidence

Current gap

Sidekick already provides the conversation, coroutine-owned execution environment, captured vocabulary, control evaluation, questions and feedback used for orchestration. An external orchestrator reaches it through terminal UI input and separately inspects public records to establish acceptance and completion.

Uncertain delivery can cause duplicate work or skipped work. Adding an independent program executor would introduce a second source of execution authority and interaction dynamics the person must reconcile. The immediate need is reliable access to the existing Sidekick relationship.

Delivery order and shared dispatch

Deliver the prerequisites independently and demonstrate each through the existing UI before adding the ACP adapter:

This remains the ACP adapter Story, not an umbrella replacing those prerequisite contracts. Shared dispatch and queue behavior are implemented and reviewed in their owning tickets; the adapter does not reimplement either.

The UI and ACP handlers call the same contextual dispatch operation, with caller context identifying user or ACP origin. Origin is captured at admission so a later queue worker retains the actual sender; it is attribution metadata rather than authority. The stream follows that exact message through its full Sidekick exchange and final result. Consuming an update or observing the result never redispatches source.

ACP serving and accepted-message processing belong within the existing Sidekick lifetime. A thin stdio bridge may be required because ACPX launches an agent command; it forwards to that live owner rather than owning another conversation, writer or evaluator. The terminal input dispatcher continues to resolve UI actions before invoking semantic Sidekick dispatch.

User UI ----- origin: user ----+
                              |
                         Sidekick dispatch
                              |
ACP handler -- origin: acp ---+
                              |
                      Visible message queue
                              |
                        Active exchange
                              |
                    Activity and result stream
                       /                 \
                      UI              ACP response

The dependency order here does not insert these tickets into or reorder the separately authorized delivery stack.

Accepted first-delivery direction

• Expose prompt-based ACP access to Sidekick attached to an already-running REPL. XMD acts as an ACP agent provider for this interaction; the external orchestrator is its client. This does not replace Sidekick's configured underlying agent or change its provider identity.
• Sidekick creates and executes the programs in its existing conversation-owned environment. The client supplies prompts, not roots for direct evaluation. XMD source inside a prompt remains message content; it is not automatically admitted as an executable response.
• Use the same visible conversation and execution activity as the UI. Identify externally supplied messages truthfully. The person can observe and steer without competing with an invisible controller.
• Preserve the existing Syntax, guides, captured controls, coroutine ownership, permission boundaries and entry admission. There is no ACP-specific XMD language or second capability catalogue.
• Consume the visible queue delivered by Queue follow-up messages visibly in Sidekick #964 through Add a shared Sidekick message dispatch stream #963's shared dispatch stream. UI messages and ACP prompts delivered to the owner use the same admission and ordering rules, with retained caller origin and queued/running/settled state. The adapter adds no overlapping evaluator, second writer or concurrent ordinary entry; human steering, priority, withdrawal and interruption behavior comes from the accepted Queue follow-up messages visibly in Sidekick #964 contract.
• Continue the actual existing Sidekick conversation and named downstream role sessions. Do not silently initialize replacements, open a second journal writer, alter configuration, or replay uncertain feedback.
• Return actual responses and make the admitted source and relevant public execution evidence accessible when a summary is insufficient. Preserve full first failures and partial results.
• Reuse existing REPL URLs for admitted entries and conversation locations where applicable. Decide whether a further interaction URL is necessary from the accepted journeys; no MCP-specific route or new job identity is prescribed.
• Starting a REPL, creating a headless environment, direct program injection and a separate MCP execution endpoint are outside the first delivery.

Use cases to support

1. Learn and inspect. Prompt Sidekick to inspect current environment facts, available components, bindings, entries and completed public Prompt results without submitting new ordinary work. The caller receives an answer with inspectable evidence. Asking Sidekick may use its model; inspecting earlier work must not replay that work or contact its original producer.
2. Delegate an exact existing-role request. Ask Sidekick to invoke an existing Architect, Planner or Implementor against a specified revision, with explicit constraints and requested evidence. Inspect the actual program, admitted entry and outcome. A completed agent turn is distinct from its product verdict, such as REQUEST CHANGES.
3. Recover after a lost response. Establish whether the exact prompt was accepted, what Sidekick actually attempted and which entries resulted. Reconnect and follow that interaction without blindly resending it. Genuine unknown acceptance remains explicit; ACP request IDs alone do not establish durable acceptance or exactly-once external effects.
4. Follow long-running work. Follow ACPX's normal prompt stream through the Sidekick turn's actual completion. Observe responses, control activity, waiting questions and entry outcomes in the existing UI. Distinguish live running work from saved unfinished history. No custom brief observation window, early-return handle or separate polling workflow is required.
5. Observe and steer as a person. The person can read the external request and Sidekick's work, ask a question, correct the request, answer an elicitation or stop applicable work under explicit shared-session rules. Neither participant silently overwrites a draft, steals an answer or starts competing work. The precise arbitration and handover behavior is a design gate.
6. Make serial handoffs. The external orchestrator asks Sidekick to continue to a successor only after the preceding work actually completes and joins. Keep both Sidekick conversation admission and the ordinary entry slot authoritative. No new scheduler, automatic retry loop or competing coordinator is implied.
7. Understand a blocked interaction. Expose the actual acceptance, provider, evaluation or feedback failure with its evidence. Do not bypass a blocked conversation, fabricate acceptance, reset history or switch identities. Report whether any safe continuation exists under the existing rules.
8. Compose XMD through Sidekick. Ask Sidekick to read saved evidence, transform it with ordinary components such as Query, and present a useful result. It uses the same Syntax and guides it uses for a person's request. The client can inspect what was actually authored and executed.
9. Use XMD as an agent provider for XMD. An XMD caller can use its existing ACP client capability to prompt an XMD-backed Sidekick and receive its responses and activity. Demonstrate this against a prepared existing Sidekick under the first-delivery attachment boundary. Automatic child-environment creation, recursive orchestration and an additional ownership framework are not requirements.

Primary journey for payload and lifecycle refinement

Use “Ask the existing Architect to review this exact commit and return its verdict” before freezing an adapter or payload schema.

1. The caller attaches to the intended existing Sidekick and sends one prompt. The person sees the same accepted message, with its origin.
2. Sidekick authors its program and invokes the existing role through normal entry admission. The actual source, entry URL and observable activity are available.
3. The caller follows normal ACP session updates until the Sidekick turn actually completes. Use ACPX's existing prompt/completion behavior; do not introduce a brief observation window or report early turn completion.
4. Sidekick returns the requested answer and evidence. Program completion, downstream agent completion and review approval remain distinct facts.
5. On lost transport, interruption or a failed downstream read, expose what completed, what failed and what remains unknown. Reconnection does not itself create a new turn or repeat an effect.

For simple inspection, return the requested answer rather than only a location for retrieving it. For consequential decisions, give access to the relevant actual public results and admitted source. Whether evidence travels inline, through existing links or through an additional bounded read remains an open design choice.

ACPX compatibility

The Product Owner explicitly requires ACPX compatibility. Demonstrate the provider through a declared released ACPX CLI version as a custom agent and through XMD's actual embedded ACPX client path. A generic ACP test client alone does not satisfy delivery.

The current repository embeds a patched ACPX 0.12.0 snapshot, documented in packages/acp/vendor/acpx/PROVENANCE.md; current public ACPX CLI documentation describes additional behavior. Record the versions and capabilities actually tested. Do not assume a newer CLI feature exists in the bundled runtime or silently upgrade that runtime as part of this Story.

Trace initialize, session setup/attachment, prompt, session updates, exact-session reconnect/load, cancellation and connection/session close against each declared client. Keep ACP JSON-RPC output separate from terminal UI and diagnostic output. Use the supported custom-agent launch/configuration path; the public command spelling remains a design decision.

Expose the same live Sidekick owner through the serving/attachment path, with no second writer or separate execution environment. Distinguish the ACPX local record, the XMD provider's ACP session, the attached Sidekick conversation and Sidekick's underlying agent session. Loading or reconnecting must not silently replace the selected Sidekick. Advertise only capabilities and controls the attached owner can actually honor.

Resolve these specific compatibility boundaries in the proposal:

• Sidekick's newly requested visible message queue versus ACPX's client-side prompt queue. The same Sidekick owner accepts eligible follow-ups from the UI and delivered ACP requests while busy. Preserve the distinction between ACPX accepting a prompt locally, Sidekick actually receiving and accepting it into its queue, and the message beginning execution. A prompt still held only by ACPX cannot be reported as visible or accepted by Sidekick. Show how ordering and backlog remain understandable without inventing delivery evidence or duplicating queues.
• ACP prompt/turn completion versus completion of one underlying model turn while Sidekick may still evaluate a program or process feedback.
• Map ACPX's supported timeout, Ctrl+C, cancellation and session-close behavior to Sidekick's existing turn and admitted-entry lifetimes. Use normal ACPX behavior as the baseline; no custom observation-window or detached-task protocol is required.
• XMD's existing explicit first-turn materialization contract, checkpoint metadata where applicable, session identity checks and normalized event surface. A successful session/new or first streamed message is not evidence of Sidekick prompt acceptance.
• Public event/results delivery versus a client's local process status: client liveness is not proof that a Sidekick entry completed or its cleanup joined.

Use finite deterministic compatibility journeys: attach to a prepared Sidekick, two prompts in the same conversation, observable program/entry results, exact-session reconnect, busy/human intervention, blocked acceptance, failure, cancellation and close. Exercise XMD-to-XMD through the existing embedded client as well as the ACPX CLI. Any demonstrated need for new generic protocol/ownership machinery or a bundled-client upgrade returns to the Product Owner before expansion.

Primary references: ACPX custom agents, sessions, and prompting. Verify behavior against the declared release and pinned local runtime, rather than relying only on evolving website documentation.

Open questions before implementation

1. Attachment and ACP session mapping. How does the client name an existing execution and Sidekick conversation? How do supported ACP initialization and session-loading capabilities map to that live owner without creating a new underlying conversation or another writer? Select and document the supported ACP revision and client behavior.
2. Human and external-client participation. Apply the ordering, steering, withdrawal and interruption contract settled in Queue follow-up messages visibly in Sidekick #964. Show how ACP requests and cancellations target the correct message through shared dispatch, while validation/permission/blocked-state refusals remain truthful. Distinguish any backlog ACPX has not yet delivered. Any demonstrated mismatch between normal ACPX behavior and the delivered Sidekick contract returns to the Product Owner; do not add a separate adapter queue or arbitration path.
3. Acceptance and reconnect. Which existing canonical records establish that this exact prompt was accepted and carried? What can a reconnecting caller inspect if the first response was lost? Distinguish request receipt, provider acceptance, program admission and completion. Identify a concrete missing fact before proposing new records or correlation mechanisms.
4. Responses and evidence. Which Sidekick events map to ACP message and tool-activity updates? How are actual XMD source, control outcomes, complete failures, paginated evidence and existing URLs available? Do not expose hidden reasoning or native credential-bearing session files, or equate streamed text with committed execution.
5. Cancellation and questions. Map normal ACP/ACPX cancellation and connection/session close to Sidekick Stop, pending questions and separately owned admitted entries. Decide who can answer or cancel each request and when cleanup has actually joined. Reconnection must not replay work or fabricate acceptance. There is no additional short-wait/detachment experience to design.
6. Blocked state and provider behavior. What does the adapter expose when Sidekick's existing acceptance or feedback guard refuses continuation? Preserve first errors and uncertainty; transport access does not authorize recovery bypass. Identify whether any separate prerequisite is necessary.
7. Minimal transport and XMD consumption. Choose the smallest ACP serving/attachment path compatible with the current live owner and existing XMD ACP client. Show configuration and one prepared-instance XMD-to-XMD journey without adding remote hosting, automatic environment creation or another execution engine.

Return one bounded architecture proposal with existing production-path tracing and a concrete attach, prompt, observe, human-intervene and reconnect journey before plan freeze or implementation. Protocol selection is ACP for this first direction; the unresolved lifecycle, presentation and mapping choices remain product decisions.

Bounded orchestration experiment

Use the ACP interface to test whether the existing Sidekick can take over the delivery-manager/orchestrator role, with the person observing the same UI. This experiment does not require transferring the external caller's conversation into Sidekick or changing the ongoing stack.

The external caller supplies one bounded goal and the necessary brief: actual source and revision, original role identities, authority limits, focused evidence, review gates and the agreed stopping point. Sidekick creates and executes the programs, coordinates the eligible roles serially, follows their actual outcomes and either reaches the stopping point or returns a concrete blocker or product question.

The caller remains an observer and evaluator after the initial brief. Success means Sidekick completes the bounded orchestration without the caller forwarding each stage, operating terminal controls or becoming a hidden second coordinator. Inspect the actual source, admissions, role results, joining and review evidence rather than accepting a self-report. Record every intervention the caller or person needed and why.

Choose a small authorized task and a finite stopping point before running the experiment. Preserve existing provider, ownership, signing, permission and delivery constraints; never duplicate active work or move the current queue implicitly. Identify specific capability or continuation gaps before proposing extensions.

This is a validation use case for the ACP interface, not approval for a generic orchestration framework, autonomous task creation or migration of the existing live stack. ACP makes it possible to conduct the experiment through the same prompt-based relationship used by the person; it does not itself establish that Sidekick can perform the role.

Product verification proposal

• Attach an actual ACP client to a prepared existing Sidekick. Submit one prompt, observe its origin in the UI, and show the real Sidekick-authored program, ordinary entry admission and outcome. Terminal keystrokes, a hidden executor or an unrelated new conversation fail this comparison.
• Use a deterministic Sidekick/provider fixture to inspect bindings and documentation, then delegate a revision-specific role request. Compare the actual submitted source and canonical public results with what the caller and person receive. A model summary alone is insufficient for claiming the requested effects occurred.
• During active work, submit eligible follow-up messages through the UI and ACP path. Show their actual arrival, origin and queued/running/settled state under the same Sidekick owner. Verify agreed ordering and human steering/withdrawal behavior, preserving drafts, questions and existing controls. Queuing starts no concurrent evaluation or ordinary entry. Distinguish ACPX-only queued acknowledgment from actual Sidekick acceptance; invalid or blocked requests retain their real refusal.
• Drop the connection at the agreed acceptance boundaries, reconnect and inspect the same interaction. Count actual provider invocations and entry effects. Unknown acceptance remains unknown; observing or reconnecting never automatically resends a prompt.
• Exercise normal ACPX streamed output and turn completion, partial output followed by failure, a question, cancellation and session close. Verify their actual effect on the Sidekick turn and any separately owned admitted entry; report cleanup and joining truthfully.
• Exercise a blocked Sidekick and existing permission/disclosure refusals. Preserve the full first failure, original conversation identity and unresolved acceptance; no provider/configuration/history bypass.
• Use the existing XMD ACP client path to prompt the XMD provider attached to a prepared Sidekick. Return its response and evidence links without introducing direct source evaluation or creating child environments.

The accepted plan identifies focused regression entrypoints at the ACP adapter, Sidekick owner, UI projection and existing client boundary. Use deterministic tests; a paid live-agent matrix or broad confidence suite is not a prerequisite.

Relationships and scope

Related to REPL Quest #827 and Sidekick #883. Existing saved-entry inspection #932 and ordinary value processing #934 provide evidence and composition capabilities. Component authoring #896, event assistance #897 and retrospective behavior #898 retain their separate scope. Check overlap with session-continuation #891 and ACPX save reliability #933 where acceptance/reconnect tracing demonstrates a dependency; neither is implicitly reimplemented here.

Relevant repository contracts are specs/repl-spec.md, architecture.md, the Sidekick owner and guides under packages/cli/src/repl/sidekick/ and packages/cli/src/documents/sidekick/, and the existing ACP adapter/session implementation. Protocol references: ACP session setup and ACP prompt lifecycle; the proposal pins the actual supported revision.

This first-delivery direction supersedes the earlier MCP/direct-program proposal for this issue. Exclude a direct-execution MCP interface, a second Syntax catalogue or control runtime, invisible outside execution, new generic persistence/ownership mechanisms, automatic orchestration or retries, provider replacement, headless startup and remote hosting. A demonstrated need for a material extension returns to the Product Owner.

This issue is recorded for delivery planning and is not added to the currently authorized implementation queue.

Delivery audit — 2026-10-31

Merged opt-in local ACP bridge and passive reads attach to the existing Sidekick using shared dispatch, exact load, cancellation and observer ownership. Headless startup remains separately owned by #1005.

Merged delivery: #989.

Existing implementation and regression evidence: packages/cli/src/sidekick-acp/, packages/cli/tests/repl-sidekick-acp.test.ts. This audit reuses the recorded evidence and inspects merged source; it runs no new tests or model calls.