Certify local ACP agents with a reusable XMD example #743

Story

As someone integrating an ACP agent with XMD, I want to configure and certify it locally using a reusable executable example, then use the same local integration for orchestration without first submitting an upstream registry change.

For example, I supply a local configuration for Kimi, run the shared certification workflow, inspect which capabilities passed, and use that configuration in an ordinary XMD workflow. The same mechanism handles Gemini or Devin with another configuration.

Common path

A checked-in workflow under examples/agent-certification/ accepts a local agent configuration and selected scenarios. Its documented entrypoint runs through xmd run and the real production ACP/ACPX integration.

The configuration identifies the agent command/adapter, selected model and effort where supported, and the capabilities the integration expects. Local configuration and credentials stay local. No upstream pull request, hard-coded built-in agent name, copied provider-specific smoke document, or global registry edit is required.

Successful local certification leaves a configuration usable by ordinary XMD orchestration through the same production registration path. Certification does not substitute a private provider path that only its own tests can use.

Current gap

The existing qualification plan is provider-neutral in intent but requires a name the production host already supports, depends on #740's older Devin integration, and requires a Devin report to close. That prevents the desired local-first workflow and treats older Devin assumptions as prerequisites.

ACPX and agent adapters have changed since that investigation. Refresh their actual released behavior before settling capabilities; do not carry forward an old restriction merely because it appeared in an earlier discovery report.

Contract

• Provide one reusable example and shared qualification machinery. Another agent supplies configuration and expectations rather than another certification implementation.
• Report basic authentication/prompt streaming/completion, same-session multi-turn continuity, session separation, model/effort selection and continuity across reconnect, cancellation, cleanup, and cross-invocation resume as distinct capabilities.
• Test supported capabilities against actual behavior. Distinguish pass, failure, unsupported and not run. A configured expectation is not evidence of capability, and an agent lacking durable resume cannot receive a durable-session certification.
• Two sequential Prompts in one Session establish continuity through a private random marker recovered by the second turn. Separate Sessions remain independent.
• A bounded cancellation scenario attributes termination to XMD's requested cancellation/deadline and joins owned cleanup. Unrelated failure, disconnection, process death or a surviving process cannot count as a pass.
• Keep invocation-scoped and durable lifetime policies explicit and production-owned. Certification grants no durable identity and never promotes display metadata or a transport acknowledgement into backend acceptance. Preserve Materialize fresh ACP sessions at the first consuming operation #648's accepted boundary where applicable.
• Live scenarios are opt-in and cost-bearing. Document authorization for the selected scenarios, including cancellation of the suite's own test turn. Ordinary CI runs deterministic controls without credentials or model calls; no recurring paid recertification matrix is introduced.
• Produce a sanitized report with a closed schema: categorical outcomes, booleans, counts, sanitized agent/adapter/runtime versions and platform facts, exit state, and reproducible configuration/source fingerprints. Prompts, responses, private markers, credentials, environment values, paths, raw identifiers and raw provider output stay out of the shared report.

Acceptance and evidence

1. A new local agent configuration runs through the example and is usable by an ordinary XMD orchestration document without an upstream change. A deliberately bad command or unsupported expectation produces a precise non-pass outcome.
2. Automated controls exercise report validation/sanitization, marker continuity, independent Sessions, model/effort mismatch, deadline attribution, incomplete runs and joined cleanup without credentials or model cost.
3. One explicitly authorized Kimi qualification uses the production local-integration path and emits the sanitized capability report at an exact source/configuration/adapter revision. Record unsupported capabilities honestly.
4. Evaluate Gemini as the next agent through the same workflow and configuration mechanism. Document actual availability and supported orchestration capabilities; an unavailable credential or unrun scenario is not a successful certification.
5. The example documents setup, invocation, selected-scenario authorization, report interpretation and ordinary local orchestration with the certified configuration.
6. Preserve production permissions, credential handling, session lifetime, cancellation and retention rules. Certification does not weaken these to make an agent pass.
7. Record exact automated test paths and sanitized live evidence. Architecture and the ACP client specification identify the shared qualification surface and its capability-specific limits.

Relationships

ACPX released-runtime adoption and patch qualification (#566/#629) precede the integration track's live qualification. The generic workflow can be developed with deterministic agents while those boundaries are settled.

#740 owns Devin-specific production support. Its older invocation-only assumptions require current discovery before implementation. The generic certification engine does not depend on a Devin implementation or Devin credentials, and Devin evidence is not required to close this issue. Devin reuses this workflow when its integration is pursued.

Out of scope

• Publishing private local configurations or distributing credentials.
• Claiming universal ACP compatibility from one agent's passing report.
• Automatically retrying uncertain model turns or weakening durable acceptance.
• Making cost-bearing live qualification part of ordinary CI.