Skip to content

feat(plugins): validate native execution in the managed replay registry - #4535

Open
thymikee wants to merge 3 commits into
t3/replay-data-contractsfrom
t3/replay-execution-plugin
Open

thymikee wants to merge 3 commits into
t3/replay-data-contractsfrom
t3/replay-execution-plugin

Conversation

@thymikee

@thymikee thymikee commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Summary

Advance #3871: route native replay execution through the existing managed registry with typed compatibility and stale-selection errors. Keep the proposed native contract in #3871 until accepted.

10 files; 316 gross lines against #4534. The replay package remains private and unpublished.

Validation

Published 894007501dd05245beb766dc36c4415e0f5c065a, base 8a215d81e08818cffb02853b13b5b027ef6e94fa. Prior focused results and source approvals retain their named SHA scope. New-head exact affected validation remains pending.

Coverage failed: 18 assertions in two host-process fixture files failed; 20,194 tests passed and 70 skipped. The changed-line gate did not run. GitHub tested synthetic merge 4b55d9f8 into f9b92418. The process owner and both failing test files are byte-identical across that merge, its base, this head and #4534. Linux /proc identity observations displaced mocked ps responses. No unchanged-base Linux execution has established causality; this remains blocked, not infrastructure-cleared.

Local slot-identity fix 19855593 passed isolated red/green and 15 focused cases. Combined review context c9fc5f65 passed 27 focused cases; independent Apex source recheck found no regressions at c9. Neither candidate is published. Size: +34.4 kB unpacked/+9.7 kB download; version startup 19.5→30.0 ms has unresolved cause.

Merge blockers remain C3 integration, native-contract acceptance, npm placeholder/trusted publisher setup, and native/platform/remote verification. Ready for review; no merge clearance.

@thymikee
thymikee added this pull request to stack #4529 November 13, 2026 23:31
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Size Report

Metric Base Current Diff
Installed (including dependencies) 6.08 MB 6.11 MB +34.4 kB
Package (unpacked) 6.08 MB 6.11 MB +34.4 kB
Package (download) 1.84 MB 1.85 MB +9.7 kB

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 19.5 ms 30.0 ms +10.5 ms
CLI --help 41.5 ms 42.8 ms +1.3 ms

@thymikee
thymikee force-pushed the t3/replay-execution-plugin branch from aedf9f5 to 0315a02 Compare November 13, 2026 23:35
@thymikee

Copy link
Copy Markdown
Member Author

At 0315a02, one defect failed CI, and the design needs a decision before merge. The newer head 8180363 changes only packages/replay/src/plugin.ts to declare the factory export explicitly, which looks like the fix for the first point; the next pass will confirm once Repo Guards reruns. The Repo Guards & Compatibility check fails at the import-direction DAG step, and this diff causes the failure. There are no conflicts.

packages/replay/src/plugin.ts:4 uses export default function replayPlugin(). This file is a manifest-declared façade, and scripts/layering/facade-surface.test.ts:21 cannot enumerate that export form. The PR cannot merge while this gate is red. Please copy the Maestro spelling: declare function replayPlugin(): ReplayEnginePlugin and end the file with export { replayPlugin as default };. The rule is that every façade a package manifest declares uses only named export forms, and facadeEntryFiles lists those façades. Later steps in that job (fallow, native-dupes, gate-of-gates) may also fail once this is fixed, because I only inspected the first failing step.

Not blocking, and you can take or leave these: (1) resolveReplayExecution, the execution union, validateReplayExecution and @agent-device/replay have no production consumer at this head, while replay-port still imports @agent-device/ad-replay and the root package still depends on it and on @agent-device/replay-test, so the new package re-bundles engine code that core also ships. (2) validateReplayEngineSetup at validate.ts:162 accepts replayBackend or execution without knowing the backend id, so a mismatch passes setup, shows as 'active' in inventory, and only throws at resolve time without recording slot.failure. It would be better to require execution exactly when the backend is 'ad', with one constant shared with registry.ts:210. (3) replay-execution.ts:5 spells the public contract as typeof the private engine functions, which adds @agent-device/ad-replay and @agent-device/replay-test as runtime dependencies of the plugins package; declaring these port types in @agent-device/contracts next to replay-backend would avoid that. (4) The core-version gate in plugin.ts:10 pins '0.21.38-dev' and will reject the next dev bump, and Maestro has no such gate, so one owner such as discovery could hold the supported range for both. (5) plugin.test.ts:22 feeds { ... } as never to the suite and asserts replay-test's own message, and the 'shares backend claims' test registers only one candidate, so a case with two 'ad' candidates is missing. (6) @agent-device/replay is wired into neither the root build script nor a packed-entry smoke like Maestro's, so nothing proves the packed dist/agent-device.mjs loads through discovery.

Could this land as one layer instead of a standalone package and an unconsumed resolver? That layer would declare the native execution port in @agent-device/contracts, add the factory keyed by the 'ad' backend in one validated constant, switch replay-port's replay/test path to registry.resolveReplayExecution, and drop ad-replay and replay-test from the root dependencies. The ~30-line packages/replay factory would then land with a packed smoke. If the stack must stay split, could this PR be cut to the registry seam with contracts-owned types and no new dependencies, with packages/replay following alongside the routing layer? Either way, the execution port types must move to contracts first, and an ADR 0034 amendment should decide whether native execution is a factory kind on 'replay-backend' or its own plugin kind.

I did not run tests, typecheck, or build and pack @agent-device/replay, so the bundled output and the unresolved-import guard are unverified. I judged the new registry tests from the diff only. Before merge, the façade export must be fixed so Repo Guards turns green, and then the contracts-owned port and its routing consumer need a decision.

@thymikee
thymikee force-pushed the t3/replay-execution-plugin branch from 8180363 to 9f1fa69 Compare November 14, 2026 00:45
@thymikee thymikee changed the title feat(plugins): add optional native replay execution package feat(plugins): validate native execution in the managed replay registry Nov 14, 2026
@thymikee
thymikee force-pushed the t3/replay-execution-plugin branch from 9f1fa69 to 25e621d Compare November 14, 2026 01:05
@thymikee
thymikee force-pushed the t3/replay-execution-plugin branch from 25e621d to 499d4bc Compare November 14, 2026 01:51
@thymikee

Copy link
Copy Markdown
Member Author

The code at 499d4bc is ready; I found no problems that need a change. Not blocking, and fine to take or leave: resolveReplayExecution in registry.ts near-duplicates resolveRegisteredReplayBackend, so one slot.loading promise and a shared createOnce helper could serve both; the SDK re-export of the ReplayExecution types in plugins.ts could wait until #4537 lands or the ADR is accepted; and the references reflow in tsconfig.json could be reverted, with NATIVE_REPLAY_BACKEND in replay-execution.ts moved below the imports. I did not run tests, typecheck or gates locally, and no production route reaches resolveReplayExecution at this head, since routing lands in #4537. The 11 jobs that did not pass were cancelled, not failed, and several of the same jobs pass on a later run set for this head, so CI looks unrelated to the change. Before merge, the maintainer needs to accept the ADR 0034 native amendment, and hosted CI needs to be green on the newest run set.

@thymikee
thymikee force-pushed the t3/replay-execution-plugin branch from 499d4bc to 3e5030d Compare November 14, 2026 05:57
@thymikee

Copy link
Copy Markdown
Member Author

Current registry head is 3e5030d2dc690bddaac1c5b713b9a19b97c2aab2. The contracts-owned execution port, explicit native capability validation, first-claim/failure inventory, named factory export and routing/packed consumers requested in the earlier review are present. The custom winning-package freshness finding was fixed at the registry/composition boundary and independently rechecked; embedded runtimes retain store-free behavior.

I am retaining the small existing registry implementation rather than adding a shared create-once abstraction solely to remove similar lines. The independent smaller-design assessment accepted the three execution entrypoints and accounted for the +34.7 kB package growth (mostly declarations); it did not establish a performance win. The native ADR0034 amendment remains Proposed, pending maintainer acceptance.

Exact-head affected admission passed: 42 passed / 25 skipped, exactCommit=true, checkoutChanged=false. All 19 hosted checks passed. Later source/help fixes are owned by the top routing layer, so this head has not been churned. No registry/source readiness statement here clears C3, live/platform, ADR or npm placeholder/trusted-publisher prerequisites.

@thymikee
thymikee marked this pull request as ready for review November 14, 2026 08:08

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 11 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="docs/adr/0034-provider-plugin-registry.md">

<violation number="1" location="docs/adr/0034-provider-plugin-registry.md:60">
P2: This adds an unaccepted proposal to an ADR marked `accepted`, contrary to the repository’s ADR format. Keep the amendment in #3871 until acceptance, or merge it only after updating the ADR to the accepted decision.</violation>

<violation number="2" location="docs/adr/0034-provider-plugin-registry.md:74">
P2: This overstates the native contract: replay execution has dispatch and progress ports, but no cancellation port, while suite cancellation is a separate dependency. Document the ports per method so plugin authors do not implement a nonexistent cancellation input.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic


Contracts own the inspection, step, suite and factory types.
Native `create()` receives no format-backend host.
Each execution call receives explicit host ports for dispatch, progress and cancellation.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This overstates the native contract: replay execution has dispatch and progress ports, but no cancellation port, while suite cancellation is a separate dependency. Document the ports per method so plugin authors do not implement a nonexistent cancellation input.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At docs/adr/0034-provider-plugin-registry.md, line 74:

<comment>This overstates the native contract: replay execution has dispatch and progress ports, but no cancellation port, while suite cancellation is a separate dependency. Document the ports per method so plugin authors do not implement a nonexistent cancellation input.</comment>

<file context>
@@ -55,6 +55,33 @@ had to fake the store to load a plugin from its own dependencies (#3747).
+
+Contracts own the inspection, step, suite and factory types.
+Native `create()` receives no format-backend host.
+Each execution call receives explicit host ports for dispatch, progress and cancellation.
+A format factory still receives `ReplayBackendHost` for source collection, execution and export.
+Both factories use the registry's claims, inventory, lazy creation, failure isolation and disposal.
</file context>
Suggested change
Each execution call receives explicit host ports for dispatch, progress and cancellation.
`runAdReplay` receives explicit dispatch and progress ports; suite execution receives explicit progress and cancellation ports, with dispatch supplied through its `runReplay` dependency.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Accepted: the sentence overstates AdReplayStepRuntime. runAdReplay receives dispatch and progress inputs, with no cancellation port; runReplayTestSuite receives its explicit progress/cancellation dependencies and dispatch through runReplay. The corrected proposal is preserved at #3871 (comment). ADR removal is queued with the stack update. Leaving this thread open until the source correction is published; no nonexistent cancellation API is being added.


## Proposed 0.22 amendment: native execution on the replay backend

This amendment is proposed for #3871 and awaits maintainer acceptance.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This adds an unaccepted proposal to an ADR marked accepted, contrary to the repository’s ADR format. Keep the amendment in #3871 until acceptance, or merge it only after updating the ADR to the accepted decision.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At docs/adr/0034-provider-plugin-registry.md, line 60:

<comment>This adds an unaccepted proposal to an ADR marked `accepted`, contrary to the repository’s ADR format. Keep the amendment in #3871 until acceptance, or merge it only after updating the ADR to the accepted decision.</comment>

<file context>
@@ -55,6 +55,33 @@ had to fake the store to load a plugin from its own dependencies (#3747).
 
+## Proposed 0.22 amendment: native execution on the replay backend
+
+This amendment is proposed for #3871 and awaits maintainer acceptance.
+The existing accepted provider contract and lifecycle rules retain their status.
+The draft API adds native execution to `kind: 'replay-backend'` with backend `ad`.
</file context>

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Accepted under docs/adr/FORMAT.md: an unaccepted proposal belongs in #3871. I preserved the full proposal, with corrected per-method port wording, at #3871 (comment). The queued patch removes only the proposed section from ADR0034; its accepted decisions and accepted status remain unchanged. This does not self-approve the native extension or clear its integration blocker. Leaving this thread open until that source deletion is published.

@blacksmith-sh

blacksmith-sh Bot commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

Found 1 test failure on Blacksmith runners:

Failure

Test View Logs
smoke:deep-link-navigation/smoke:deep-link-navigation View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@apex-sentinel-staging

apex-sentinel-staging Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

Apex Sentinel · ✅ Approve

Since the last review: 1 new · 0 still open · 0 no longer raised

1 finding(s): 1 minor.

🟡 Minor (1)

Where Finding
packages/plugins/src/registry.ts:209 🆕 resolveReplayExecution can pick another backend's slot and report invalid_plugin, not plugin-not-installed

1 finding below minor hidden (user_settings.min_severity).

Reviewed with this repository's own Sentinel configuration (d38e35e61193).

💬 Talking to Sentinel

In a finding's thread, reply with:

Command What it does
@apex-sentinel-staging note <context> Adds context; Sentinel re-checks the cited code
@apex-sentinel-staging pivot <why> Says the finding is wrong; Sentinel re-checks the cited code
@apex-sentinel-staging like Marks the finding as useful
@apex-sentinel-staging dislike <why> Marks it as not useful, so it is not raised again

On this conversation, comment @apex-sentinel-staging review to run a full review once again.

Earlier reviews

Review of 2c5f4d13 · 14.11.2026 10:04 UTC

Apex Sentinel · ✅ Approve

Since the last review: 0 new · 0 still open · 0 no longer raised

No open findings.

Reviewed with this repository's own Sentinel configuration (d38e35e61193).

Review of 3e5030d2 · 14.11.2026 08:21 UTC

Apex Sentinel · ✅ Approve

No open findings.

3 findings below minor hidden (user_settings.min_severity).

Reviewed with this repository's own Sentinel configuration (d38e35e61193).

@thymikee

Copy link
Copy Markdown
Member Author

Confirmed at 3e5030d2: smoke:deep-link-navigation fails when the stored-URL Maestro YAML is replayed without @agent-device/maestro installed (INVALID_ARGS, details.reason: plugin-not-installed). Blacksmith reports that same failure, and the selected Apple aggregate correctly fails with iOS.

The lower Maestro-layer Apple setup correction is already queued in the coordinated stack refresh, so it will reach this PR too. It installs the built plugin after the shared build under a persistent job-scoped plugin home. No additional engine change or CI retry is needed for this diagnostic. Fresh hosted iOS success remains required after publication.

@thymikee
thymikee force-pushed the t3/replay-execution-plugin branch from 2c5f4d1 to 8940075 Compare November 14, 2026 10:11

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 1 new finding on 8940075. The verdict and the full list are in the summary comment.

Comment on lines +209 to +227
resolveReplayExecution: async (packageName) => {
if (closed)
throw new AppError('COMMAND_FAILED', 'Plugin registry is closed', {
reason: 'plugin-registry-closed',
});
const slot = requireReplaySlot(resolved, NATIVE_REPLAY_BACKEND, packageName);
if (slot.failure !== undefined) throw slot.failure;
const execution = slot.replaySetup?.execution;
if (!execution) throw invalidPlugin('Native replay plugin must provide execution.create()');
return await (slot.executionLoading ??= Promise.resolve().then(async () => {
try {
return validateReplayExecution(await execution.create());
} catch (error) {
slot.failure = asPluginFailure(error, 'plugin_load_failed');
inventory.splice(0, inventory.length, ...resolved.inventory());
throw slot.failure;
}
}));
},

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🆕 🟡 Minor · resolveReplayExecution can pick another backend's slot and report invalid_plugin, not plugin-not-installed

A native request with ad not installed but packageName matching an installed slot (e.g. @agent-device/maestro) gets invalid_plugin instead of plugin-not-installed because requireReplaySlot falls back to a package-name match, and slot.replaySetup?.execution is undefined on non-ad backends. The consumer in replay-backend-registry.ts that maps plugin-not-installed into an install hint never triggers.

Suggested change
resolveReplayExecution: async (packageName) => {
if (closed)
throw new AppError('COMMAND_FAILED', 'Plugin registry is closed', {
reason: 'plugin-registry-closed',
});
const slot = requireReplaySlot(resolved, NATIVE_REPLAY_BACKEND, packageName);
if (slot.failure !== undefined) throw slot.failure;
const execution = slot.replaySetup?.execution;
if (!execution) throw invalidPlugin('Native replay plugin must provide execution.create()');
return await (slot.executionLoading ??= Promise.resolve().then(async () => {
try {
return validateReplayExecution(await execution.create());
} catch (error) {
slot.failure = asPluginFailure(error, 'plugin_load_failed');
inventory.splice(0, inventory.length, ...resolved.inventory());
throw slot.failure;
}
}));
},
resolveReplayExecution: async (packageName) => {
if (closed)
throw new AppError('COMMAND_FAILED', 'Plugin registry is closed', {
reason: 'plugin-registry-closed',
});
const slot = requireReplaySlot(resolved, NATIVE_REPLAY_BACKEND, packageName);
if (slot.failure !== undefined) throw slot.failure;
const execution = slot.backend === NATIVE_REPLAY_BACKEND ? slot.replaySetup?.execution : undefined;
if (!execution) throw invalidPlugin('Native replay plugin must provide execution.create()');
Details

Also raised here

Also raised at this location by persona maintainability: resolveReplayExecution's error is wrong when a slot is matched by package name — requireReplaySlot finds a slot by backend id first and then by packageName, so resolveReplayExecution('some-provider') can land on a slot whose backend is not ad — the NATIVE_REPLAY_BACKEND argument only names the preferred match, it is never asserted. That slot's setup holds replayBackend, the slot.replaySetup?.execution lookup misses, and the caller gets invalidPlugin('Native replay plugin must provide execution.create()') for a plugin that correctly supplied replayBackend.create(host) — while validateReplayEngineSetup simultaneously rejects any native setup that carries both fields. Either restrict the native lookup to slots whose backend is NATIVE_REPLAY_BACKEND (and report plugin-not-installed otherwise) or say in the error that the resolved slot is not the native backend.

Also raised at this location by persona senior-developer: requireReplaySlot package fallback returns a foreign-backend slot, wrong error reason — With only @​agent-device/maestro installed, registry.resolveReplayExecution('@​agent-device/maestro') finds the maestro slot through the candidate.package fallback in requireReplaySlot and throws invalid_plugin ('Native replay plugin must provide execution.create()') even though the native plugin is simply absent. The caller should get reason 'plugin-not-installed' so it can render the install hint the way replay-backend-registry.missingReplayBackend does; today the reason says the installed plugin is malformed. requireReplaySlot matches on packageName without checking slot.backend === id, so a provider-plugin slot or any other backend's slot can be returned for a native request.

persona: javascript, maintainability, senior-developer

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if it is wrong — I will re-check the cited code.

@thymikee thymikee Nov 14, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid at published 894007501dd05245beb766dc36c4415e0f5c065a: package-name fallback can select an active foreign backend when ad is absent. The shared-owner fix is local in 19855593dd57088c277897418d100813484dbc27: backend identity stays authoritative; package fallback is allowed only for an unidentified failed candidate, preserving typed broken-package failures. The suggested execution-only check would still produce invalid_plugin.

The isolated unchanged-test regression is now proven: published registry bytes failed with invalid_plugin where native plugin-not-installed was required; fixed bytes passed. All 15 focused registry cases, typecheck, lint, full format and layering passed. The controls confirm no foreign Maestro factory call, successful own-Maestro resolution, and retained plugin_load_failed for an unreadable selected replay package. These are unit-fixture results, not live execution. The fix is also in review-only c9fc5f653aaa730d6793351b63b40e1765c1f821, preserving that layer’s stale-selection guards; its independent source recheck completed with no findings. Publication remains pending, and this thread stays open until reviewed source is published and verified. The repeated notification requires no second fix or duplicate tests.

@thymikee

Copy link
Copy Markdown
Member Author

I reviewed 8940075 and found no code problems. The earlier review of 3e5030d (#4535 (comment)) found the code clean, and the code commits are unchanged since then. The new commit 8940075 only reverts the ADR changes, and ADR 0034 now matches base.

Hosted CI is unclear. Coverage failed and ten jobs were cancelled, which fits a superseding push. I could not read the Coverage failure, so I cannot say whether it relates to this PR. I also did not rerun tests locally. Please rerun CI on 8940075 and check the Coverage result.

On the open threads: the minor thread about the package-name fallback in the registry still holds (#4535 (comment)). It is not urgent, because resolveReplayExecution has no production caller yet. When a caller lands, take the package fallback only for slots whose backend is undefined or equals the id. The two ADR threads no longer apply, so please resolve them. The first is moot because 8940075 deletes the section with the cancellation-port sentence (#4535 (comment)). The second is fixed because the proposal is gone from the accepted ADR (#4535 (comment)).

Before merge, CI must pass on 8940075 and the two moot threads must be resolved. The maintainer's native-contract acceptance, which the PR names as a merge condition, is still needed. I could not confirm the preserved proposal text in #3871 beyond your reply.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Nov 14, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant