Skip to content

feat: Record & Replay Controller#1146

Merged
MistEO merged 38 commits intomainfrom
feat/proxy_controller
Mar 27, 2026
Merged

feat: Record & Replay Controller#1146
MistEO merged 38 commits intomainfrom
feat/proxy_controller

Conversation

@MistEO
Copy link
Copy Markdown
Member

@MistEO MistEO commented Feb 24, 2026

fix #1133

Summary by Sourcery

引入专用的记录和回放控制器以及简化版的调试图像控制器,实现跨语言的控制器操作可重现(确定性)记录/回放,并将其集成到核心框架、绑定层和测试中。

新功能:

  • 添加 ReplayControllerRecordController 的实现,并在 C、C++、Python 和 NodeJS 绑定中暴露创建 API,用于记录和回放控制器操作。
  • 添加一个调试控制器,从目录中提供轮播截图,同时将所有输入操作视为成功的空操作(no-op)。

增强:

  • 将控制器能力重构为类似 mixin 的接口(scrollrelative_moveshell),并更新 ControllerAgent 和控制单元,使其基于支持的能力而非具体平台类型进行分发。
  • 扩展 Python 和 NodeJS 中的自定义控制器回调和绑定,以支持 relative_moveshell 操作,并在记录格式和解析器中加入更多操作类型和元数据,包括滚动和 shell 输出。
  • 调整调试和回放控制器的 controller info 语义,使其报告简化的、模式特定的字段和类型。

构建:

  • 为回放和记录控制单元添加新的共享库和构建选项,并将它们接入 LibraryHolderMaaFrameworkdlopen 测试。
  • 在用于流水线测试的 CI 构建配置中启用回放控制器。

文档:

  • 更新集成接口文档,描述新的调试、记录和回放控制器 API 及其 info 字段。

测试:

  • 添加并接入一个 pipeline_smoking 测试,通过 ReplayController 使用录制的输入数据运行流水线,以实现确定性的验证。
  • 更新 Python、NodeJS、agent、pipeline 和 dlopen 测试,以覆盖新的记录/回放和调试控制器行为,以及负 ROI/target 处理。
  • 将流水线节点测试重构为可复用的辅助工具,并在 Python 测试中复用 PipelineSmoking 测试资源。
Original summary in English

Summary by Sourcery

Introduce dedicated record and replay controllers and a simplified debug image controller, enabling deterministic recording/replay of controller operations across languages and integrating them into the core framework, bindings, and tests.

New Features:

  • Add ReplayController and RecordController implementations and expose creation APIs in C, C++, Python, and NodeJS bindings for recording and replaying controller operations.
  • Add a debug controller that serves carousel screenshots from a directory while treating all input operations as successful no-ops.

Enhancements:

  • Refactor controller capabilities into mixin-style interfaces (scroll, relative_move, shell) and update ControllerAgent and control units to dispatch based on supported capabilities instead of concrete platform types.
  • Extend custom controller callbacks and bindings in Python and NodeJS to support relative_move and shell operations, and enrich recording formats and parsers with additional operation types and metadata including scroll and shell output.
  • Adjust controller info semantics for debug and replay controllers to report simplified, mode-specific fields and types.

Build:

  • Add new shared libraries and build options for replay and record control units and wire them into LibraryHolder, MaaFramework, and dlopen testing.
  • Enable replay controller in CI build configurations used for pipeline testing.

Documentation:

  • Update integrated interface documentation to describe the new debug, record, and replay controller APIs and their info fields.

Tests:

  • Add and wire a pipeline_smoking test that runs pipelines against recorded input data via ReplayController for deterministic validation.
  • Update Python, NodeJS, agent, pipeline, and dlopen tests to cover the new record/replay and debug controller behaviors and negative ROI/target handling.
  • Refactor pipeline node tests into reusable helpers and reuse PipelineSmoking test resources in Python tests.

Summary by Sourcery

引入专用的录制与回放控制器,以及一个简化的调试图像控制器,将它们接入核心框架和各语言绑定,并用来实现基于文件驱动的确定性流水线测试。

New Features:

  • 在核心框架以及 C/C++/Python/NodeJS 绑定中新增 ReplayControllerRecordController 类型及创建 API,用于录制和回放控制器操作。
  • 新增一个调试控制器,从目录中轮播截图,同时将所有输入操作视为成功的空操作(no-op)。

Enhancements:

  • 将控制器能力重构为混入(mixin)风格的能力接口(scrollrelative_moveshell),并更新 ControllerAgent 和控制单元,使其基于支持的能力而非具体平台类型来分发行为。
  • 扩展 Python 和 NodeJS 中的自定义控制器回调和绑定以支持 relative_moveshell 操作,并在录制格式、解析器和控制器信息语义中加入更多操作类型和元数据。

Build:

  • 为调试、回放和录制控制单元新增共享库、CMake 选项和链接配置,并将它们集成到 LibraryHolderMaaFramework 以及 dlopen 测试中。
  • 在用于流水线测试的 CI 编译预设中启用回放控制器。

Documentation:

  • 更新集成接口文档,描述新的调试、录制和回放控制器 API 及其信息字段。

Tests:

  • 新增 pipeline_smoking 测试,通过 ReplayController 使用录制的输入来驱动流水线,实现确定性验证,并将流水线节点测试重构为可复用的助手方法,以便在 Python 测试中复用。
  • 更新 Python、NodeJS、agent、pipeline 和 dlopen 测试,以覆盖新的录制/回放和调试控制器,以及修改后的负 ROI/目标行为。
Original summary in English

Summary by Sourcery

Introduce dedicated record and replay controllers and a simplified debug image controller, wire them into the core framework and language bindings, and use them to enable deterministic, file-driven pipeline testing.

New Features:

  • Add ReplayController and RecordController types and creation APIs in the core framework and C/C++/Python/NodeJS bindings for recording and replaying controller operations.
  • Add a debug controller that cycles screenshots from a directory while treating all input operations as successful no-ops.

Enhancements:

  • Refactor controller capabilities into mixin-style capability interfaces (scroll, relative_move, shell) and update ControllerAgent and control units to dispatch behavior based on supported capabilities rather than concrete platform types.
  • Extend custom controller callbacks and bindings in Python and NodeJS to support relative_move and shell operations, and enrich recording formats, parser, and controller info semantics with additional operation types and metadata.

Build:

  • Add new shared libraries, CMake options, and linkage for debug, replay, and record control units, and integrate them with LibraryHolder, MaaFramework, and dlopen testing.
  • Enable the replay controller in CI build presets used for pipeline testing.

Documentation:

  • Update integrated interface documentation to describe the new debug, record, and replay controller APIs and their info fields.

Tests:

  • Add a pipeline_smoking test that drives pipelines from recorded input via ReplayController for deterministic validation, and refactor pipeline node tests into reusable helpers reused from Python tests.
  • Update Python, NodeJS, agent, pipeline, and dlopen tests to exercise the new record/replay and debug controllers and revised negative ROI/target behavior.

新功能:

  • 添加 ReplayControllerRecordController 抽象,并通过 C、C++、Python 和 NodeJS 绑定对外暴露,用于录制和回放控制器操作。
  • 添加一个调试控制器,从目录中循环播放截图,并将所有输入操作视为成功的空操作(no-op)。

增强内容:

  • 将控制器能力重构为混入(mixin)风格的接口(scrollrelative_moveshell),并更新 ControllerAgent 和控制单元,以基于支持的能力而不是具体平台类型来进行分发。
  • 扩展 Python 和 NodeJS 中的自定义控制器回调和绑定,以支持 relative_moveshell 操作,并在录制格式、解析器和控制器信息中加入更多操作类型和元数据。

构建:

  • 为回放和录制控制单元引入单独的共享库和构建选项,并将它们接入 LibraryHolderMaaFrameworkdlopen 测试以及包含 CI 预设在内的 CMake 选项中。

文档:

  • 更新集成接口文档,描述新的调试、录制和回放控制器 API、其参数以及修订后的控制器信息字段。

测试:

  • 添加 pipeline_smoking 测试及相关辅助工具,通过 ReplayController 使用录制的输入来驱动流水线以实现确定性验证,并更新 Python、NodeJS、agent、pipeline 和 dlopen 测试,以支持新的控制器和负 ROI/目标行为。
Original summary in English

Summary by Sourcery

引入专用的录制与回放控制器,以及一个简化的调试图像控制器,将它们接入核心框架和各语言绑定,并用来实现基于文件驱动的确定性流水线测试。

New Features:

  • 在核心框架以及 C/C++/Python/NodeJS 绑定中新增 ReplayControllerRecordController 类型及创建 API,用于录制和回放控制器操作。
  • 新增一个调试控制器,从目录中轮播截图,同时将所有输入操作视为成功的空操作(no-op)。

Enhancements:

  • 将控制器能力重构为混入(mixin)风格的能力接口(scrollrelative_moveshell),并更新 ControllerAgent 和控制单元,使其基于支持的能力而非具体平台类型来分发行为。
  • 扩展 Python 和 NodeJS 中的自定义控制器回调和绑定以支持 relative_moveshell 操作,并在录制格式、解析器和控制器信息语义中加入更多操作类型和元数据。

Build:

  • 为调试、回放和录制控制单元新增共享库、CMake 选项和链接配置,并将它们集成到 LibraryHolderMaaFramework 以及 dlopen 测试中。
  • 在用于流水线测试的 CI 编译预设中启用回放控制器。

Documentation:

  • 更新集成接口文档,描述新的调试、录制和回放控制器 API 及其信息字段。

Tests:

  • 新增 pipeline_smoking 测试,通过 ReplayController 使用录制的输入来驱动流水线,实现确定性验证,并将流水线节点测试重构为可复用的助手方法,以便在 Python 测试中复用。
  • 更新 Python、NodeJS、agent、pipeline 和 dlopen 测试,以覆盖新的录制/回放和调试控制器,以及修改后的负 ROI/目标行为。
Original summary in English

Summary by Sourcery

Introduce dedicated record and replay controllers and a simplified debug image controller, wire them into the core framework and language bindings, and use them to enable deterministic, file-driven pipeline testing.

New Features:

  • Add ReplayController and RecordController types and creation APIs in the core framework and C/C++/Python/NodeJS bindings for recording and replaying controller operations.
  • Add a debug controller that cycles screenshots from a directory while treating all input operations as successful no-ops.

Enhancements:

  • Refactor controller capabilities into mixin-style capability interfaces (scroll, relative_move, shell) and update ControllerAgent and control units to dispatch behavior based on supported capabilities rather than concrete platform types.
  • Extend custom controller callbacks and bindings in Python and NodeJS to support relative_move and shell operations, and enrich recording formats, parser, and controller info semantics with additional operation types and metadata.

Build:

  • Add new shared libraries, CMake options, and linkage for debug, replay, and record control units, and integrate them with LibraryHolder, MaaFramework, and dlopen testing.
  • Enable the replay controller in CI build presets used for pipeline testing.

Documentation:

  • Update integrated interface documentation to describe the new debug, record, and replay controller APIs and their info fields.

Tests:

  • Add a pipeline_smoking test that drives pipelines from recorded input via ReplayController for deterministic validation, and refactor pipeline node tests into reusable helpers reused from Python tests.
  • Update Python, NodeJS, agent, pipeline, and dlopen tests to exercise the new record/replay and debug controllers and revised negative ROI/target behavior.

@neko-para
Copy link
Copy Markdown
Contributor

看了下得给js binding加下ref,明天我做下

@neko-para
Copy link
Copy Markdown
Contributor

草, 我看了下现在的binding里面tasker也没持有resource和controller, 算了(

@MistEO MistEO force-pushed the feat/proxy_controller branch from 1db6bda to 0766e1d Compare March 2, 2026 08:38
MistEO added 8 commits March 20, 2026 03:01
Resolve merge conflict in ReplayRecording.cpp (remove scroll from base
ControlUnitAPI impl). Add macOS support to handle_scroll and
handle_relative_move in ControllerAgent.

Made-with: Cursor
…xy support

Add default virtual scroll/relative_move to ControlUnitAPI so
DbgControlUnit and ProxyControlUnit can support all control methods.
Simplify ControllerAgent dispatch to call via base pointer directly.

Made-with: Cursor
DbgController now exclusively handles ReplayRecording. Removed the
MaaDbgControllerType parameter from all APIs since type selection
is no longer needed.

Made-with: Cursor
Remove unused write_path and config parameters, rename read_path to
dump_dir to match MaaRecordControllerCreate, and resolve the path
mismatch by having ReplayController read recording.jsonl from the
directory internally.

Made-with: Cursor
@MistEO MistEO marked this pull request as ready for review March 26, 2026 14:17
Copilot AI review requested due to automatic review settings March 26, 2026 14:17
Copy link
Copy Markdown
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - 我发现了 4 个问题

给 AI 代理的提示
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="source/MaaRecordControlUnit/RecordController.cpp" line_range="202-204" />
<code_context>
+    return inner_->get_info();
+}
+
+bool RecordController::shell(const std::string& cmd, std::string& output, std::chrono::milliseconds timeout)
+{
+    if (auto p = std::dynamic_pointer_cast<ShellableUnit>(inner_)) return p->shell(cmd, output, timeout);
+    LogError << "Inner controller does not support shell";
+    return false;
</code_context>
<issue_to_address>
**suggestion (bug_risk):** RecordController.shell 会转发调用,但不会记录 shell 操作,这可能会让期望完全可重放性的用户感到意外。

由于 RecordController 的文档宣称其会“记录所有操作”,在录制中省略 shell 会导致重放结果变得不确定,并可能违背使用方的预期。请考虑增加一种 shell 的记录类型(command、timeout、output),以便 ReplayController 可以重放/校验这些调用;或者,如果你是有意排除 shell,请更新文档,明确说明 shell 会被转发但不会记录。

建议实现如下:

```cpp
bool RecordController::shell(const std::string& cmd, std::string& output, std::chrono::milliseconds timeout)
{
    RecordShell record { cmd, timeout.count(), std::string {} };

    auto action = [&]() -> bool {
        if (auto p = std::dynamic_pointer_cast<ShellableUnit>(inner_)) {
            const bool ok = p->shell(cmd, output, timeout);
            if (ok) {
                // Capture the actual output for deterministic replay/validation
                record.output = output;
            }
            return ok;
        }
        LogError << "Inner controller does not support shell";
        return false;
    };

    return forward_and_record(RecordType::shell, record, action);
}

```

为了完整支持 shell 的记录和重放,你还需要:

1. **定义记录负载结构**(放在与其他记录结构体相同的记录模型头/源文件中),例如:
   ```cpp
   struct RecordShell {
       std::string command;
       std::int64_t timeout_ms;
       std::string output;
   };
   ```
2. **扩展记录类型枚举**(在定义 `RecordType::scroll``RecordType::click` 等的地方)增加:
   ```cpp
   shell,
   ```
3. **在录制格式(JSON/CBOR 等)中接好 `RecordShell` 的序列化/反序列化**,放在处理其他记录负载的同一处。
4. **更新 ReplayController**(或等价的重放单元)以:
   - 识别 `RecordType::shell`- 反解 `RecordShell`- 并且要么:
     - 重新执行 shell 命令并将实时 `output``record.output` 比较进行校验,或
     - 跳过实际执行而直接返回记录下来的 `output`,用于纯重放。
5. **调整 RecordController 的文档**,明确说明 `shell` 会被记录并重放/校验;或者记录下任何有意偏离严格确定性的行为(例如:“shell 会被记录但只用于校验,不会被重新执行”)。
6. 如果 `timeout` 在其他单元中以不同类型存储(例如 `std::chrono::milliseconds` 或其他标量),请在 `RecordShell` 中对其类型和字段命名进行统一,以符合你现有的约定。
</issue_to_address>

### Comment 2
<location path="source/MaaDbgControlUnit/DbgController.cpp" line_range="82-84" />
<code_context>
+        return MAA_NS::path_to_utf8_string(a) < MAA_NS::path_to_utf8_string(b);
+    });
+
+    for (const auto& p : paths) {
+        cv::Mat img = MAA_NS::imread(p);
+        if (!img.empty()) {
+            images_.push_back(std::move(img));
+        }
</code_context>
<issue_to_address>
**suggestion:** DbgController 会静默跳过无法读取的图像;如果能暴露这些失败信息会有助于排查路径配置错误。

在 `connect()` 中,当前对加载失败的图像只是忽略(只保存非空的 `cv::Mat`),因此当目录中部分文件无效时,很难排查问题。建议在每次 `imread` 失败时记录一条调试日志,例如:

```cpp
cv::Mat img = MAA_NS::imread(p);
if (img.empty()) {
    LogDebug << "Failed to load debug image" << VAR(p);
    continue;
}
images_.push_back(std::move(img));
```

这样可以在保持现有行为的同时,提高对损坏或不支持的图像的可调试性。

建议实现如下:

```cpp
    for (const auto& p : paths) {
        cv::Mat img = MAA_NS::imread(p);
        if (img.empty()) {
            LogDebug << "Failed to load debug image" << VAR(p);
            continue;
        }
        images_.push_back(std::move(img));

```

1. 上述修改假设 `LogDebug` 和 `VAR` 已在此翻译单元中可用(通常通过已有的日志头文件引入)。如果还不可用,你需要在 `DbgController.cpp` 顶部包含相应的日志头文件。
2. 如果这个循环在你的代码中缩进或格式略有不同,请相应调整 `SEARCH` 块,使其与现有代码精确匹配。
</issue_to_address>

### Comment 3
<location path="test/nodejs/binding.ts" line_range="102" />
<code_context>
     })
     console.log('rsource', resource)

-    const dbg_controller = new maa.DbgController(
-        '../../install/test/PipelineSmoking/Screenshot',
-        '../../install/test/user',
-        maa.DbgControllerType.CarouselImage,
-        '{}'
+    const replay_controller = new maa.DbgController(
+        '../../install/test/PipelineSmoking/Screenshot'
     )
</code_context>
<issue_to_address>
**suggestion (testing):** NodeJS 测试没有覆盖新的 ReplayController/RecordController 或自定义的 relative_move/shell 回调

当前 NodeJS 测试仍然只使用 `DbgController`/`replay_controller`,从未实际调用 `ReplayController`、`RecordController` 或新的 `CustomController` 回调(`relative_move`、`shell`),因此这些新绑定实际上没有被测试。

请补充以下测试:
- 用 `new maa.RecordController(innerController, recordingPath)` 包装一个简单的控制器,执行几次操作(包括 `post_scroll` / `post_relative_move`),并断言录制文件和截图目录已经创建。
- 使用 `new maa.ReplayController(recordingPath)` 进行连接并运行一个小任务,参考 Python 中的重放测试。
- 定义一个带有 `relative_move` 和 `shell` 处理函数的 `CustomControllerActor`,验证 `post_relative_move` / `post_shell` 会调用到 JS 回调并正确传递返回值,包括 shell 返回 `null`/失败的情况。

这将有助于保证 NodeJS 绑定与核心控制器行为保持一致。

建议实现如下:

```typescript
    })
    console.log('rsource', resource)

    //
    // RecordController 测试:包装一个简单控制器,执行若干操作,
    // 并断言录制文件和截图输出已创建。
    //
    const screenshotDir = '../../install/test/PipelineSmoking/Screenshot'
    const recordingPath = path.join(__dirname, 'node_recording_test.mr')
    const recordingScreenshotDir = path.join(__dirname, 'node_recording_screenshots')

    // 清理之前的测试产物
    if (fs.existsSync(recordingPath)) {
        fs.unlinkSync(recordingPath)
    }
    if (fs.existsSync(recordingScreenshotDir)) {
        fs.rmSync(recordingScreenshotDir, { recursive: true, force: true })
    }

    const dbg_controller = new maa.DbgController(screenshotDir)
    dbg_controller.add_sink((res, msg) => {
        console.log('[DbgController]', msg)
    })

    const record_controller = new maa.RecordController(
        dbg_controller,
        recordingPath,
        recordingScreenshotDir
    )
    record_controller.add_sink((res, msg) => {
        console.log('[RecordController]', msg)
    })

    console.log('record_controller', record_controller)
    await record_controller.post_connection().wait()

    // 覆盖控制器操作,包括 scroll 和 relative_move
    await record_controller.post_screenshot().wait()
    await record_controller.post_scroll(0, 100).wait()
    await record_controller.post_relative_move(10, -5).wait()

    // 确认录制文件和截图目录已创建
    if (!fs.existsSync(recordingPath)) {
        throw new Error(`RecordController did not create recording file at ${recordingPath}`)
    }
    if (!fs.existsSync(recordingScreenshotDir)) {
        throw new Error(
            `RecordController did not create screenshot directory at ${recordingScreenshotDir}`
        )
    }

    //
    // ReplayController 测试:使用录制好的会话运行一个小任务。
    //
    const replay_controller = new maa.ReplayController(recordingPath)
    replay_controller.add_sink((res, msg) => {
        console.log('[ReplayController]', msg)
    })
    console.log('replay_controller', replay_controller)
    await replay_controller.post_connection().wait()

    const tasker = new maa.Tasker()

    //
    // CustomControllerActor 测试:验证 relative_move 和 shell 回调。
    //
    const relativeMoves: Array<{ x: number; y: number }> = []
    const shellCommands: string[] = []
    const shellResults: Array<string | null> = []

    const customActor = new maa.CustomControllerActor({
        // 由 post_relative_move 调用
        async relative_move(x: number, y: number): Promise<maa.ErrorCode> {
            console.log('[CustomControllerActor.relative_move]', x, y)
            relativeMoves.push({ x, y })
            return maa.ErrorCode.Success
        },
        // 由 post_shell 调用
        async shell(cmd: string): Promise<string | null> {
            console.log('[CustomControllerActor.shell]', cmd)
            shellCommands.push(cmd)
            if (cmd === 'fail') {
                shellResults.push(null)
                return null
            }
            const result = `executed:${cmd}`
            shellResults.push(result)
            return result
        },
    })

    const custom_controller = new maa.CustomController(customActor)
    custom_controller.add_sink((res, msg) => {
        console.log('[CustomController]', msg)
    })

    await custom_controller.post_connection().wait()

    // 通过自定义控制器测试 relative_move
    await custom_controller.post_relative_move(5, 7).wait()
    if (
        relativeMoves.length !== 1 ||
        relativeMoves[0].x !== 5 ||
        relativeMoves[0].y !== 7
    ) {
        throw new Error(
            `CustomControllerActor.relative_move was not invoked correctly: ${JSON.stringify(
                relativeMoves
            )}`
        )
    }

    // 测试 shell 成功场景
    const okShell = await custom_controller.post_shell('echo ok').wait()
    if (okShell !== 'executed:echo ok') {
        throw new Error(
            `CustomControllerActor.shell success case did not propagate result, got: ${okShell}`
        )
    }

    // 测试 shell 失败/返回 null 的场景
    const failShell = await custom_controller.post_shell('fail').wait()
    if (failShell !== null) {
        throw new Error(
            `CustomControllerActor.shell failure case did not propagate null, got: ${failShell}`
        )
    }

    if (
        shellCommands.length !== 2 ||
        shellCommands[0] !== 'echo ok' ||
        shellCommands[1] !== 'fail'
    ) {
        throw new Error(
            `CustomControllerActor.shell was not invoked with expected commands: ${JSON.stringify(
                shellCommands
            )}`
        )
    }

```

1.`test/nodejs/binding.ts` 顶部,请确保已有如下导入(或在你的代码库中等价的导入):
   - `import * as fs from 'fs'`
   - `import * as path from 'path'`
2. 如果你的 Node 绑定暴露的类名或工厂函数不同,请相应调整:
   - `maa.RecordController``maa.ReplayController``maa.CustomControllerActor``maa.CustomController` 为实际导出的名称;
   - `post_screenshot``post_scroll``post_relative_move``post_shell` 等方法名为真实的异步 API 名称,并根据你的 Promise 封装是否不同决定是否保留 `.wait()`3. 如果测试框架期望通过断言库(如 Mocha/Chai 或 Jest)而不是 `throw new Error` 来进行断言,请将 `throw new Error(...)` 替换为对应的 `expect`/`assert` 调用。
4. 确认 `maa.ErrorCode.Success` 是否为正确的成功枚举值;如果不是,请替换为实际的成功值。
</issue_to_address>

### Comment 4
<location path="test/pipeline/module/PipelineSmoking.cpp" line_range="7" />
<code_context>

 bool pipeline_smoking(const std::filesystem::path& testset_dir)
 {
-    auto testing_path = testset_dir / "PipelineSmoking" / "MaaRecording.txt";
-    auto result_path = testset_dir / "debug";
+    auto recording_path = testset_dir / "PipelineSmoking" / "MaaRecording.jsonl";

-    auto controller_handle =
</code_context>
<issue_to_address>
**suggestion (testing):** 当前 C++ 的 pipeline smoking 测试已经改为使用 MaaReplayControllerCreate,但没有校验重放元数据或失败场景

该测试现在使用 JSONL 录制文件调用 `MaaReplayControllerCreate`,但只断言整体是否成功。增加如下检查会更有价值:
- 通过 `MaaControllerGetInfo`(如果可用)断言控制器报告的 `type = "replay"`,且 `record_count` 为正数;
- 添加一个负向测试:使用缺失或非法的 JSONL 文件调用 `MaaReplayControllerCreate`,并验证它返回 `nullptr`。

这样既能覆盖重放相关的特定行为,也能测试新的 C API 的基本错误处理能力。

建议实现如下:

```cpp
bool pipeline_smoking(const std::filesystem::path& testset_dir)
{
    auto recording_path = testset_dir / "PipelineSmoking" / "MaaRecording.jsonl";

    // 正向路径:从合法录制文件创建重放控制器
    auto controller_handle = MaaReplayControllerCreate(recording_path.string().c_str());
    if (!controller_handle)
    {
        // 对于合法录制文件,应当创建成功
        return false;
    }

    // 校验控制器元数据(重放特定行为)
    MaaControllerInfo info{};
    if (!MaaControllerGetInfo(controller_handle, &info))
    {
        return false;
    }

    // type 应为 "replay"
    if (std::string_view(info.type) != "replay")
    {
        return false;
    }

    // record_count 应为正数
    if (info.record_count <= 0)
    {
        return false;
    }

    // 负向路径:非法/缺失的 JSONL 文件应当失败
    auto invalid_recording_path = testset_dir / "PipelineSmoking" / "NonExistentRecording.jsonl";
    auto invalid_controller_handle = MaaReplayControllerCreate(invalid_recording_path.string().c_str());
    if (invalid_controller_handle != nullptr)
    {
        // 对非法输入,MaaReplayControllerCreate 应返回 nullptr
        return false;
    }

    auto ctrl_id = MaaControllerPostConnection(controller_handle);

```

根据你的代码库,可能还需要:

1. 确认 `MaaControllerInfo` 和 `MaaControllerGetInfo` 已通过包含正确的头文件(例如 `#include "MaaController.h"` 或类似文件)在 `PipelineSmoking.cpp` 顶部声明。
2. 如果 `MaaControllerInfo` 的结构或字段名不同,请调整相应代码(如 `info.type` 可能是枚举或字段名不同;`info.record_count` 可能有不同命名)。
3. 如果 `info.type` 不是 C 字符串,请相应更改比较方式(例如使用枚举值比较而不是 `std::string_view`)。
4. 如果此测试使用不同的错误报告习惯(如维护一个 `bool ok` 或使用测试宏),请将这些 `return false` 调整为符合现有约定的写法。
</issue_to_address>

Sourcery 对开源项目免费——如果你觉得我们的评审有帮助,欢迎分享 ✨
帮我变得更有用!请在每条评论上点 👍 或 👎,我会根据你的反馈改进今后的评审。
Original comment in English

Hey - I've found 4 issues

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="source/MaaRecordControlUnit/RecordController.cpp" line_range="202-204" />
<code_context>
+    return inner_->get_info();
+}
+
+bool RecordController::shell(const std::string& cmd, std::string& output, std::chrono::milliseconds timeout)
+{
+    if (auto p = std::dynamic_pointer_cast<ShellableUnit>(inner_)) return p->shell(cmd, output, timeout);
+    LogError << "Inner controller does not support shell";
+    return false;
</code_context>
<issue_to_address>
**suggestion (bug_risk):** RecordController.shell forwards calls but does not record shell operations, which may surprise users expecting full replayability.

Since RecordController is documented as "records all operations", omitting shell from the recording makes replay non-deterministic and may violate consumer expectations. Please consider adding a shell record type (command, timeout, output) so ReplayController can either replay/validate these calls, or, if shell is intentionally excluded, update the documentation to clarify that shell is forwarded but not recorded.

Suggested implementation:

```cpp
bool RecordController::shell(const std::string& cmd, std::string& output, std::chrono::milliseconds timeout)
{
    RecordShell record { cmd, timeout.count(), std::string {} };

    auto action = [&]() -> bool {
        if (auto p = std::dynamic_pointer_cast<ShellableUnit>(inner_)) {
            const bool ok = p->shell(cmd, output, timeout);
            if (ok) {
                // Capture the actual output for deterministic replay/validation
                record.output = output;
            }
            return ok;
        }
        LogError << "Inner controller does not support shell";
        return false;
    };

    return forward_and_record(RecordType::shell, record, action);
}

```

To fully support shell recording and replay, you will also need to:

1. **Define the record payload** (in the appropriate record model header/source, where other record structs live), e.g.:
   ```cpp
   struct RecordShell {
       std::string command;
       std::int64_t timeout_ms;
       std::string output;
   };
   ```
2. **Extend the record type enum** (where `RecordType::scroll`, `RecordType::click`, etc., are defined) with:
   ```cpp
   shell,
   ```
3. **Wire up serialization/deserialization** of `RecordShell` in the recording format (JSON/CBOR/etc.) in the same place other record payloads are handled.
4. **Update ReplayController** (or equivalent replay unit) to:
   - Recognize `RecordType::shell`.
   - Decode `RecordShell`.
   - Either:
     - Re-execute the shell command and compare the live `output` with `record.output` for validation, or
     - Bypass execution and directly return the recorded `output` for pure replay.
5. **Adjust documentation** for `RecordController` to explicitly state that `shell` is recorded and replayed/validated, or document any intentional deviation from strict determinism (e.g., “shell is recorded but only validated, not re-executed”).
6. If `timeout` is stored elsewhere in a different unit (e.g. as `std::chrono::milliseconds` or another scalar), align the type and field naming in `RecordShell` to match your existing conventions.
</issue_to_address>

### Comment 2
<location path="source/MaaDbgControlUnit/DbgController.cpp" line_range="82-84" />
<code_context>
+        return MAA_NS::path_to_utf8_string(a) < MAA_NS::path_to_utf8_string(b);
+    });
+
+    for (const auto& p : paths) {
+        cv::Mat img = MAA_NS::imread(p);
+        if (!img.empty()) {
+            images_.push_back(std::move(img));
+        }
</code_context>
<issue_to_address>
**suggestion:** DbgController silently skips unreadable images; surfacing failures could aid debugging of misconfigured paths.

In `connect()`, failed image loads are currently ignored (only non-empty `cv::Mat` are stored), so partially invalid directories are hard to diagnose. Consider logging a debug message on `imread` failure for each path, e.g.:

```cpp
cv::Mat img = MAA_NS::imread(p);
if (img.empty()) {
    LogDebug << "Failed to load debug image" << VAR(p);
    continue;
}
images_.push_back(std::move(img));
```

This preserves behavior while improving debuggability for corrupted or unsupported images.

Suggested implementation:

```cpp
    for (const auto& p : paths) {
        cv::Mat img = MAA_NS::imread(p);
        if (img.empty()) {
            LogDebug << "Failed to load debug image" << VAR(p);
            continue;
        }
        images_.push_back(std::move(img));

```

1. This change assumes that `LogDebug` and `VAR` are already available in this translation unit (likely via an existing logging header). If they are not, you will need to include the appropriate logging header at the top of `DbgController.cpp`.
2. If this loop appears with different indentation or slight formatting differences, adjust the `SEARCH` block accordingly so it matches the existing code exactly.
</issue_to_address>

### Comment 3
<location path="test/nodejs/binding.ts" line_range="102" />
<code_context>
     })
     console.log('rsource', resource)

-    const dbg_controller = new maa.DbgController(
-        '../../install/test/PipelineSmoking/Screenshot',
-        '../../install/test/user',
-        maa.DbgControllerType.CarouselImage,
-        '{}'
+    const replay_controller = new maa.DbgController(
+        '../../install/test/PipelineSmoking/Screenshot'
     )
</code_context>
<issue_to_address>
**suggestion (testing):** NodeJS tests do not cover the new ReplayController/RecordController or custom relative_move/shell callbacks

The NodeJS tests still only use `DbgController`/`replay_controller` and never exercise `ReplayController`, `RecordController`, or the new `CustomController` callbacks (`relative_move`, `shell`), so the new bindings are effectively untested.

Please add tests that:
- Wrap a simple controller in `new maa.RecordController(innerController, recordingPath)`, perform a few operations (including `post_scroll` / `post_relative_move`), and assert the recording file and screenshot directory are created.
- Use `new maa.ReplayController(recordingPath)` to connect and run a small task, mirroring the Python replay tests.
- Define a `CustomControllerActor` with `relative_move` and `shell` handlers, and verify `post_relative_move` / `post_shell` invoke the JS callbacks and propagate return values, including the `null`/failure shell case.

This will help keep the NodeJS bindings aligned with the core controller behavior.

Suggested implementation:

```typescript
    })
    console.log('rsource', resource)

    //
    // RecordController test: wrap a simple controller, perform operations,
    // and assert that the recording and screenshot outputs are created.
    //
    const screenshotDir = '../../install/test/PipelineSmoking/Screenshot'
    const recordingPath = path.join(__dirname, 'node_recording_test.mr')
    const recordingScreenshotDir = path.join(__dirname, 'node_recording_screenshots')

    // Clean up any previous artifacts
    if (fs.existsSync(recordingPath)) {
        fs.unlinkSync(recordingPath)
    }
    if (fs.existsSync(recordingScreenshotDir)) {
        fs.rmSync(recordingScreenshotDir, { recursive: true, force: true })
    }

    const dbg_controller = new maa.DbgController(screenshotDir)
    dbg_controller.add_sink((res, msg) => {
        console.log('[DbgController]', msg)
    })

    const record_controller = new maa.RecordController(
        dbg_controller,
        recordingPath,
        recordingScreenshotDir
    )
    record_controller.add_sink((res, msg) => {
        console.log('[RecordController]', msg)
    })

    console.log('record_controller', record_controller)
    await record_controller.post_connection().wait()

    // Exercise controller operations, including scroll and relative_move
    await record_controller.post_screenshot().wait()
    await record_controller.post_scroll(0, 100).wait()
    await record_controller.post_relative_move(10, -5).wait()

    // Ensure recording file and screenshot directory were created
    if (!fs.existsSync(recordingPath)) {
        throw new Error(`RecordController did not create recording file at ${recordingPath}`)
    }
    if (!fs.existsSync(recordingScreenshotDir)) {
        throw new Error(
            `RecordController did not create screenshot directory at ${recordingScreenshotDir}`
        )
    }

    //
    // ReplayController test: use the recorded session to run a small task.
    //
    const replay_controller = new maa.ReplayController(recordingPath)
    replay_controller.add_sink((res, msg) => {
        console.log('[ReplayController]', msg)
    })
    console.log('replay_controller', replay_controller)
    await replay_controller.post_connection().wait()

    const tasker = new maa.Tasker()

    //
    // CustomControllerActor test: verify relative_move and shell callbacks.
    //
    const relativeMoves: Array<{ x: number; y: number }> = []
    const shellCommands: string[] = []
    const shellResults: Array<string | null> = []

    const customActor = new maa.CustomControllerActor({
        // Called by post_relative_move
        async relative_move(x: number, y: number): Promise<maa.ErrorCode> {
            console.log('[CustomControllerActor.relative_move]', x, y)
            relativeMoves.push({ x, y })
            return maa.ErrorCode.Success
        },
        // Called by post_shell
        async shell(cmd: string): Promise<string | null> {
            console.log('[CustomControllerActor.shell]', cmd)
            shellCommands.push(cmd)
            if (cmd === 'fail') {
                shellResults.push(null)
                return null
            }
            const result = `executed:${cmd}`
            shellResults.push(result)
            return result
        },
    })

    const custom_controller = new maa.CustomController(customActor)
    custom_controller.add_sink((res, msg) => {
        console.log('[CustomController]', msg)
    })

    await custom_controller.post_connection().wait()

    // Exercise relative_move through the custom controller
    await custom_controller.post_relative_move(5, 7).wait()
    if (
        relativeMoves.length !== 1 ||
        relativeMoves[0].x !== 5 ||
        relativeMoves[0].y !== 7
    ) {
        throw new Error(
            `CustomControllerActor.relative_move was not invoked correctly: ${JSON.stringify(
                relativeMoves
            )}`
        )
    }

    // Exercise shell with success
    const okShell = await custom_controller.post_shell('echo ok').wait()
    if (okShell !== 'executed:echo ok') {
        throw new Error(
            `CustomControllerActor.shell success case did not propagate result, got: ${okShell}`
        )
    }

    // Exercise shell with failure/null
    const failShell = await custom_controller.post_shell('fail').wait()
    if (failShell !== null) {
        throw new Error(
            `CustomControllerActor.shell failure case did not propagate null, got: ${failShell}`
        )
    }

    if (
        shellCommands.length !== 2 ||
        shellCommands[0] !== 'echo ok' ||
        shellCommands[1] !== 'fail'
    ) {
        throw new Error(
            `CustomControllerActor.shell was not invoked with expected commands: ${JSON.stringify(
                shellCommands
            )}`
        )
    }

```

1. At the top of `test/nodejs/binding.ts`, ensure the following imports (or their equivalents in your codebase) exist:
   - `import * as fs from 'fs'`
   - `import * as path from 'path'`
2. If your Node bindings expose different class names or factories, adjust:
   - `maa.RecordController`, `maa.ReplayController`, `maa.CustomControllerActor`, and `maa.CustomController` to the actual exported names.
   - Method names like `post_screenshot`, `post_scroll`, `post_relative_move`, and `post_shell` to match the real async API, including `.wait()` if the promise-wrapping differs.
3. If the test harness expects assertions via a framework (e.g. Mocha/Chai or Jest) instead of throwing `Error`, replace the `throw new Error(...)` checks with the appropriate `expect`/`assert` calls.
4. Confirm that `maa.ErrorCode.Success` is the correct success enum; if not, replace with the appropriate success value.
</issue_to_address>

### Comment 4
<location path="test/pipeline/module/PipelineSmoking.cpp" line_range="7" />
<code_context>

 bool pipeline_smoking(const std::filesystem::path& testset_dir)
 {
-    auto testing_path = testset_dir / "PipelineSmoking" / "MaaRecording.txt";
-    auto result_path = testset_dir / "debug";
+    auto recording_path = testset_dir / "PipelineSmoking" / "MaaRecording.jsonl";

-    auto controller_handle =
</code_context>
<issue_to_address>
**suggestion (testing):** C++ pipeline smoking test now uses MaaReplayControllerCreate but does not validate replay metadata or failure modes

This test now exercises `MaaReplayControllerCreate` with the JSONL recording but only asserts overall success. It would be helpful to also:
- Assert via `MaaControllerGetInfo` (if available) that the controller reports `type = "replay"` and a positive `record_count`.
- Add a negative test that calls `MaaReplayControllerCreate` with a missing or invalid JSONL file and verifies it returns `nullptr`.

That would cover both replay-specific behavior and basic error handling for the new C API.

Suggested implementation:

```cpp
bool pipeline_smoking(const std::filesystem::path& testset_dir)
{
    auto recording_path = testset_dir / "PipelineSmoking" / "MaaRecording.jsonl";

    // Positive path: create replay controller from valid recording
    auto controller_handle = MaaReplayControllerCreate(recording_path.string().c_str());
    if (!controller_handle)
    {
        // Expected to succeed for a valid recording
        return false;
    }

    // Validate controller metadata (replay-specific behavior)
    MaaControllerInfo info{};
    if (!MaaControllerGetInfo(controller_handle, &info))
    {
        return false;
    }

    // type should be "replay"
    if (std::string_view(info.type) != "replay")
    {
        return false;
    }

    // record_count should be positive
    if (info.record_count <= 0)
    {
        return false;
    }

    // Negative path: invalid / missing JSONL file should fail cleanly
    auto invalid_recording_path = testset_dir / "PipelineSmoking" / "NonExistentRecording.jsonl";
    auto invalid_controller_handle = MaaReplayControllerCreate(invalid_recording_path.string().c_str());
    if (invalid_controller_handle != nullptr)
    {
        // MaaReplayControllerCreate should return nullptr on invalid input
        return false;
    }

    auto ctrl_id = MaaControllerPostConnection(controller_handle);

```

Depending on the existing codebase, you may need to:

1. Ensure `MaaControllerInfo` and `MaaControllerGetInfo` are declared by including the correct header (e.g., `#include "MaaController.h"` or similar) at the top of `PipelineSmoking.cpp`.
2. Adjust the `MaaControllerInfo` structure and field names if they differ (e.g., `info.type` might be an enum or a different field name; `info.record_count` might be named differently).
3. If `info.type` is not a C-string, change the comparison accordingly (e.g., compare enums instead of using `std::string_view`).
4. If this test uses a different error-reporting convention (e.g., accumulating a `bool ok` or using test macros), adapt the `return false` lines to match that convention.
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a record/replay controller stack (RecordController + ReplayController) to enable deterministic “record control ops → replay later” workflows (targeting issue #1133), and updates the framework/bindings/tests to use the new APIs and capability model.

Changes:

  • Added new control-unit modules: MaaRecordControlUnit (proxy/record) and MaaReplayControlUnit (replay from JSONL).
  • Refactored controller capability support (scroll / relative_move / shell) via mixin interfaces and updated ControllerAgent + CustomController callback surfaces accordingly.
  • Updated C/C++/Python/NodeJS APIs, tests, docs, and build options/workflows to expose and exercise the new controller types.

Reviewed changes

Copilot reviewed 69 out of 71 changed files in this pull request and generated 8 comments.

Show a summary per file
File Description
test/python/pipeline_test.py Updates pipeline tests and adds a replay-based “smoking” run.
test/python/binding_test.py Updates Python binding tests for new DbgController signature (but has naming/message inconsistencies).
test/pipeline/module/RunWithoutFile.cpp Adjusts pipeline module test to new Dbg controller constructor signature.
test/pipeline/module/PipelineSmoking.cpp Switches pipeline smoking test to the new Replay controller API and JSONL recording format.
test/nodejs/binding.ts Updates NodeJS test usage to new Dbg controller constructor signature.
test/dlopen/main.cpp Updates dlopen test to new DbgControlUnit creation and adds ReplayControlUnit load test.
test/dlopen/CMakeLists.txt Adds WITH_REPLAY_CONTROLLER compile definition.
test/agent/agent_tcp_main_test.py Updates agent TCP test to new DbgController signature.
test/agent/agent_main_test.py Updates agent main test to new DbgController signature.
test/agent/agent_child_test.py Updates expectations for controller info.type to match replay controller (replay).
source/modules/MaaFramework.cppm Exports new controller create APIs (Replay/Record) in C++ module surface.
source/include/LibraryHolder/ControlUnit.h Adds Replay/Record control unit library holders; adjusts Dbg holder API.
source/include/ControlUnit/ReplayControlUnitAPI.h Adds Replay control unit C API header.
source/include/ControlUnit/RecordTypes.h Introduces shared record schema types + JSON helpers.
source/include/ControlUnit/RecordControlUnitAPI.h Adds Record control unit C API header.
source/include/ControlUnit/DbgControlUnitAPI.h Simplifies Dbg control unit C API (removes “type” selector).
source/include/ControlUnit/ControlUnitAPI.h Introduces capability mixin interfaces and FullControlUnitAPI.
source/binding/Python/maa/define.py Removes MaaDbgControllerType enum; extends custom controller callbacks (relative_move/shell).
source/binding/Python/maa/controller.py Updates DbgController API and adds ReplayController/RecordController Python bindings.
source/binding/NodeJS/src/apis/loader.h Declares loaders for ReplayController and RecordController.
source/binding/NodeJS/src/apis/loader.cpp Exposes ReplayController/RecordController constructors in NodeJS binding.
source/binding/NodeJS/src/apis/ext.h Adds GC markers/ctors for ReplayController and RecordController.
source/binding/NodeJS/src/apis/controller.h Updates DbgController ctor params; adds Replay/Record controller impl types.
source/binding/NodeJS/src/apis/controller.d.ts Updates TS declarations for controller capabilities + new Replay/Record controller classes.
source/binding/NodeJS/src/apis/controller.cpp Implements Replay/Record controller ctors; wires custom callbacks for relative_move/shell.
source/binding/NodeJS/src/apis/constant.d.ts Removes exported DbgControllerType constants from Node typings.
source/binding/NodeJS/src/apis/constant.cpp Removes DbgControllerType constant loader from Node binding.
source/binding/NodeJS/src/apis/callback.h Adds custom callback declarations for relative_move and shell.
source/binding/NodeJS/src/apis/callback.cpp Implements custom callback bridges for relative_move and shell.
source/MaaReplayControlUnit/ReplayRecording/ReplayControllerMgr.h Adds Replay controller factory API.
source/MaaReplayControlUnit/ReplayRecording/ReplayControllerMgr.cpp Implements Replay controller factory using RecordParser.
source/MaaReplayControlUnit/ReplayRecording/ReplayController.h Defines ReplayController implementing FullControlUnitAPI.
source/MaaReplayControlUnit/ReplayRecording/ReplayController.cpp Implements replay execution logic (currently strict-aborts on mismatch).
source/MaaReplayControlUnit/ReplayRecording/RecordParser.h Declares JSONL record parser.
source/MaaReplayControlUnit/ReplayRecording/RecordParser.cpp Implements JSONL record parsing and image loading for screencaps.
source/MaaReplayControlUnit/ReplayRecording/Record.h Defines parsed Recording/Record structures used by replay.
source/MaaReplayControlUnit/CMakeLists.txt Adds new shared library target for replay control unit.
source/MaaReplayControlUnit/API/ReplayControlUnitAPI.cpp Exposes Replay control unit C API entrypoints.
source/MaaRecordControlUnit/RecordController.h Defines RecordController proxy that forwards to an inner control unit and records to JSONL.
source/MaaRecordControlUnit/RecordController.cpp Implements record writing, screencap image emission, and forwarding logic.
source/MaaRecordControlUnit/CMakeLists.txt Adds new shared library target for record control unit.
source/MaaRecordControlUnit/API/RecordControlUnitAPI.cpp Exposes Record control unit C API entrypoints.
source/MaaFramework/Controller/ControllerAgent.h Exposes underlying control_unit() accessor (used for RecordControllerCreate).
source/MaaFramework/Controller/ControllerAgent.cpp Uses capability mixins for scroll/relative_move/shell dispatch.
source/MaaFramework/CMakeLists.txt Adds build dependencies for replay/record control units.
source/MaaFramework/API/MaaFramework.cpp Updates MaaDbgControllerCreate signature; adds MaaReplayControllerCreate/MaaRecordControllerCreate.
source/MaaDbgControlUnit/ReplayRecording/ReplayRecordingMgr.h Removes old dbg replay recording manager.
source/MaaDbgControlUnit/ReplayRecording/ReplayRecordingMgr.cpp Removes old dbg replay recording manager impl.
source/MaaDbgControlUnit/ReplayRecording/ReplayRecording.cpp Removes old ReplayRecording implementation from dbg unit.
source/MaaDbgControlUnit/ReplayRecording/RecordParser.h Removes old dbg replay record parser.
source/MaaDbgControlUnit/ReplayRecording/RecordParser.cpp Removes old dbg replay record parser impl.
source/MaaDbgControlUnit/ReplayRecording/Record.h Removes old dbg replay record schema.
source/MaaDbgControlUnit/DbgController.h Renames/reworks dbg controller as a simple image-carousel controller.
source/MaaDbgControlUnit/DbgController.cpp New DbgController implementation: loads images recursively and cycles screencaps.
source/MaaDbgControlUnit/CarouselImage/CarouselImage.cpp Removes old CarouselImage implementation.
source/MaaDbgControlUnit/API/DbgControlUnitAPI.cpp Updates dbg control unit creation API to the new DbgController.
source/MaaCustomControlUnit/Manager/CustomControlUnitMgr.h Adds relative_move and shell to CustomControlUnitMgr surface.
source/MaaCustomControlUnit/Manager/CustomControlUnitMgr.cpp Implements relative_move and shell forwarding to custom callbacks.
source/MaaAgentServer/API/MaaAgentServerNotImpl.cpp Updates “not implemented” stubs for new controller creation APIs.
source/LibraryHolder/ControlUnit/ControlUnit.cpp Adds dynamic loading support for Replay/Record control units; updates Dbg signature.
source/LibraryHolder/CMakeLists.txt Adds dependencies for replay/record control units (and wlroots).
source/CMakeLists.txt Adds subdirectories for replay/record control units.
include/MaaFramework/MaaDef.h Removes MaaDbgControllerType from the public C headers.
include/MaaFramework/Instance/MaaCustomController.h Extends custom controller callbacks with relative_move and shell.
include/MaaFramework/Instance/MaaController.h Documents and exposes new MaaDbg/MaaReplay/MaaRecord controller C APIs.
docs/en_us/2.2-IntegratedInterfaceOverview.md Updates integrated interface docs for new controller APIs/info fields.
CMakeLists.txt Adds build options for replay/record controllers; changes pipeline testing gating.
.github/workflows/test.yml Enables replay controller build in CI test matrices.
.cursor/.gitignore Ignores Cursor plans directory.

@MistEO
Copy link
Copy Markdown
Member Author

MistEO commented Mar 26, 2026

@sourcery-ai summary

@MistEO MistEO changed the title feat: ProxyController feat: Record & Replay Controller Mar 26, 2026
@MistEO
Copy link
Copy Markdown
Member Author

MistEO commented Mar 26, 2026

@sourcery-ai summary

@MistEO
Copy link
Copy Markdown
Member Author

MistEO commented Mar 27, 2026

@sourcery-ai review

Copy link
Copy Markdown
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - 我这边发现了 1 个问题,并给出了一些高层次的反馈:

  • 现在 Python 的 CustomController 类把 relative_moveshell 声明为抽象方法;这会对现有子类造成破坏性变更——建议提供默认实现(例如返回 False/None),或者将其设为可选,以保持向后兼容性。
  • ReplayController 的析构函数在并非所有记录都被消费时会调用 std::abort(),这在严格测试场景下是合适的,但在其他用法中可能会让人意外;可以考虑用构建标志或配置开关来加以控制,避免在生产环境或临时回放运行中直接导致整个进程退出。
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- The Python `CustomController` class now declares `relative_move` and `shell` as abstract methods; this is a breaking change for existing subclasses—consider providing default implementations (e.g., returning `False`/`None`) or making them optional to preserve backward compatibility.
- The `ReplayController` destructor calls `std::abort()` whenever not all records are consumed, which is appropriate for strict testing but may be surprising in other usages; it may be worth guarding this with a build flag or configuration toggle so production or ad‑hoc replay runs don’t bring down the whole process.

## Individual Comments

### Comment 1
<location path="source/MaaRecordControlUnit/RecordController.cpp" line_range="35-37" />
<code_context>
+        LogError << "Failed to create screenshot directory:" << screenshot_dir_ << VAR(ec.message());
+    }
+
+    record_file_.open(recording_path_, std::ios::out | std::ios::trunc);
+    if (!record_file_.is_open()) {
+        LogError << "Failed to open recording file:" << recording_path_;
+    }
+}
</code_context>
<issue_to_address>
**suggestion (bug_risk):** 当无法打开录制文件时,应立即失败或向外暴露错误。

按当前写法,即使 `record_file_` 打开失败,`connect()` 仍然会报告成功,而后续操作会默默丢弃所有记录(日志除外)。请考虑要么将此视为硬性失败(例如让 `connect()` 失败),要么暴露错误状态(例如通过 `get_info()`/状态标志),从而让调用方可以检测到录制实际上没有进行。

建议实现:

```cpp
    record_file_.open(recording_path_, std::ios::out | std::ios::trunc);
    if (!record_file_.is_open()) {
        LogError << "Failed to open recording file:" << recording_path_;
        recording_enabled_ = false;
    } else {
        recording_enabled_ = true;
    }

```

为了完整实现“快速失败 / 错误外显”的行为,你还应该:

1. **在对应的头文件**(例如 `RecordController.h`)中添加一个成员标志:
   ```cpp
   bool recording_enabled_ = false;
   ```
2. **`connect()` 的返回值 / 状态上使用该标志**- 如果 `connect()` 返回状态或 bool,将 `recording_enabled_` 纳入结果中,这样当文件无法打开时可以让 `connect()` 报告失败。
   - 如果不能更改 `connect()` 的签名/契约,确保通过 `get_info()` 或类似接口暴露的状态中包含由 `recording_enabled_` 派生出的字段,以便调用方可以明确检测到录制因文件打开失败而被禁用。
3. **在录制相关操作前加保护**(例如任何向 `record_file_` 写入的函数):
   ```cpp
   if (!recording_enabled_) {
       // 可选择只记录一次日志,然后提前返回以避免静默丢弃记录
       return;
   }
   ```
4. 可选地,**扩展 `get_info()` 或等价接口**,增加一个显式的“recording active/available(录制是否激活/可用)”布尔值或错误字符串,使 UI / 上层代码在因文件 I/O 错误而无法开始录制时,可以向用户给出反馈。
</issue_to_address>

Sourcery 对开源项目免费——如果你觉得我们的评审有帮助,请考虑帮忙分享 ✨
帮我变得更有用!请在每条评论上点 👍 或 👎,我会根据这些反馈改进后续评审。
Original comment in English

Hey - I've found 1 issue, and left some high level feedback:

  • The Python CustomController class now declares relative_move and shell as abstract methods; this is a breaking change for existing subclasses—consider providing default implementations (e.g., returning False/None) or making them optional to preserve backward compatibility.
  • The ReplayController destructor calls std::abort() whenever not all records are consumed, which is appropriate for strict testing but may be surprising in other usages; it may be worth guarding this with a build flag or configuration toggle so production or ad‑hoc replay runs don’t bring down the whole process.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- The Python `CustomController` class now declares `relative_move` and `shell` as abstract methods; this is a breaking change for existing subclasses—consider providing default implementations (e.g., returning `False`/`None`) or making them optional to preserve backward compatibility.
- The `ReplayController` destructor calls `std::abort()` whenever not all records are consumed, which is appropriate for strict testing but may be surprising in other usages; it may be worth guarding this with a build flag or configuration toggle so production or ad‑hoc replay runs don’t bring down the whole process.

## Individual Comments

### Comment 1
<location path="source/MaaRecordControlUnit/RecordController.cpp" line_range="35-37" />
<code_context>
+        LogError << "Failed to create screenshot directory:" << screenshot_dir_ << VAR(ec.message());
+    }
+
+    record_file_.open(recording_path_, std::ios::out | std::ios::trunc);
+    if (!record_file_.is_open()) {
+        LogError << "Failed to open recording file:" << recording_path_;
+    }
+}
</code_context>
<issue_to_address>
**suggestion (bug_risk):** Fail-fast or surface an error when the recording file cannot be opened.

As written, `connect()` still reports success when `record_file_` fails to open, and later operations silently drop all records (aside from logs). Please either treat this as a hard failure (e.g., make `connect()` fail) or expose the error state (e.g., via `get_info()`/a status flag) so callers can detect that recording is not occurring.

Suggested implementation:

```cpp
    record_file_.open(recording_path_, std::ios::out | std::ios::trunc);
    if (!record_file_.is_open()) {
        LogError << "Failed to open recording file:" << recording_path_;
        recording_enabled_ = false;
    } else {
        recording_enabled_ = true;
    }

```

To fully implement the fail-fast / error-surfacing behavior, you should also:

1. **Add a member flag** to the corresponding header (e.g., `RecordController.h`):
   ```cpp
   bool recording_enabled_ = false;
   ```
2. **Use this flag in `connect()`’s return / status reporting path**:
   - If `connect()` returns a status or bool, incorporate `recording_enabled_` into that result so `connect()` can report failure when the file cannot be opened.
   - If `connect()` cannot change its signature/contract, ensure that the status exposed via `get_info()` or similar includes a field derived from `recording_enabled_` so callers can explicitly detect that recording is disabled due to file open failure.
3. **Guard recording operations** (e.g., any method that writes to `record_file_`) with:
   ```cpp
   if (!recording_enabled_) {
       // optionally log once, then return early to avoid silently discarding records
       return;
   }
   ```
4. Optionally, **extend `get_info()` or equivalent** to expose an explicit “recording active/available” boolean or error string so UI / higher-level code can give user feedback when recording cannot start due to file I/O errors.
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 69 out of 71 changed files in this pull request and generated 9 comments.

@MistEO MistEO requested a review from Copilot March 27, 2026 02:02
@MistEO
Copy link
Copy Markdown
Member Author

MistEO commented Mar 27, 2026

@sourcery-ai review

Copy link
Copy Markdown
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - 我在这里给出了一些高层次的反馈:

  • 当记录索引没有到达末尾时,ReplayController::~ReplayController 会无条件调用 std::abort()。这在 CI 环境下尚可,但在其他任何嵌入式上下文中都很危险;建议将其置于仅调试/仅测试标志之后,或者至少使用可配置的断言钩子,这样生产环境的宿主就不会被直接硬中止。
  • DbgController::connect 会立即将路径下的所有图片加载到内存中,对于大型目录来说,这会非常耗资源;建议要么限制加载图片的数量/大小,要么按截图懒加载,以避免出现意外的内存峰值。
面向 AI Agent 的提示
Please address the comments from this code review:

## Overall Comments
- ReplayController::~ReplayController unconditionally calls std::abort() when the record index does not reach the end, which is fine for CI but dangerous in any other embedding context; consider gating this behind a debug/test-only flag or at least a configurable assertion hook so production hosts are not hard-aborted.
- DbgController::connect eagerly loads all images under the path into memory, which can be very heavy for large directories; consider either limiting the number/size of images loaded or lazily loading per screencap to avoid unexpected memory spikes.

Sourcery 对开源项目是免费的——如果你觉得我们的代码审查有帮助,欢迎分享 ✨
帮我变得更有用!请在每条评论上点击 👍 或 👎,我会根据你的反馈改进后续的代码审查。
Original comment in English

Hey - I've left some high level feedback:

  • ReplayController::~ReplayController unconditionally calls std::abort() when the record index does not reach the end, which is fine for CI but dangerous in any other embedding context; consider gating this behind a debug/test-only flag or at least a configurable assertion hook so production hosts are not hard-aborted.
  • DbgController::connect eagerly loads all images under the path into memory, which can be very heavy for large directories; consider either limiting the number/size of images loaded or lazily loading per screencap to avoid unexpected memory spikes.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- ReplayController::~ReplayController unconditionally calls std::abort() when the record index does not reach the end, which is fine for CI but dangerous in any other embedding context; consider gating this behind a debug/test-only flag or at least a configurable assertion hook so production hosts are not hard-aborted.
- DbgController::connect eagerly loads all images under the path into memory, which can be very heavy for large directories; consider either limiting the number/size of images loaded or lazily loading per screencap to avoid unexpected memory spikes.

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 69 out of 71 changed files in this pull request and generated 5 comments.

@MistEO
Copy link
Copy Markdown
Member Author

MistEO commented Mar 27, 2026

@MaaXYZ/binding-developers 搞好了,来康康,DbgController重构了一下,然后新增了两个 Controller 用于录制和重播

@MistEO MistEO merged commit bec73b3 into main Mar 27, 2026
18 checks passed
@MistEO MistEO deleted the feat/proxy_controller branch March 27, 2026 03:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

用于单测的 ProxyController

3 participants