Skip to content

feat(connector): add Mem0 Platform V3 ADD sink - #12233

Merged
davidzollo merged 14 commits into
apache:devfrom
CryoThrust:feat/mem0-v3-add
Sep 13, 2026
Merged

davidzollo merged 14 commits into
apache:devfrom
CryoThrust:feat/mem0-v3-add

Conversation

@CryoThrust

Copy link
Copy Markdown
Contributor

Summary

  • add a connector-local sink for the documented Mem0 Platform V3 asynchronous add API
  • require a non-blank event_id before acknowledging accepted delivery
  • reuse the existing HTTP connector configuration while keeping provider response validation local
  • add focused payload and missing-event-id tests

Scope

This draft targets only the hosted Mem0 Platform V3 profile:

  • POST /v3/memories/add/
  • Authorization: Token <api-key>
  • Content-Type: application/json and Accept: application/json
  • asynchronous accepted delivery; event polling is outside this PR

There is no DELETE operation, self-hosted OSS profile, generic Mem0-compatible claim, or core/SPI change.

Closes #12102

Verification

  • module Spotless check passed
  • git diff --check passed
  • full reactor compilation is currently blocked in this checkout by the existing seatunnel-config-shade source/dependency issue (ConfigSyntax, ConfigIncluder, and related shaded Typesafe Config symbols unresolved); this is independent of the new connector

@CryoThrust

Copy link
Copy Markdown
Contributor Author

Rebased the draft branch into one DCO-signed commit (d5b37ec2) without changing the implementation or documentation. The queued notification/labeler workflows have been retriggered.

@CryoThrust
CryoThrust marked this pull request as ready for review September 9, 2026 12:12
@CryoThrust

Copy link
Copy Markdown
Contributor Author

The initial implementation now covers the approved V3 ADD-only boundary, required headers, nonblank event_id acceptance, focused payload/response tests, and EN/ZH documentation. Converting from Draft so the repository CI can compile the module and expose any project-integration issues; the remaining status/retry/redaction matrix will be completed from those results.

@CryoThrust

Copy link
Copy Markdown
Contributor Author

The repository notification and labeler runs for the updated head have remained queued since 12:10 CST and have not created the full test workflow. Could a maintainer re-run or trigger the standard PR test workflow when convenient? The branch is ready for that CI gate.

@CryoThrust

Copy link
Copy Markdown
Contributor Author

Follow-up pushed as signed commit aaec5bc6: optional scope fields are now ignored when null/blank per row, and the writer accepts any non-empty user_id, agent_id, app_id, or run_id scope. Added a regression test for agent_id when user_id is null. Module Spotless remains green.

@CryoThrust

Copy link
Copy Markdown
Contributor Author

Added signed follow-up commit 5ad38f32: explicit retries for HTTP 408/429/5xx using the existing retry/backoff settings, terminal handling for other non-2xx responses, and tests for transient-status classification and retry sequencing. Module Spotless remains green.

@CryoThrust

Copy link
Copy Markdown
Contributor Author

Added signed follow-up af73d8d6: exposed the existing HTTP retry, backoff multiplier, and max-backoff options in the Mem0 factory rule so the new status retry behavior is configurable.

@DanielLeens DanielLeens left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What Problem Does This PR Solve?

This adds a new connector-mem0 sink under connector-http that writes SeaTunnel rows to the hosted Mem0 Platform V3 asynchronous add API (POST /v3/memories/add/), acknowledging a row only when the response contains a non-empty event_id. The scope is explicitly limited to ADD for the hosted profile — no delete, no self-hosted OSS profile, no event polling. Closes #12102.

1. Code Change Review

1.1 Core Logic Analysis

Before → After: there is no existing Mem0 connector; this adds Mem0Sink/Mem0SinkWriter/Mem0SinkFactory/Mem0Options under seatunnel-connectors-v2/connector-http/connector-mem0, reusing connector-http-base's HttpClientProvider/HttpParameter for the actual HTTP transport.

Runtime flow:

Mem0SinkFactory.createSink() -> Mem0Sink(ReadonlyConfig, CatalogTable)
  constructor resolves messages/user_id/agent_id/app_id/run_id/metadata field names,
  builds HttpParameter with url = base_url + "/v3/memories/add/", headers (Authorization/Content-Type/Accept)
Mem0Sink.createWriter() -> Mem0SinkWriter
  write(row):
    serialize row to JSON via JsonSerializationSchema -> ObjectNode
    require non-empty "messages" array, at least one of the four scope fields non-empty
    send(request) -> httpClient.doPost(url, headers, body)
      loop: on 2xx -> validateAcceptedResponse (require non-empty "event_id"), return
            on retryable status (408/429/5xx) and attempt < maxRetries -> sleep(backoff), retry
            otherwise -> throw IOException

This hits the normal write path for every row, not a recovery/boundary path — there's no split/checkpoint state at all in this connector (a plain AbstractSimpleSink), so there's no restore-path complexity to trace.

Key Findings — I found one severe issue that has nothing to do with Mem0 itself:

Issue 1: This PR silently deletes the packaging registration for an unrelated, already-in-dev connector (connector-amazondocumentdb)

  • Location: seatunnel-dist/pom.xml (removed hunk around the former line 619; git diff shows a clean 6-line removal with nothing re-added elsewhere)
  • Problem description: I compared seatunnel-dist/pom.xml at dev's current tip against this PR's head directly (not just the diff hunk): dev has a <dependency><artifactId>connector-amazondocumentdb</artifactId>...</dependency> block; this PR's head has no occurrence of amazondocumentdb anywhere in the file. This isn't a rename or a reorder — the entry is simply gone. connector-amazondocumentdb is not this PR's connector, is not mentioned anywhere in the PR description, and nothing in this PR's own diff touches AmazonDocumentDB.
  • Potential risk: if merged as-is, the distribution build (seatunnel-dist) would stop bundling connector-amazondocumentdb, silently breaking any user relying on that connector via the standard dist package — a "half-dead" connector exactly matching the failure pattern this project has been burned by before (an existing, unrelated connector's registration dropped as collateral damage of an unrelated PR's branch history). Given the branch went through several rebase/squash cycles (the author's own comments mention "rebased the draft branch into one DCO-signed commit" and multiple follow-up pushes) while connector-amazondocumentdb was being added to dev in parallel, this has all the signatures of an accidental bad merge/rebase resolution, not an intentional change.
  • Best improvement: restore the connector-amazondocumentdb dependency block in seatunnel-dist/pom.xml exactly as it exists on current dev, then add the new connector-mem0 entry alongside it (the PR does correctly add connector-mem0 itself — that part is fine). Please also do a final git diff --stat/git diff self-review specifically for deletion lines before repushing, to make sure no other unrelated entry was dropped by the same mechanism.
  • Severity: High (blocking — this is an unrelated regression, not a Mem0-specific defect, and per this project's own "any deletion of an existing registration must be justified" review standard it cannot be waved through even though the new connector's own logic is fine).
  • Raised by another reviewer: No.

Beyond that, the Mem0-specific logic itself is reasonable: Mem0SinkWriter.required()/copyIfPresent() correctly treat blank/whitespace-only strings the same as absent (not just null), validateAcceptedResponse() correctly distinguishes "HTTP 2xx but no real event_id" from a true failure, and normalizeBaseUrl() correctly avoids a double slash when api_base_url already ends in /.

1.2 Compatibility Impact

Partially incompatible, entirely due to Issue 1 above — a brand-new connector cannot itself break compatibility for existing jobs, but the accidental removal of connector-amazondocumentdb's dist packaging is a real compatibility regression for that connector's users. Excluding that one file, everything else is purely additive (new module, new doc pages, new plugin-mapping.properties-eligible identifier — see Issue 3 below).

1.3 Performance / Side-Effect Analysis

Issue 2: The connector's own status-code retry loop and the base HttpClientProvider's built-in retryer both key off the same retry/backoff options, but retry on different failure types

  • Location: Mem0SinkWriter.java send() (the for (int attempt = 0; ; attempt++) loop, using maxRetries/retryBackoffMultiplierMillis/retryBackoffMaxMillis copied from HttpParameter) vs. connector-http-base's HttpClientProvider#buildRetryer()/getResponse() (which already wraps every doPost() call in a Retryer built from the exact same httpParameter.getRetry()/backoff fields).
  • Problem description: I read HttpClientProvider directly: its retryer only retries when the underlying HTTP call throws an IOException (connection-level failure) — retryIfException(ex -> ExceptionUtils.indexOfType(ex, IOException.class) != -1) — it does not look at the response status code at all. So Mem0SinkWriter's own loop, which retries on HTTP-level 408/429/5xx responses (not exceptions), is not redundant — it covers a genuinely different failure class the base client doesn't handle. However, both mechanisms are driven by the same api.http.retry config value, so a user configuring "retry = 3" gets up to 3 status-code-based retries in Mem0SinkWriter, and each of those individual doPost() calls can independently retry up to 3 more times internally on a connection-level IOException. This isn't an unbounded-retry or stability bug (both loops are still bounded), but it means the effective worst-case attempt count and total wait time under a persistent connection issue is larger than what a user configuring "retry = 3" would reasonably expect, and the two layers' backoff delays stack rather than compose predictably.
  • Potential risk: Low — bounded, not a stability risk, just a config-semantics surprise that could make troubleshooting a slow/failing sink harder to reason about ("why did this take so much longer than retry=3 implies").
  • Best improvement: either document this layering explicitly (a short comment on Mem0SinkWriter's retry loop noting it's independent of and stacks with the base client's connection-level retryer), or consider disabling the base retryer for this connector's usage (e.g. pass retry=0 when constructing the HttpParameter actually used for the transport, and drive all retry/backoff exclusively from Mem0SinkWriter's own status-aware loop) so there's a single, predictable retry budget.
  • Severity: Low.
  • Raised by another reviewer: No.

No new unbounded collections, no thread creation (retries block the writer's own thread via TimeUnit.MILLISECONDS.sleep, which is the same blocking-retry pattern already used by every other connector built on HttpClientProvider/AbstractSinkWriter, so this isn't a new architectural risk).

1.4 Error Handling and Logging

send()'s outer catch (IOException e) { throw e; } catch (Exception e) { throw new IOException(..., e); } correctly preserves the original IOException type/message rather than double-wrapping, and correctly converts any other unexpected exception into a typed failure with context. waitBeforeRetry() correctly restores the interrupt flag (Thread.currentThread().interrupt()) before converting an InterruptedException into an IOException rather than swallowing it. No sensitive data (the api_key, once embedded in the Authorization header) is logged anywhere in the new code — I checked for any log.info/log.debug of headers or the request body and found none.

2. Code Quality Assessment

2.1 Coding Standards

Issue 3: New public config class is missing project-required license header on 6 new files, and is misnamed relative to the project's <ConnectorClassName>Options convention

Two separate, CI-confirmed findings bundled here since they're both "coding standards" issues surfaced by the project's own automated checks:

3a — License header. I pulled the fork's actual failing CI job (CryoThrust/seatunnel run 34351043343, job "License header") directly: it fails with Totally checked 9193 files ... invalid: 6 and lists all 6 new source files (connector-mem0/pom.xml, Mem0Options.java, Mem0Sink.java, Mem0SinkFactory.java, Mem0SinkWriter.java, Mem0SinkWriterTest.java). Looking at the diff, each of these carries a shortened license comment (4 lines: "Licensed to the ASF ... Version 2.0.") instead of the full standard ASF header used everywhere else in this repository (the complete grant/URL/AS-IS-disclaimer block). That's why skywalking-eyes' header linter rejects them.

  • Best improvement: replace the short header on all 6 files with the project's standard full ASF header (see any existing file in connector-http-base for the exact text, or the template documented in this project's contribution guide).
  • Severity: High (blocking CI).

3b — Options class naming. The same CI run's "Code style" job fails ConnectorOptionCheckTest.checkConnectorOptionExist with: "Connector class does not have correspondingly [Options] class ... Those [Options] class are missing: Mem0SinkOptions". This project's convention (enforced by that test, and stated in this repo's own contribution guide) is <ConnectorClassName>Options, e.g. ActivemqSink -> ActivemqSinkOptions. This PR instead defines a single shared Mem0Options (extending HttpCommonOptions) referenced from both Mem0Sink and Mem0SinkFactory.

  • Best improvement: rename Mem0Options to Mem0SinkOptions (there's no source connector in this PR, so no ambiguity about which side it belongs to).
  • Severity: High (blocking CI).

Issue 4: plugin-mapping.properties is not updated for the new connector

  • Location: plugin-mapping.properties (not touched by this PR's diff at all)
  • Problem description: every other sink of this kind in the codebase registers a seatunnel.sink.<Identifier> = <module> line (e.g. seatunnel.sink.SNMP = connector-snmp, confirmed by reading the current dev copy of this file directly). This PR adds Mem0 as a sink identifier via Mem0SinkFactory.factoryIdentifier() but never adds a corresponding seatunnel.sink.Mem0 = connector-mem0 line.
  • Potential risk: the connector is still bundled into the full "all-connectors" seatunnel-dist build (via the pom.xml change under provided scope) and will work in that packaging mode, but a user doing a plugin-scoped install (bin/install-plugin.sh selecting individual connectors, or any tooling that reads plugin-mapping.properties to resolve which module a named connector lives in) will not be able to locate/install Mem0 by name — a "half-registered" connector matching the registration-matrix gap this project's review standards specifically call out.
  • Best improvement: add seatunnel.sink.Mem0 = connector-mem0 to plugin-mapping.properties.
  • Severity: Medium.
  • Raised by another reviewer: No.

2.2 Test Coverage and Test Stability

Mem0SinkWriterTest has five tests: happy-path payload/event_id round-trip, rejecting a 2xx response without event_id, retrying 429-then-503-then-success and asserting exactly 3 total calls, a static-method table test for isRetryableStatus, and a fallback-to-secondary-scope-field test. These are genuinely targeted at the correctness-critical logic (scope-field validation, event_id gating, retry-then-succeed counting), not just padding.

Stability rating: Stable. All tests are Mockito-based with fixed, deterministic stub sequences (.thenReturn(a).thenReturn(b).thenReturn(c)), no Thread.sleep-based waits in the test code itself (the retry test's production code does sleep between attempts via waitBeforeRetry, but with retryBackoffMultiplierMillis/MaxMillis left at their small defaults via HttpParameter's own defaults — not egregious, and doesn't affect determinism of the assertion), no shared static state, no ordering dependence, no floating-point/string-normalization concerns.

Coverage gaps (non-blocking but worth calling out): there is no test at the Mem0Sink/Mem0SinkFactory level — nothing exercises OptionRule completeness (required vs. optional fields), the normalizeBaseUrl() trailing-slash handling, or the "no scope field configured at all" / "metadata field present but not a JSON object" rejection paths in write(). Given how small and self-contained this connector is, I'd treat these as recommended follow-ups rather than blockers.

2.3 Documentation Updates

Issue 5: New connector docs don't follow this project's own doc-page convention, and this is a second CI-confirmed failure

  • Location: docs/en/connectors/sink/Mem0.md:1, docs/zh/connectors/sink/Mem0.md:1
  • Problem description: I pulled the same failing "Code style" job: MarkdownTest.testPrimaryHeadersHaveNoTextAbove and testConnectorDocWithChangeLogFlagAndFile both fail because the first line of Mem0.md is an ASF license HTML comment (<!-- Licensed to ... -->) rather than either a first-level # heading or a import ChangeLog from '../changelog/connector-mem0.md'; directive. I checked two existing connector doc pages directly (Activemq.md, and the recently-added SNMP.md) and both start with the import ChangeLog from ... line, immediately followed by # <Name> — no license comment at the top of doc pages anywhere else in this repository (the .licenserc.yaml license-header check doesn't even flag .md files, confirming docs aren't expected to carry that header).
  • Best improvement: remove the <!-- ... --> license block from both Mem0.md files, add import ChangeLog from '../changelog/connector-mem0.md'; as the new first line (matching SNMP.md's pattern), and add the corresponding docs/en/connectors/changelog/connector-mem0.md (and, for consistency with the SNMP precedent, docs/zh/connectors/changelog/connector-mem0.md) stub file with a <details><summary> Change Log </summary> table entry for this feature.
  • Severity: High (blocking CI — same root cause class as Issue 3, both are format/convention mismatches the project's own test suite already codifies).
  • Raised by another reviewer: No.

Content-wise (once the format issue above is fixed), both the English and Chinese doc bodies are accurate and substantively equivalent — they correctly describe the ADD-only Phase-1 scope, the four scope fields, and the event_id acknowledgment semantics matching the actual code.

3. Architectural Soundness

3.1 Elegance of the Solution

Precise fix for what it sets out to do — reusing connector-http-base's HttpParameter/HttpClientProvider rather than hand-rolling a new HTTP client is the right call, and scoping this PR tightly to ADD-only (deferring delete/self-hosted/polling) is a sensible, reviewable slice. The severity here is entirely in packaging/registration completeness (Issues 1, 3, 4, 5), not the core connector design.

3.2 Maintainability

Good structure (Options/Sink/SinkFactory/SinkWriter split matches every other connector in this codebase), once the naming/header/doc conventions above are corrected.

3.3 Extensibility

A future "Phase 2" (delete, self-hosted OSS profile, event polling) has a clear place to land: Mem0Options(→Mem0SinkOptions) for new config, a new Mem0Source/additional writer methods without touching the ADD-only path added here.

3.4 Historical-Version Compatibility

No checkpoint/savepoint state (this is a stateless AbstractSimpleSink), no serialized split format, no protocol field touched. The one compatibility concern is Issue 1 (the unrelated connector-amazondocumentdb dist-packaging removal), which is a regression risk for that connector's users, not a Mem0-versioning concern.

4. Issue Summary

Number Issue Location Severity
1 Unrelated deletion of connector-amazondocumentdb's dist-packaging dependency seatunnel-dist/pom.xml High
2 Connector's own retry loop and base HttpClientProvider's built-in retryer both consume the same retry/backoff config for different failure classes, stacking unpredictably Mem0SinkWriter.java send() Low
3 Missing full ASF license header (6 files) and Mem0Options misnamed vs. required Mem0SinkOptions convention — both CI-blocking Mem0Options.java, Mem0Sink.java, Mem0SinkFactory.java, Mem0SinkWriter.java, Mem0SinkWriterTest.java, pom.xml High
4 Missing plugin-mapping.properties entry for the new sink identifier plugin-mapping.properties Medium
5 Doc pages don't follow project convention (license comment instead of ChangeLog import + heading) — CI-blocking docs/en/connectors/sink/Mem0.md, docs/zh/connectors/sink/Mem0.md High

5. Merge Recommendation

Conclusion: Not recommended for merge

  1. Blockers — must be fixed

    • Issue 1 (High): restore the connector-amazondocumentdb dependency in seatunnel-dist/pom.xml. This is the most important item in this review — it's an unrelated regression, not something reviewable as part of "is the Mem0 connector correct."
    • Issue 3 (High): add the full standard ASF license header to all 6 new files, and rename Mem0Options to Mem0SinkOptions. Both are directly why the fork's CI "License header" and "Code style" jobs are currently failing.
    • Issue 5 (High): fix the two new doc pages to follow this project's import ChangeLog ... + heading convention instead of a license comment; add the corresponding changelog stub file(s). This is the third and last cause of the current CI failure.
  2. Recommended fixes — non-blocking

    • Issue 2: document or eliminate the retry-layering ambiguity between Mem0SinkWriter's status-code retry loop and HttpClientProvider's connection-level retryer.
    • Issue 4: add the seatunnel.sink.Mem0 = connector-mem0 line to plugin-mapping.properties for plugin-scoped installs.
    • Add Mem0Sink/Mem0SinkFactory-level tests (option-rule completeness, base-URL normalization, all-scope-fields-absent rejection).

CI status: the fork's current head (af73d8d6ca15b34f44f759d333086ed6ac89dc1b) fails with conclusion: failure. I dereferenced the actual failing jobs directly (not just the apache-side pointer): Run / License header and Run / Code style both fail, and I traced every individual assertion above to a specific, PR-related cause (missing license header, wrong Options class name, wrong doc format) — none of this is environmental/flaky, and none of it should be blamed on the PR description's claim about an unrelated seatunnel-config-shade reactor issue (that claim describes the author's local checkout, not what's actually failing in CI).

Thank you for the clean scoping of this connector to ADD-only — the core write/retry/validation logic is solid work. The blockers above are all mechanical (packaging/naming/doc-format) rather than logic defects, so I'd expect this to be quick to turn around; Issue 1 in particular needs a careful look since it's easy to miss in a routine self-review (it shows up as a deletion, not something obviously connected to "did I break my own connector").

@CryoThrust

Copy link
Copy Markdown
Contributor Author

Addressed the packaging regression from review in signed commit 01cbb9540: restored the existing connector-amazondocumentdb dependency in seatunnel-dist/pom.xml while keeping the new connector-mem0 registration. git diff --check passes. A standalone seatunnel-dist validate was attempted; dependency resolution is blocked by unavailable/private SNAPSHOT mirrors, not by this POM change.

@CryoThrust

Copy link
Copy Markdown
Contributor Author

Follow-up pushed as signed commit. Addressed the remaining mechanical CI/review blockers: renamed Mem0Options to Mem0SinkOptions, added standard ASF headers to the six new files, registered seatunnel.sink.Mem0 in plugin-mapping.properties, and switched EN/ZH docs to the required ChangeLog import with changelog entries. Also preserved connector-amazondocumentdb in seatunnel-dist/pom.xml. git diff --check passes.

@CryoThrust

Copy link
Copy Markdown
Contributor Author

Spotless check now passes for connector-mem0 after formatting follow-up b005eb7ed (signed). Waiting for repository CI and maintainer review.

@CryoThrust

Copy link
Copy Markdown
Contributor Author

Fixed the remaining MarkdownTest failure in signed commit 29bf8103d: moved <ChangeLog /> to the end of both EN/ZH Mem0 connector docs, matching the repository convention. The docs now retain the ChangeLog import at the top and the marker at EOF.

@CryoThrust

Copy link
Copy Markdown
Contributor Author

The full CI run identified one test-only failure in Mem0SinkWriterTest.retriesRateLimitAndServerResponses: the test set retry backoff multiplier to 0, but the underlying retryer requires multiplier > 0 and failed before exercising the status retry sequence.

I pushed signed commit cf5782e19 to the PR branch, setting multiplier/max backoff to 1 ms. This keeps the test deterministic and preserves the intended three-request assertion without changing production behavior. The focused module test is being re-run; the failure was not a connector runtime failure.

@DanielLeens DanielLeens left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for missing this in the last review — that's on me. I re-reviewed the current head (cf5782e) end to end rather than just the delta, and want to flag one item below that I should have caught the first time around. It's a carryover from the version I reviewed yesterday, not something newly introduced by these commits.

First, the good news: all five blockers from my previous review are now fixed, and I verified each one directly against the current head rather than trusting the commit messages:

  • Issue 1 (unrelated connector-amazondocumentdb deletion) — fixed. I fetched seatunnel-dist/pom.xml at the current head and at dev directly and confirmed the connector-amazondocumentdb dependency block is present, identical, in both. I also re-scanned the full current diff for any deletion lines (grep -n "^-" against the diff) and found none anywhere outside the standard --- a/... diff headers — this PR is now purely additive.
  • Issue 3a (license headers) — fixed. All 6 previously-flagged files now carry the full standard ASF header.
  • Issue 3b (Mem0Options → Mem0SinkOptions naming) — fixed, confirmed by reading the current file.
  • Issue 4 (plugin-mapping.properties) — fixed: seatunnel.sink.Mem0 = connector-mem0 is now present.
  • Issue 5 (doc page format) — fixed: both docs/en/connectors/sink/Mem0.md and docs/zh/connectors/sink/Mem0.md now start with the import ChangeLog from '../changelog/connector-mem0.md'; line, and the corresponding changelog stub files were added.

I confirmed via the fork's own CI (run 34586780929, same head SHA) that Run / License header, Run / Code style, Run / Dead links, and Run / Sanity check results all now pass — matching my independent read of the diff. The unit-test matrix jobs were still in progress at the time of this review; the one prior test failure I traced (Mem0SinkWriterTest.retriesRateLimitAndServerResponses on run 34570172097, IllegalArgumentException: multiplier must be > 0 but is [0]) was purely a test-fixture bug — the base HttpClientProvider's Guava-backed retryer rejects a 0 backoff multiplier — and the final commit (cf5782e19) fixed it by using 1 instead of 0 in the test. That's a test-only change; it doesn't touch Mem0SinkWriter's production retry logic at all, so it doesn't affect my analysis below.

1. Code Change Review

1.1 Core Logic Analysis

The write path is unchanged from my prior review and still reads correctly:

Mem0SinkFactory.createSink() -> Mem0Sink(ReadonlyConfig, CatalogTable)
  resolves messages/user_id/agent_id/app_id/run_id/metadata field names,
  builds HttpParameter with url = base_url + "/v3/memories/add/"
Mem0Sink.createWriter() -> Mem0SinkWriter
  write(row): serialize row to JSON -> require non-empty "messages" array,
              at least one of the four scope fields non-empty -> send(request)
    send(): httpClient.doPost() -> on 2xx, require non-empty "event_id";
             on 408/429/5xx and attempt < maxRetries, backoff and retry;
             otherwise throw IOException

Key findings:

  • The core write/validate/retry logic is unchanged and still correct: required()/copyIfPresent() correctly treat blank strings the same as absent, validateAcceptedResponse() correctly distinguishes "HTTP 2xx but no real event_id" from a true failure, and normalizeBaseUrl() avoids a double slash.
  • All 5 previously-blocking packaging/naming/doc issues are now resolved, as detailed above.
  • One issue that existed in the version I reviewed yesterday, and still exists today, slipped past me: user_id_field is read via config.get(Mem0SinkOptions.USER_ID_FIELD) in Mem0Sink's constructor and is a real, working, documented option — but it is not declared anywhere in Mem0SinkFactory.optionRule() (not required(...), not optional(...)). See Issue 1 below; I traced this to a real, reproducible failure of a specific, actively-used validation command, not just a schema-hygiene nitpick.

Issue 1: user_id_field is missing from Mem0SinkFactory.optionRule(), which breaks --dry-run check/--dry-run connect for the connector's own documented example

  • Location: seatunnel-connectors-v2/connector-http/connector-mem0/src/main/java/org/apache/seatunnel/connectors/seatunnel/mem0/sink/Mem0SinkFactory.java (the optionRule() method — USER_ID_FIELD is absent from both the .required(...) and .optional(...) lists) and Mem0SinkOptions.java:203-207 (where USER_ID_FIELD is defined but never wired into the factory's rule) and Mem0Sink.java:283 (this.userIdField = config.get(Mem0SinkOptions.USER_ID_FIELD);).
  • Problem description: I traced the actual runtime consumer of optionRule() rather than assuming it's cosmetic. SeaTunnelConfValidateCommand.execute() (seatunnel-core/seatunnel-starter/.../command/SeaTunnelConfValidateCommand.java:176-192) runs validatePluginConfigs(sinkConfigs, TableSinkFactory.class, sinkClassLoader) unconditionally — for both --dry-run check (static analysis) and --dry-run connect (connectivity check; per the project's own docs, --dry-run connect "runs the static checks first"). That method calls ConfigValidator.validateUnknownKeys(readonlyConfig, factory.optionRule(), pluginName) (SeaTunnelConfValidateCommand.java:325), whose own Javadoc states: "Validates that all user-provided config keys are declared in the connector's OptionRule. Unknown keys are treated as errors to catch typos and misconfigured options early." I read ConfigValidator.validateUnknownKeys directly (seatunnel-api/.../configuration/util/ConfigValidator.java:82-100): it collects every key from rule.getRequiredOptions() + rule.getOptionalOptions() and throws OptionValidationException on anything else present in the user's config. Since user_id_field is in neither list, a config using it — including the connector's own documented example in docs/en/connectors/sink/Mem0.md (user_id_field = "user_id") — deterministically fails with Connector 'Mem0' has unknown option keys: [user_id_field]. Please check for typos. when run through --dry-run check or --dry-run connect.
    I also confirmed this does not affect ordinary job submission/execution: the real Zeta execution path (MultipleTableJobConfigParser -> FactoryUtil.createAndPrepareSink -> ConfigValidator.of(context.getOptions()).validate(factory.optionRule())) only checks that required options are present, and never calls validateUnknownKeys; and ReadonlyConfig.get() on a no-default option (get(option) = getOptional(option).orElseGet(option::defaultValue), and defaultValue() is null for .noDefaultValue()) returns null rather than throwing, so Mem0Sink's constructor itself doesn't crash. So a real job with user_id_field set still runs correctly end to end — this is specifically a --dry-run/config-validate-command bug, not a job-execution bug.
  • Potential risk: This breaks a real, actively-developed, project-endorsed CLI workflow (--dry-run check/--dry-run connect, the exact feature area PR #12242 elsewhere in this batch is extending to a third connector) for the exact example this PR's own docs tell users to copy. A user following the Mem0 doc page verbatim and then running the recommended pre-flight check before submitting their job will get a confusing "unknown option key" error on a field the docs explicitly told them to set. This is the same class of "half-registered" completeness gap my own previous review flagged as a blocker for the missing plugin-mapping.properties entry (Issue 4 last round) — the option is real and functional, but invisible to the schema that gates the project's own validation tooling.
  • Best improvement: Add Mem0SinkOptions.USER_ID_FIELD to the .optional(...) list in Mem0SinkFactory.optionRule(), alongside AGENT_ID_FIELD/APP_ID_FIELD/RUN_ID_FIELD (it should not be .required(...), since the connector's own design — correctly — allows any one of the four scope fields to satisfy the "at least one" requirement, enforced per-row in Mem0SinkWriter.write()).
  • Severity: High (deterministically breaks a supported CLI validation command for the connector's own documented example; does not affect ordinary job execution).
  • Raised by another reviewer: No.

This is a small, mechanical, one-line fix (moving USER_ID_FIELD into the existing .optional(...) call), so I don't expect it to hold this PR up long — but it does need a new commit before merge, same as the packaging/naming/doc items from last round.

1.2 Compatibility Impact

Fully compatible. This is a new connector with no prior released version; there is no historical behavior to break, no checkpoint/savepoint state, and no serialized split format. Issue 1 above is a schema-completeness bug in the new connector's own tooling surface, not a compatibility regression.

1.3 Performance / Side-Effect Analysis

Unchanged from my last review: bounded retries, blocking TimeUnit.MILLISECONDS.sleep on the writer's own thread (consistent with every other HttpClientProvider-based connector), no new unbounded collections or thread creation.

Issue 2 (carried forward from last review, still open, still non-blocking): Mem0SinkWriter's own status-code retry loop (send()) and the base HttpClientProvider's built-in connection-level retryer both key off the same retry/backoff config for different failure classes (HTTP status vs. IOException), so they stack rather than compose predictably under a persistent connection issue. Not fixed in this round, and I'm not asking it to be — it's low-risk and orthogonal to what changed. Location: Mem0SinkWriter.java send(). Severity: Low.

1.4 Error Handling and Logging

Unchanged and still correct: IOException is preserved rather than double-wrapped, waitBeforeRetry() correctly restores the interrupt flag, and no credential material (the api_key embedded in the Authorization header) is logged anywhere.

2. Code Quality Assessment

2.1 Coding Standards

Issue 3 (new, Low): PR title and commit messages don't follow the project's [Type][Module] Description convention. The PR title is currently feat(connector): add Mem0 Platform V3 ADD sink, and the individual commits use the same Conventional-Commits-style prefixes (feat(connector):, fix(connector):, style(connector):, docs(connector):, test(mem0):). This repo's contribution guide asks for [Type][Module] Description, e.g. [Feature][Connector-V2] Add Mem0 Platform V3 ADD sink connector (which is, encouragingly, exactly the wording already used in the new changelog entries in docs/en/connectors/changelog/connector-mem0.md and docs/zh/connectors/changelog/connector-mem0.md — so the right title text already exists, just not in the PR title itself). This is squash-mergeable either way since the final commit message is normally set at merge time by whoever merges it, so I'm flagging it as a low-severity, easy-to-fix note rather than a blocker. Severity: Low. Raised by another reviewer: No.

Everything else here is unchanged from last round and still holds: the Options/Sink/SinkFactory/SinkWriter split matches the rest of the codebase, and the class/field-level documentation is adequate for this connector's size (Mem0SinkOptions fields are documented via .withDescription(...), matching the established pattern in HttpCommonOptions).

2.2 Test Coverage and Test Stability

Unchanged from my last review — still Stable. The 5 tests in Mem0SinkWriterTest are deterministic (fixed Mockito stub sequences, no Thread.sleep-based waits in the test code itself, no shared static state, no ordering dependence). The one thing that changed is the retry-test's backoff values (1/1 instead of the previous 0/0), which — as covered above — was a required fix for a Guava constructor precondition, not a stability regression; if anything it makes the test more clearly a deliberate, deterministic timing choice rather than an accidental zero.

Coverage gaps I noted last round are still open and still non-blocking: no Mem0Sink/Mem0SinkFactory-level test exercising OptionRule completeness. Had such a test existed, it likely would have caught Issue 1 above directly (many other connectors' *FactoryTest classes in this codebase explicitly assert against ConfigValidator.validateUnknownKeys) — worth adding along with the Issue 1 fix.

2.3 Documentation Updates

Fixed from last round (doc-page format, Issue 5) and still accurate content-wise on both docs/en and docs/zh pages.

3. Architectural Soundness

3.1 Elegance of the Solution

Still a precise fix for what it sets out to do, per my last review. Issue 1 is a schema-registration gap, not a design problem.

3.2 Maintainability

Good, now that the naming/header/doc conventions are all correct.

3.3 Extensibility

Unchanged assessment from last round — a future Phase 2 (delete, self-hosted OSS, event polling) has a clear place to land.

3.4 Historical-Version Compatibility

No checkpoint/savepoint state, no serialized split format — unchanged from last review.

4. Issue Summary

Number Issue Location Severity
1 user_id_field missing from Mem0SinkFactory.optionRule(), breaking --dry-run check/--dry-run connect for the connector's own documented example Mem0SinkFactory.java, Mem0SinkOptions.java:203-207, Mem0Sink.java:283 High
2 Connector's own retry loop and base HttpClientProvider's retryer both consume the same retry/backoff config for different failure classes (carried forward, unaddressed, non-blocking) Mem0SinkWriter.java send() Low
3 PR title / commit messages don't follow [Type][Module] Description convention PR title, commit history Low

5. Merge Recommendation

Conclusion: Ready to merge after fixes

  1. Blockers — must be fixed

    • Issue 1 (High): add Mem0SinkOptions.USER_ID_FIELD to the .optional(...) list in Mem0SinkFactory.optionRule(). This is a one-line fix; I'd suggest also adding a quick Mem0SinkFactoryTest asserting ConfigValidator.validateUnknownKeys passes for a config using all four scope fields (one at a time), so this class of gap can't silently reappear.
  2. Recommended fixes — non-blocking

    • Issue 2: document or resolve the retry-layering ambiguity between the connector's own status-code retries and the base client's connection-level retryer.
    • Issue 3: consider using [Feature][Connector-V2] ... for the PR title (the changelog entries already use the right wording) — cosmetic, no action needed before merge if a maintainer prefers to just retitle at merge time.

Again, sorry for not catching Issue 1 in the first pass — it required tracing through ConfigValidator/SeaTunnelConfValidateCommand rather than just reading the connector's own files, which I should have done given how central optionRule() completeness is to this project's tooling. Everything else from my previous review is now correctly and cleanly fixed: the connector-amazondocumentdb regression is gone, license headers and naming match convention, plugin-mapping.properties is registered, and the docs follow the standard format. Once Issue 1 is in, I don't see anything else standing in the way of merge.

@CryoThrust

Copy link
Copy Markdown
Contributor Author

Addressed the new review finding in signed commit 1552041af: registered Mem0SinkOptions.USER_ID_FIELD in Mem0SinkFactory.optionRule() as an optional field. This keeps the documented user_id_field configuration visible to dry-run/config validation while preserving the per-row requirement that any one scope field may be used. Spotless passes for the module.

@CryoThrust

Copy link
Copy Markdown
Contributor Author

The repository Build exposed a test-fixture-only failure on all JDK/OS unit-test jobs: Guava Retryer requires retryBackoffMultiplierMillis < retryBackoffMaxMillis, while the test used 1 and 1. I pushed signed commit 3a7280c0e to set the test max backoff to 2 ms. Production retry behavior is unchanged; this keeps the test deterministic and satisfies the retryer precondition.

@CryoThrust

Copy link
Copy Markdown
Contributor Author

The full Build found one integration contract issue in ConnectorSpecificationCheckTest: the new Mem0Sink declared SupportMultiTableSink but its writer is a regular AbstractSinkWriter, so the checker rejected the factory. Since Mem0 is row-oriented and does not implement multi-table writer semantics, I removed the unsupported marker in signed commit 8a5d858fd. The focused module test remains green (5/5); no production write behavior changed.

@DanielLeens DanielLeens left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What Problem Does This PR Solve?

This adds a new connector-mem0 sink under connector-http that writes SeaTunnel rows to the hosted Mem0 Platform V3 asynchronous add API (POST /v3/memories/add/), acknowledging a row only when the response contains a non-empty event_id. Scope is explicitly limited to ADD for the hosted profile. Closes #12102.

This is my third pass on this PR. I've done a complete, from-scratch re-review of the entire current diff and runtime call chain (not just the delta since my last review), per our process for re-review after a new commit lands.

Recap of where things stood: my first review found 5 blocking issues (an unrelated connector-amazondocumentdb dist-packaging deletion, missing license headers, an Options-class naming violation, a missing plugin-mapping.properties entry, and a doc-format violation) — all fixed in the next round. My second review found one more High-severity gap (user_id_field missing from Mem0SinkFactory.optionRule(), breaking --dry-run check/--dry-run connect) — I want to note again that I should have caught that the first time around, since it required tracing ConfigValidator/SeaTunnelConfValidateCommand rather than just reading the connector's own files.

What changed since my last review: one new commit, 8a5d858fd4 ("fix(connector): remove unsupported multi-table marker"). I pulled the exact diff of that commit directly rather than trusting the commit message:

-import org.apache.seatunnel.api.sink.SupportMultiTableSink;
...
-public class Mem0Sink extends AbstractSimpleSink<SeaTunnelRow, Void>
-        implements SupportMultiTableSink {
+public class Mem0Sink extends AbstractSimpleSink<SeaTunnelRow, Void> {

This was in direct response to the full Build run catching a real integration-contract violation via ConnectorSpecificationCheckTest (seatunnel-dist/src/test/java/.../ConnectorSpecificationCheckTest.java): that test's checkSupportMultiTableSink() asserts that any sink implementing SupportMultiTableSink must have a createWriter() return type assignable to SupportMultiTableSinkWriter. I read Mem0SinkWriter directly — it extends plain AbstractSinkWriter<SeaTunnelRow, Void>, not SupportMultiTableSinkWriter — so the contract was genuinely violated, and removing the marker interface (rather than retrofitting multi-table-writer semantics onto a row-oriented connector that doesn't need them) is the correct fix, not a workaround.

1. Code Change Review

1.1 Core Logic Analysis

The write path is unchanged from my last two reviews and still reads correctly end to end:

Mem0SinkFactory.createSink() -> Mem0Sink(ReadonlyConfig, CatalogTable)
  resolves messages/user_id/agent_id/app_id/run_id/metadata field names,
  builds HttpParameter with url = base_url + "/v3/memories/add/"
Mem0Sink.createWriter() -> Mem0SinkWriter
  write(row): serialize row to JSON -> require non-empty "messages" array,
              at least one of the four scope fields non-empty -> send(request)
    send(): httpClient.doPost() -> on 2xx, require non-empty "event_id";
             on 408/429/5xx and attempt < maxRetries, backoff and retry;
             otherwise throw IOException

New finding this round — a real, previously-latent contract gap surfaced by the SupportMultiTableSink removal:

Issue 1: SinkConnectorCommonOptions.MULTI_TABLE_SINK_REPLICA is still declared in Mem0SinkFactory.optionRule() even though Mem0Sink no longer implements SupportMultiTableSink

  • Location: Mem0SinkFactory.java optionRule() (still lists SinkConnectorCommonOptions.MULTI_TABLE_SINK_REPLICA in .optional(...), unchanged by commit 8a5d858fd4, which only touched Mem0Sink.java).
  • Problem description: I traced the actual runtime consumer of this option rather than assuming it's cosmetic. MULTI_TABLE_SINK_REPLICA is read in exactly one place at runtime: MultiTableSink (seatunnel-api/.../sink/multitablesink/MultiTableSink.java:92), the wrapper class the engine substitutes in place of a connector's own sink when it participates in multi-table sink merging. That substitution is gated in MultipleTableJobConfigParser.java:727 on action.getSink() instanceof SupportMultiTableSink. Since Mem0Sink no longer implements that marker (correctly, per Issue removed above), MultiTableSink is never constructed for this connector, and MULTI_TABLE_SINK_REPLICA is consequently never read for any Mem0 job. Declaring it as a valid optional key is now pure dead weight: it doesn't do anything, it isn't referenced anywhere in Mem0Sink/Mem0SinkWriter, and it wasn't there because of a copy-paste — it was live and meaningful right up until the previous commit removed the interface that made it relevant.
  • Potential risk: Low functionally (declaring an unused optional key doesn't fail --dry-run check/--dry-run connect, since validateUnknownKeys only rejects undeclared keys, not declared-but-unused ones, and it isn't required, so omitting it is also fine). The real risk is user-facing confusion: a user reading the generated option-rule/schema output (or ConnectorHelpCommand-style tooling that lists a factory's option rule) would see multi_table_sink_replica listed as a legitimate Mem0 tuning knob, set it expecting sink-replica fan-out behavior, and get silent no-op behavior instead — this is the same "looks configurable, does nothing" class of gap this project's review standards call out for registration completeness, just in the opposite direction (an over-registered rather than under-registered option).
  • Best improvement: remove SinkConnectorCommonOptions.MULTI_TABLE_SINK_REPLICA from the .optional(...) list in Mem0SinkFactory.optionRule() now that Mem0Sink doesn't implement SupportMultiTableSink. This is a one-line follow-up to the same commit that already correctly removed the marker interface.
  • Severity: Medium (not a crash or data-correctness bug, but a real, currently-shippable option-schema/behavior mismatch introduced by the very last commit).
  • Raised by another reviewer: No.

Everything else in the write/validate/retry path is unchanged and still correct: required()/copyIfPresent() treat blank strings the same as absent, validateAcceptedResponse() distinguishes "HTTP 2xx but no real event_id" from a true failure, and normalizeBaseUrl() avoids a double slash.

Separately, I independently confirmed the user_id_field fix from my last review is correctly and completely in place: Mem0SinkFactory.optionRule()'s .optional(...) list now includes Mem0SinkOptions.USER_ID_FIELD alongside AGENT_ID_FIELD/APP_ID_FIELD/RUN_ID_FIELD, matching the field's actual consumption in Mem0Sink's constructor.

1.2 Compatibility Impact

Fully compatible. New connector, no prior released version, no checkpoint/savepoint state, no serialized split format. Issue 1 above is a schema-completeness/cleanliness gap in the new connector's own option surface, not a compatibility regression against anything else.

1.3 Performance / Side-Effect Analysis

Unchanged from my prior reviews: bounded retries, blocking TimeUnit.MILLISECONDS.sleep on the writer's own thread (consistent with every other HttpClientProvider-based connector in this codebase), no new unbounded collections or thread creation.

Issue 2 (carried forward, still open, still non-blocking): Mem0SinkWriter's own status-code retry loop (send()) and the base HttpClientProvider's connection-level retryer both key off the same retry/backoff config for different failure classes (HTTP status vs. IOException), so they stack rather than compose predictably under a persistent connection issue. Not addressed in this round, and I'm not asking it to be — it remains low-risk and orthogonal to what changed. Location: Mem0SinkWriter.java send(). Severity: Low.

1.4 Error Handling and Logging

Unchanged and still correct: IOException is preserved rather than double-wrapped, waitBeforeRetry() correctly restores the interrupt flag, and no credential material (the api_key embedded in the Authorization header) is logged anywhere in the diff.

2. Code Quality Assessment

2.1 Coding Standards

Issue 3 (new, Low): the new module directory/artifactId breaks the established connector-http-<vendor> naming convention for sub-modules of connector-http

  • Location: seatunnel-connectors-v2/connector-http/connector-mem0 (directory and Maven artifactId); seatunnel-connectors-v2/connector-http/pom.xml's <modules> list.
  • Problem description: I listed every sibling module under connector-http on the current head: connector-http-airtable, connector-http-base, connector-http-feishu, connector-http-github, connector-http-gitlab, connector-http-jira, connector-http-klaviyo, connector-http-lemlist, connector-http-myhours, connector-http-notion, connector-http-onesignal, connector-http-persistiq, connector-http-posthog, connector-http-shopify, connector-http-stripe, connector-http-wechat, connector-http-zendesk — all 17 existing HTTP-family connectors follow the connector-http-<vendor> pattern. This PR's new module is named connector-mem0, the sole exception. Everything that references it (plugin-mapping.properties's seatunnel.sink.Mem0 = connector-mem0, seatunnel-dist/pom.xml's new dependency, connector-http/pom.xml's module entry) is internally consistent with that name, so nothing is actually broken or half-registered — this is purely a project-convention deviation, not a functional defect.
  • Best improvement: rename the module directory and artifactId to connector-http-mem0 for consistency with its 17 siblings, updating the three references above accordingly. Non-blocking given it's cosmetic and mechanical, but worth doing before merge since renaming gets harder once released.
  • Severity: Low.
  • Raised by another reviewer: No.

Issue 4 (carried forward from last review, unaddressed, Low): PR title / commit messages still don't follow [Type][Module] Description

The PR title remains feat(connector): add Mem0 Platform V3 ADD sink, and commits still use Conventional-Commits-style prefixes (feat(connector):, fix(connector):, etc.) rather than this project's [Type][Module] Description convention (e.g. [Feature][Connector-V2] Add Mem0 Platform V3 ADD sink connector, which is, encouragingly, already the exact wording used in the changelog entries). Squash-mergeable either way since the final commit message is normally set at merge time. Severity: Low. Raised by another reviewer: No.

Everything else here is unchanged and still holds: the Options/Sink/SinkFactory/SinkWriter split matches the rest of the codebase, and field-level documentation via .withDescription(...) is adequate.

2.2 Test Coverage and Test Stability

Unchanged from my last review — still Stable. The 5 tests in Mem0SinkWriterTest are deterministic (fixed Mockito stub sequences, no Thread.sleep-based waits in the test code itself, no shared static state, no ordering dependence). I re-confirmed none of the currently-open issues (1, 2, 3, 4 above) are the kind a unit test would typically catch — Issue 1 (orphaned option) would only be caught by a Mem0SinkFactoryTest-level test asserting on optionRule() contents, which still doesn't exist (a coverage gap I flagged as non-blocking last round and which remains open).

2.3 Documentation Updates

Unchanged and still accurate on both docs/en and docs/zh — I re-read both pages fresh and confirmed they remain substantively equivalent and describe the ADD-only scope, scope-field requirement, and event_id semantics matching the actual code.

3. Architectural Soundness

3.1 Elegance of the Solution

Still a precise fix for what it sets out to do. Removing SupportMultiTableSink rather than retrofitting multi-table-writer semantics onto a connector that doesn't need them is the right call — Issue 1 is simply the one-line follow-up cleanup that commit should have included alongside it.

3.2 Maintainability

Good, pending the Issue 1 cleanup and (non-blocking) Issue 3 rename.

3.3 Extensibility

Unchanged assessment — a future Phase 2 (delete, self-hosted OSS, event polling) has a clear place to land.

3.4 Historical-Version Compatibility

No checkpoint/savepoint state, no serialized split format — unchanged from prior reviews.

4. Issue Summary

# Issue Location Severity Raised by another reviewer
1 MULTI_TABLE_SINK_REPLICA still declared in optionRule() after SupportMultiTableSink was removed — now a dead, misleading option Mem0SinkFactory.java optionRule() Medium No
2 Connector's own retry loop and base HttpClientProvider's retryer both consume the same retry/backoff config for different failure classes (carried forward, unaddressed) Mem0SinkWriter.java send() Low No
3 New module breaks connector-http-<vendor> naming convention (connector-mem0 vs. sibling connector-http-*) seatunnel-connectors-v2/connector-http/connector-mem0, connector-http/pom.xml Low No
4 PR title / commit messages don't follow [Type][Module] Description convention (carried forward, unaddressed) PR title, commit history Low No

CI status: the fork's current head (8a5d858fd43a) shows Build conclusion failure on the apache-side check. I dereferenced the actual fork run (CryoThrust/seatunnel run 34606627318) directly: the only failing job is unit-test (8, windows-latest). Its step log shows run all modules unit test (Windows) starting and never reaching a completed_at (job wall-clock ran the full ~49 minutes before being marked failure with no recorded failing test), which is consistent with a hung/timed-out Windows CI runner rather than an actual assertion failure — Mem0SinkWriterTest's own five tests are plain, fast, network-free Mockito tests with nothing that would explain a full-repo Windows hang. I can't rule this out with certainty without a completed log, so I'm not treating it as evidence of a real defect, but I also can't independently confirm it's environmental the way I could for the all-connectors-it/transform-v2-it failures on PR #12269 elsewhere in this batch (no comparable concurrent-PR cross-reference available for this specific job/run). This should be re-run and re-checked rather than assumed clean.

5. Merge Recommendation

Conclusion: Ready to merge after fixes

  1. Blockers — must be fixed

    • CI must be green (or the unit-test (8, windows-latest) failure specifically confirmed as an unrelated Windows-runner hang via a clean re-run) before merge.
  2. Recommended fixes — non-blocking, but I'd like to see them in before merge given how small they are

    • Issue 1 (Medium): remove SinkConnectorCommonOptions.MULTI_TABLE_SINK_REPLICA from Mem0SinkFactory.optionRule() now that Mem0Sink no longer implements SupportMultiTableSink — this is a one-line companion to the fix already made in 8a5d858fd4.
    • Issue 2: document or resolve the retry-layering ambiguity between the connector's own status-code retries and the base client's connection-level retryer.
    • Issue 3: consider renaming the module to connector-http-mem0 for consistency with its 17 siblings under connector-http, before this ships in a release and the rename becomes disruptive.
    • Issue 4: consider [Feature][Connector-V2] ... for the PR title — cosmetic, a maintainer can also just retitle at merge time.

Great progress since my last two passes — every previously-blocking issue (the unrelated connector-amazondocumentdb deletion, license headers, naming, plugin-mapping.properties, doc format, and the user_id_field schema gap) is now correctly and verifiably fixed, and the multi-table-sink contract violation caught by CI was fixed the right way rather than papered over. Issue 1 is a small, mechanical loose end left by that same fix, easy to close out alongside a green re-run of CI.

@DanielLeens

Copy link
Copy Markdown
Contributor

Thanks for the continued progress here. GitHub is currently reporting merge conflicts between this branch and the latest dev. Could you please sync with the latest dev and resolve the conflicts first? Once that's done and the checks are rerun, I'm happy to take another look.

@CryoThrust

Copy link
Copy Markdown
Contributor Author

Synced the PR branch with the latest dev and resolved the merge conflict in seatunnel-connectors-v2/connector-http/pom.xml, preserving both the existing connector-http-linear module and the new Mem0 module.

Also removed the now-unused MULTI_TABLE_SINK_REPLICA option from Mem0SinkFactory.optionRule() because the connector no longer implements SupportMultiTableSink. The rebased branch was force-pushed to the fork as signed-off commits; the new head is fde7b6894.

Please rerun the standard PR checks when available. The prior Windows/JDK 8 failure remained stuck in the test step without a recorded test failure, so I will treat the fresh run as the authoritative result.

@CryoThrust

Copy link
Copy Markdown
Contributor Author

The fresh Build run exposed one remaining registration gap: config/plugin_config did not list connector-mem0, so install-plugin.sh could not download the connector. I added the missing connector-mem0 entry in signed commit 40a7b733d and pushed the updated rebased branch to feat/mem0-v3-add. This is the only failure in the run; license, docs, changes, and other static checks passed.

@DanielLeens DanielLeens left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What Problem Does This PR Solve?

This adds a new connector-mem0 sink under connector-http that writes SeaTunnel rows to the hosted Mem0 Platform V3 asynchronous add API (POST /v3/memories/add/), acknowledging a row only when the response contains a non-empty event_id. Scope is explicitly limited to ADD for the hosted profile. Closes #12102.

This is my fourth pass on this PR, and per our re-review process I've done a complete, from-scratch re-review of the entire current diff and runtime call chain rather than just the delta.

First, closing out the pending thread from my last review comment: I had asked you to sync with the latest dev and resolve the reported merge conflict before I looked again. You replied that you did so (resolving the conflict in connector-http/pom.xml and also removing the now-dead MULTI_TABLE_SINK_REPLICA option), then found and fixed one more registration gap (config/plugin_config missing the connector-mem0 entry) that the fresh CI run surfaced. I independently verified both, not just the commit messages:

  • connector-http/pom.xml at the current head still lists <module>connector-http-linear</module> (untouched, correctly preserved) alongside the new <module>connector-mem0</module> — the conflict resolution is clean, nothing else was dropped.
  • Mem0SinkFactory.optionRule() no longer references SinkConnectorCommonOptions.MULTI_TABLE_SINK_REPLICA — this closes Issue 1 from my previous (third) review.
  • config/plugin_config now contains a connector-mem0 line (verified directly against the file content at the current head), closing the gap the fresh CI run caught.
  • git mergeable_state for the PR is currently blocked with mergeable: true — that's the standard "branch protection / review required" state, not an actual code conflict. The conflict you resolved is gone.

Thanks for turning both of these around quickly and for tracing the CI-caught gap yourself rather than waiting on me to point it out.

Recap of where things stood across my first three passes, all independently re-verified as still correctly fixed at the current head (40a7b733d):

  • Pass 1: unrelated connector-amazondocumentdb dist-packaging deletion, missing license headers, Options-class naming violation, missing plugin-mapping.properties entry, doc-format violation — all fixed.
  • Pass 2: user_id_field missing from Mem0SinkFactory.optionRule() (broke --dry-run check/--dry-run connect) — fixed, confirmed USER_ID_FIELD is now in the .optional(...) list.
  • Pass 3: MULTI_TABLE_SINK_REPLICA left dangling in optionRule() after SupportMultiTableSink removal — fixed this round, per above.

1. Code Change Review

1.1 Core Logic Analysis

The write path is unchanged from all three prior reviews and still reads correctly end to end (re-verified against the current head's actual file contents, not the diff hunks):

Mem0SinkFactory.createSink() -> Mem0Sink(ReadonlyConfig, CatalogTable)
  resolves messages/user_id/agent_id/app_id/run_id/metadata field names,
  builds HttpParameter with url = normalizeBaseUrl(base_url) + "/v3/memories/add/"
Mem0Sink.createWriter() -> Mem0SinkWriter
  write(row): serialize row to JSON -> require non-empty "messages" array,
              at least one of the four scope fields non-empty -> send(request)
    send(): httpClient.doPost() -> on 2xx, require non-empty "event_id";
             on 408/429/5xx and attempt < maxRetries, backoff and retry;
             otherwise throw IOException

This is the normal write path for every row (a plain AbstractSimpleSink/AbstractSinkWriter, no split/checkpoint state), not a boundary or recovery path.

I did a fresh line-by-line read of Mem0Sink.java, Mem0SinkWriter.java, and Mem0SinkOptions.java at the current head rather than trusting my prior notes, and everything still holds: required()/copyIfPresent() treat blank/whitespace-only strings the same as absent, validateAcceptedResponse() correctly distinguishes "HTTP 2xx but no real event_id" from a true failure, normalizeBaseUrl() avoids a double slash, and RETRY/RETRY_BACKOFF_MULTIPLIER_MS/RETRY_BACKOFF_MAX_MS referenced from Mem0SinkFactory resolve correctly to the inherited HttpCommonOptions fields (RETRY has noDefaultValue(), and Mem0Sink's constructor correctly uses getOptional(RETRY).orElse(0) rather than .get(), so a config that omits retry gets 0 retries rather than an NPE).

No new logic issues found this round.

1.2 Compatibility Impact

Fully compatible. New connector, no prior released version, no checkpoint/savepoint state, no serialized split format, no protocol field shared with any other connector. Nothing in this PR's actual diff touches a pre-existing file's behavior — the only pre-existing files touched (seatunnel-dist/pom.xml, connector-http/pom.xml, plugin-mapping.properties, config/plugin_config) each receive a single, purely additive line, independently re-confirmed against current dev.

1.3 Performance / Side-Effect Analysis

Unchanged from my prior reviews: bounded retries, blocking TimeUnit.MILLISECONDS.sleep on the writer's own thread (consistent with every other HttpClientProvider-based connector in this codebase), no new unbounded collections or thread creation, close() correctly releases the underlying HttpClientProvider.

Issue 1 (carried forward from my first review, still open, still non-blocking): Mem0SinkWriter's own status-code retry loop (send()) and the base HttpClientProvider's connection-level retryer both key off the same retry/backoff config for different failure classes (HTTP status vs. IOException), so they stack rather than compose predictably under a persistent connection issue. Still not addressed, still low-risk and orthogonal. Location: Mem0SinkWriter.java send(). Severity: Low.

1.4 Error Handling and Logging

Unchanged and still correct: IOException is preserved rather than double-wrapped in send(), waitBeforeRetry() correctly restores the interrupt flag before converting InterruptedException to IOException, and no credential material (the api_key embedded in the Authorization header) is logged anywhere in the current diff.

2. Code Quality Assessment

2.1 Coding Standards

Issue 2 (carried forward from my third review, unaddressed, Low): module directory/artifactId still doesn't follow the connector-http-<vendor> convention used by all 17 sibling modules (connector-http-airtable, connector-http-feishu, ..., connector-http-zendesk). This PR's module is still connector-mem0 rather than connector-http-mem0. Everything referencing it (plugin-mapping.properties, config/plugin_config, seatunnel-dist/pom.xml, connector-http/pom.xml's module list) is internally consistent with the current name, so nothing is functionally broken — purely a naming-convention deviation, easier to fix now than after a release. Severity: Low.

Issue 3 (carried forward from my second review, unaddressed, Low): PR title / commit messages still don't follow [Type][Module] Description. Title remains feat(connector): add Mem0 Platform V3 ADD sink; commit history still uses Conventional-Commits-style prefixes. The changelog entries already use the correct wording ([Feature][Connector-V2] Add Mem0 Platform V3 ADD sink connector), so this is squash-mergeable either way — a maintainer can retitle at merge time. Severity: Low.

All three previously-blocking coding-standard issues (license headers on the 6 new files, Mem0Options -> Mem0SinkOptions rename, Mem0SinkFactory.optionRule() schema completeness) remain correctly fixed on re-inspection of the current file contents. The Options/Sink/SinkFactory/SinkWriter split still matches the rest of the codebase, and field-level documentation via .withDescription(...) is adequate for a connector this size.

2.2 Test Coverage and Test Stability

Unchanged from my last review — still Stable. I re-read Mem0SinkWriterTest.java fresh at the current head: 5 deterministic Mockito-based tests (happy-path payload/event_id round-trip, rejecting a 2xx response without event_id, retry-then-succeed with an exact times(3) call-count assertion, a static-method table test for isRetryableStatus, and a fallback-to-secondary-scope-field test). No Thread.sleep-based waits in the test code itself (the retry test uses retryBackoffMultiplierMillis=1/retryBackoffMaxMillis=2, both fixed, tiny, deterministic values — not flaky timing), no shared static state, no ordering dependence, no environment/OS-specific assumptions.

Coverage gaps flagged in my earlier reviews are still open and still non-blocking: no Mem0SinkFactory-level test asserting on optionRule() completeness (which would have caught both the user_id_field and MULTI_TABLE_SINK_REPLICA gaps directly), and no test exercising normalizeBaseUrl() or the "no scope field at all" rejection path at the Mem0Sink level. Given the connector's small surface, these remain recommended follow-ups rather than blockers.

2.3 Documentation Updates

Unchanged and still accurate — I re-read both docs/en/connectors/sink/Mem0.md and docs/zh/connectors/sink/Mem0.md fresh at the current head. Both start with the import ChangeLog from '../changelog/connector-mem0.md'; line followed by the # Mem0 heading (matching the project's doc-page convention), end with <ChangeLog />, and are substantively equivalent between languages. The documented example (api_key, messages_field, user_id_field, api_base_url) matches the actual, now-fully-registered Mem0SinkOptions fields exactly — this is what makes the doc's example actually usable end to end, including through --dry-run check, now that user_id_field is properly declared.

3. Architectural Soundness

3.1 Elegance of the Solution

Still a precise fix for what it sets out to do — reusing connector-http-base's HttpParameter/HttpClientProvider rather than a hand-rolled client, and scoping tightly to ADD-only. Every issue found across all four passes has been in packaging/registration/naming completeness, not core connector logic.

3.2 Maintainability

Good. All naming/header/doc/registration issues found in my first three reviews are now correctly resolved; only the two Low-severity, purely-cosmetic items above remain.

3.3 Extensibility

Unchanged assessment — a future Phase 2 (delete, self-hosted OSS profile, event polling) has a clear place to land without touching the ADD-only path added here.

3.4 Historical-Version Compatibility

No checkpoint/savepoint state, no serialized split format, no protocol field shared with any existing connector — unchanged from all prior reviews.

4. Issue Summary

# Issue Location Severity Raised by another reviewer
1 Connector's own retry loop and base HttpClientProvider's retryer both consume the same retry/backoff config for different failure classes (carried forward, unaddressed) Mem0SinkWriter.java send() Low No
2 Module still named connector-mem0 instead of the connector-http-<vendor> convention used by all 17 sibling modules (carried forward, unaddressed) seatunnel-connectors-v2/connector-http/connector-mem0, connector-http/pom.xml Low No
3 PR title / commit messages don't follow [Type][Module] Description convention (carried forward, unaddressed) PR title, commit history Low No

CI status: re-confirmed live against the fork's actual run for the current head (40a7b733d, fork run 34740087728), not just the apache-side pointer. Every real job is green: License header, Code style, Sanity check results, Dead links, Build website, unit-test on all four OS/JDK combinations (8/11 x ubuntu/windows), and both updated-modules-integration-test parts on both JDKs. The prior Windows unit-test hang I flagged as unresolved in my last review is gone on this fresh run — it was indeed the environmental/runner issue I suspected, not a real defect. No skipped job here represents a coverage gap for this PR's own scope.

5. Merge Recommendation

Conclusion: Ready to merge

  1. Blockers — must be fixed

    None. Every High-severity item found across my four passes (the unrelated connector-amazondocumentdb deletion, license headers, Options naming, plugin-mapping.properties, doc format, user_id_field schema gap) is verifiably fixed at the current head, and CI is fully green.

  2. Recommended fixes — non-blocking

    • Issue 1: document or resolve the retry-layering ambiguity between the connector's own status-code retries and the base client's connection-level retryer.
    • Issue 2: consider renaming the module to connector-http-mem0 for consistency with its 17 siblings under connector-http — worth doing before this ships in a release, since renaming gets more disruptive afterward.
    • Issue 3: consider [Feature][Connector-V2] ... for the PR title (the changelog entries already use the right wording) — cosmetic, a maintainer can also just retitle at merge time.

This has been a genuinely productive back-and-forth: every blocking issue across four rounds of review (packaging regression, license/naming/doc conventions, a schema-completeness gap, a stale option left after a contract fix, and the merge-conflict/plugin_config registration gap from this round) was fixed promptly and correctly, and verified independently at each step rather than taken on faith. I don't see anything left standing in the way of merge. If I had approve/merge rights on this repo I would mark this "Ready to merge" as a formal review state; posting as a comment in case that's still the case, per my account's usual permission level here.

@davidzollo davidzollo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
LGTM

Thanks for your contribution! This is a solid addition to the project, and your attention to detail really shows. Contributors like you are what keep this community moving forward. Feel free to keep the PRs coming — we're always glad to have you here!

Please ignore the following message if you have joined in the Slack~

To facilitate focused communication among contributors, we’ve created a brand-new dedicated #contributors channel in official Slack workspace.
Slack workspace link: https://s.apache.org/seatunnel-slack

This channel is designed for technical discussions, feature design brainstorming, roadmap sync and direct communication with project core Contributors. Everyone who contributes code, docs, test cases or community ideas is warmly invited to join, share thoughts and collaborate together under The Apache Way. Welcome to join 😄

@davidzollo
davidzollo added this pull request to the merge queue Sep 13, 2026
Merged via the queue into apache:dev with commit 7b88e18 Sep 13, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[STIP-38][Feature][Connector-V2] Add a Mem0-compatible agent memory sink

3 participants