Skip to content

[Improve][Core] Add reusable config validation result - #12065

Merged
davidzollo merged 6 commits into
apache:devfrom
CryoThrust:codex/seatunnel-10651-validation-model
Sep 4, 2026
Merged

davidzollo merged 6 commits into
apache:devfrom
CryoThrust:codex/seatunnel-10651-validation-model

Conversation

@CryoThrust

Copy link
Copy Markdown
Contributor

Purpose

Add a reusable, config-level validation result model for SeaTunnel integrations such as the AI CLI, while preserving the existing --check human-readable output and exit behavior.

Changes

  • add versioned ConfigValidationResult and ConfigValidationError models
  • expose SeaTunnelConfValidateCommand.validateResult() for future CLI, MCP, and REST adapters
  • include stable fields for validity, phase, location, plugin, option path, rule category, and sanitized message
  • classify representative parse, option, plugin, and validation failures without changing the existing command contract
  • add focused serialization and command integration tests

This change intentionally does not introduce a transport-specific JSON flag, bind an LLM provider, or claim runtime-equivalent validation.

Tests

  • ConfigValidationResultTest
  • SeaTunnelConfValidateCommandTest

Local Maven execution was blocked before compilation by unavailable internal mirror dependencies (software.amazon.awssdk:bom, com.google.cloud:libraries-bom, and os-maven-plugin).

Closes #10651

@davidzollo davidzollo added the First-time contributor First-time contributor label Sep 3, 2026

@DanielLeens DanielLeens left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What Problem Does This PR Solve?

  • User pain point: per the linked design issue (#10651), SeaTunnel wants to let AI generate task config files and validate them through an official path, but today seatunnel.sh --check/--dry-run only produces plain, unstructured exception text — there is no reusable, machine-readable validation result that a future CLI JSON flag, an MCP tool, or a REST adapter could consume without screen-scraping human-oriented error strings.
  • Fix approach: adds two new value classes (ConfigValidationResult, ConfigValidationError) plus a new SeaTunnelConfValidateCommand.validateResult() method that runs the existing execute() validation, catches the resulting ConfigCheckException, and repackages it into a small versioned model with valid/phase/errors[], where each error carries location/plugin/optionPath/ruleCategory/message. ruleCategory and location/plugin are derived by pattern-matching the free-text exception message after the fact, since no structured classification exists in the underlying validation code today.
  • One-sentence summary: this is a well-motivated, low-risk, purely additive foundation stone for a larger AI-config-generation roadmap — execute() itself and the existing --check CLI output are untouched, so there is no compatibility risk — but the new model does not yet deliver on two of its own five headline fields (location, plugin) for the default/primary validation phase, and optionPath is never populated at all, which is worth fixing before any real adapter starts depending on this "stable" schema.

1. Code Change Review

1.1 Core Logic Analysis

Confirming execute() and existing --check behavior are untouched

I diffed SeaTunnelConfValidateCommand.java directly: the entire existing execute() method (SeaTunnelConfValidateCommand.java:97-193) is byte-for-byte unchanged — the diff only adds new imports and appends validateResult() and its private helpers after the method's closing brace. validateResult() (:200-214) calls execute() internally and catches ConfigCheckException, so the pre-existing CLI --check/--dry-run text output and exit-code behavior cannot be affected by this PR; the new method is a pure additive wrapper with no other caller anywhere in the repo (verified via grep -rn "validateResult()" — the only call sites are the five new test methods).

Issue 1 (Medium, new): location and plugin are only ever populated for the untested DryRun.CONNECT phase; optionPath is never populated at all

I traced where the "source[N](Plugin)" location string that toValidationError()'s regex (SeaTunnelConfValidateCommand.java:229-231) is designed to match actually gets produced in the codebase:

  • It is built by DryRunConnectValidator.location()/wrap() (DryRunConnectValidator.java:472-479, pre-existing, unmodified by this PR) — String.format("%s[%d](%s)", pluginType.getType(), configIndex, factoryId), embedded into the wrapped exception's message as location + " failed: " + sanitizedMessage. This code path only runs when clientCommandArgs.getDryRun() == DryRun.CONNECT (execute(), line 171-179, calls new DryRunConnectValidator(...).validate() only in that branch).
  • The static-analysis phase (validatePluginConfigs, ConfigParserUtil.checkGraph, ConfigValidator.validate/validateUnknownKeys — all the code exercised in the default --dry-run=static/--check path) never produces a message containing this pluginType[index](factoryId) shape — I checked every throw new ...Exception(...) in ConfigParserUtil.java and ConfigValidator.java, none embed it. So for the static phase, locationMatcher.find() will always return false, and location/plugin will always come out null.
  • Since all five new tests (testValidationResultForValidConfig, testValidationResultClassifiesMissingPluginAsOptionError, ...ClassifiesParseFailure, ...ClassifiesOptionFailure, ...ClassifiesPluginLoadFailure) build their ClientCommandArgs via buildArgsFromPath/buildArgs, which hard-code "--dry-run", "static" (SeaTunnelConfValidateCommandTest.java:566-569), none of them exercise the one code path (DryRun.CONNECT) where location/plugin extraction could actually succeed. ConfigValidationResultTest's two tests construct ConfigValidationError objects directly with a hand-supplied location string, bypassing toValidationError() entirely. So this PR ships a location/plugin-extraction feature that is functionally dead for its default/primary phase and completely unverified for the one phase where it could theoretically work.
  • Separately, optionPath (the fourth constructor argument passed by toValidationError(), SeaTunnelConfValidateCommand.java:254) is hard-coded to null unconditionally — there is no attempt to extract it from anything. This is a real, avoidable gap: OptionUtil.formatError/formatOptionsError (seatunnel-api/.../OptionUtil.java:99-105, pre-existing) already renders option-validation failures as "option: %s\n type: %s\n constraint: %s", so the option key is sitting right there in a structured, parseable position in the very messages toValidationError() already processes for ruleCategory classification — the field is skippable-but-easy to populate, and the PR's own description explicitly lists "option path" as one of the model's headline fields.
  • Net effect: of the five documented per-error fields, only ruleCategory and message are reliably populated for the default/static validation phase that essentially all current SeaTunnel --check/--dry-run=static users hit; location and plugin require an untested, non-default code path, and optionPath never works at all. This directly undercuts the stated goal in the linked design issue #10651 ("Validation failures are mostly plain exception text and are not machine-readable") for the phase that matters most.
  • Location: seatunnel-core/seatunnel-starter/src/main/java/org/apache/seatunnel/core/starter/seatunnel/command/SeaTunnelConfValidateCommand.java:226-255.

Issue 2 (Medium, new): "sanitized message" is only true for the connect phase, not the (default) static phase

The PR description explicitly lists "sanitized message" as one of the model's stable fields. Tracing the actual sanitization: execute()'s catch block only calls the pre-existing DryRunConnectFailureMessageSanitizer.sanitize(message) (seatunnel-common/.../DryRunConnectFailureMessageSanitizer.java, unmodified) when clientCommandArgs.getDryRun() == DryRun.CONNECT (SeaTunnelConfValidateCommand.java:187-189); the static-phase branch (:191, throw new ConfigCheckException(validationMode + " failed: " + message, e)) never sanitizes. validateResult()/toValidationError() add no sanitization of their own for either phase — they only strip a known prefix and classify. So a ConfigValidationError.message produced from a --dry-run=static failure carries the raw, unsanitized exception text end to end. This is not a new leak (the pre-existing CLI --check text output was already unsanitized for the static phase, and this PR does not change that output), but the PR's own new, structured ConfigValidationResult model — explicitly intended for "future CLI, MCP, and REST adapters" per the PR description, i.e. broader/automated consumption than a human reading their own terminal — inherits that gap silently while advertising "sanitized message" as a general property of the model.

  • Location: seatunnel-core/seatunnel-starter/src/main/java/org/apache/seatunnel/core/starter/seatunnel/command/SeaTunnelConfValidateCommand.java:181-214.

Issue 3 (Low, new): ruleCategory classification is free-text keyword-sniffing over messages from unrelated, unowned call sites

toValidationError() (:238-253) classifies every failure into parse/option/plugin/validation purely by lower-casing the final exception message and checking String#contains against a fixed keyword list, in a fixed check order (parse-keywords first, then option-keywords, then plugin-keywords, else "validation"). The messages being sniffed are produced by several independent subsystems this PR does not own or version together with this classifier: FactoryUtil (factory-discovery messages), ConfigValidator/OptionUtil (option-rule messages), ConfigParserUtil (graph-check messages), ConfigBuilder/Typesafe Config (HOCON parse messages), and JSqlParser's own exception text (SQL syntax messages, :329-332). None of these are contractually tied to this keyword list — a future wording tweak in any of them (e.g. changing "Could not find any factory..." to "Could not find any type for identifier...") would silently reclassify a plugin failure as option (since the option-keyword check, which includes the very broad word "type", runs before the plugin-keyword check), with no compiler error and no test catching it unless a test happens to cover that exact message. Given this field is explicitly meant to be a "stable" contract for external automated consumers, a typed/structured signal (e.g. having the underlying exceptions carry an explicit category, or at minimum an enum instead of four magic strings) would be materially more robust than reverse-engineering intent from prose designed for humans.

  • Location: seatunnel-core/seatunnel-starter/src/main/java/org/apache/seatunnel/core/starter/seatunnel/command/SeaTunnelConfValidateCommand.java:226-255.

1.2 Compatibility Impact

Fully compatible. execute() — the sole existing entry point behind --check/--dry-run=static/--dry-run=connect — is byte-for-byte unmodified by this PR (verified via direct diff read, not just the stat summary), so existing CLI text output and exit-code behavior are unaffected. validateResult() is a brand-new public method with zero other callers in the repository today, so there is no existing behavior it could regress. No config option, default value, protocol, or serialization format is touched. ConfigValidationResult/ConfigValidationError are new, self-contained, Serializable value classes with no interaction with checkpoint/savepoint state.

1.3 Performance / Side-Effect Analysis

All new code runs once per CLI invocation of validateResult() (not currently wired to any hot path or per-record loop, and not called anywhere in production code yet). The regex compilation in toValidationError() (Pattern.compile(...) on every call, :230) is a minor, non-hot-path inefficiency — could be hoisted to a static final Pattern — but at CLI-invocation frequency this is immaterial; not worth a formal issue. No new threads, connections, or unbounded collections.

1.4 Error Handling and Logging

Issue Location Problem Risk Fix Severity Raised by another reviewer
1 seatunnel-core/seatunnel-starter/.../command/SeaTunnelConfValidateCommand.java:226-255 location/plugin extraction only works for the untested DryRun.CONNECT phase and is always null for the default static phase; optionPath is hard-coded null in all cases despite being readily extractable from OptionUtil's existing "option: <key>" message format Medium — no active consumer today, but silently fails to deliver 2-3 of the 5 documented per-error fields for the primary use case, undermining the PR's own stated goal (issue #10651) Extract optionPath from the existing "option: %s" / "options: %s" line in OptionUtil-formatted messages; add a validateResult() test that runs with DryRun.CONNECT to prove location/plugin extraction actually works; document (or fix) that location/plugin are static-phase-null by design if that is intentional Medium No
2 seatunnel-core/seatunnel-starter/.../command/SeaTunnelConfValidateCommand.java:181-214 message is only sanitized (via the pre-existing DryRunConnectFailureMessageSanitizer) for the connect phase; the default static phase carries raw exception text into the new structured model, despite the PR description promising a "sanitized message" field Medium — not a new leak (pre-existing CLI text output was already unsanitized for static phase), but the new model is explicitly designed for broader/automated consumption and its own field name overstates the guarantee Apply the same sanitizer (or a static-phase-appropriate equivalent) to the static-phase message before building ConfigValidationError, or rename/document the field to make clear sanitization is connect-phase-only Medium No
3 seatunnel-core/seatunnel-starter/.../command/SeaTunnelConfValidateCommand.java:226-255 ruleCategory is derived via ad hoc lowercase substring matching over free-text exception messages from several unrelated, unversioned subsystems (FactoryUtil, ConfigValidator, ConfigParserUtil, JSqlParser, Typesafe Config) Low — no test currently breaks, but classification can silently drift if any of those subsystems' message wording changes, since nothing ties message text to category Consider an enum for the four category values instead of magic strings, and/or a longer-term move toward typed classification at the exception-throwing site rather than post-hoc text sniffing Low No

Exceptions caught by validateResult() are not logged (no log.error/log.warn call in the new code) — appropriate here, since the caller (a future CLI/MCP/REST adapter) owns how to surface the structured result; this mirrors the existing execute() contract of throwing rather than logging.

2. Code Quality Assessment

2.1 Coding Standards

Both new classes (ConfigValidationError, ConfigValidationResult) have class-level Javadoc describing their purpose and (for ConfigValidationResult) an explicit disclaimer that this is config-level, not runtime-equivalent, validation — good practice given the "AI CLI" framing in the linked issue. However, per project convention ("new fields must have comments... except simple boilerplate getters/setters"), ConfigValidationError's five fields (location, plugin, optionPath, ruleCategory, message) have no field-level Javadoc explaining their expected format or the fixed set of legal ruleCategory values (parse/option/plugin/validation, which exist only as inline logic in SeaTunnelConfValidateCommand, not documented on the field they populate) — for a class explicitly billed as a "stable," externally-consumed schema, this is a meaningful documentation gap, not boilerplate. Flagging as Medium per project convention.
New private helper methods (validationPhase, validationMode, toValidationError) in SeaTunnelConfValidateCommand are short and self-explanatory; validateResult() itself has a clear Javadoc comment. Naming/style otherwise follow existing conventions (Google/AOSP via Spotless, no wildcard imports), and both new files carry the ASF license header.

2.2 Test Coverage and Test Stability

  • ConfigValidationResultTest covers toJson() field-order/shape for both the success and failure cases, toHumanReadable() for both, and round-trips through JsonUtils.readTree to confirm the JSON is actually parseable — good, focused serialization coverage.
  • SeaTunnelConfValidateCommandTest adds five tests through the real SeaTunnelConfValidateCommand.validateResult() entry point, reusing existing test fixtures (valid_static_dryrun.json, invalid_hocon_syntax.conf, invalid_option_type.json, invalid_plugin_loadability.json — all pre-existing files, verified present) plus one new inline temp-file config for the "missing plugin name" case. This is a reasonable spread across the four ruleCategory buckets.
  • Stability rating: Stable. All new/modified tests are synchronous, deterministic, operate on local files or in-memory objects — no Thread.sleep, no timing assumptions, no external I/O or network, no shared mutable static state, no unordered-collection assertions without normalization.
  • Gap (ties into Issue 1): no test exercises validateResult() with DryRun.CONNECT, so the only code path where location/plugin extraction is even theoretically reachable is completely untested by this PR.

2.3 Documentation Updates

Not required for this change: validateResult() is not wired to any CLI flag, output format, or user-visible behavior (no --output=json or similar flag is introduced — the PR description explicitly says so), so there is nothing in docs/en/docs/zh for an end user to read yet. This should be revisited once a real CLI/MCP/REST entry point starts exposing this model to users.

3. Architectural Soundness

3.1 Elegance of the Solution

Mixed. The overall shape — a small, versioned (SCHEMA_VERSION), immutable value model with separate toJson()/toHumanReadable() projections, kept fully decoupled from the existing execute() control flow — is a clean, precise way to introduce a machine-readable surface without risking the existing CLI contract. The classification mechanism underneath it (toValidationError()), however, is a temporary workaround: it retrofits structure onto free-text exception messages after the fact, rather than having the validation code itself produce structured failures. That is a reasonable, low-risk way to ship an initial version, but it is the part of this PR most likely to need revisiting as soon as a real adapter starts depending on location/plugin/optionPath being reliably populated.

3.2 Maintainability

Good for the two new value classes (small, focused, well-tested). The classification logic in SeaTunnelConfValidateCommand carries the maintainability risk described in Issue 3 — it silently couples to message wording in files it does not own.

3.3 Extensibility

The SCHEMA_VERSION constant and the List<ConfigValidationError> shape (rather than a single error) both anticipate future growth (multiple errors per run, schema evolution) without requiring a breaking change — good forward design for a model meant to be consumed by external adapters.

3.4 Historical-Version Compatibility

Fully preserved — this is purely additive, dead-code-path-today functionality with no interaction with existing config options, serialization formats, or checkpoint/savepoint state.

4. Issue Summary

Number Issue Location Severity
1 location/plugin extraction only works for the untested connect phase and is always null for the default static phase; optionPath is never populated seatunnel-core/seatunnel-starter/.../command/SeaTunnelConfValidateCommand.java:226-255 Medium
2 "Sanitized message" is only true for the connect phase; the default static phase carries raw exception text into the new structured model seatunnel-core/seatunnel-starter/.../command/SeaTunnelConfValidateCommand.java:181-214 Medium
3 ruleCategory classified via fragile free-text keyword matching over messages from several unrelated, unversioned subsystems seatunnel-core/seatunnel-starter/.../command/SeaTunnelConfValidateCommand.java:226-255 Low
4 ConfigValidationError's five fields lack field-level Javadoc explaining format/legal values, despite being billed as a stable external schema seatunnel-core/seatunnel-core-starter/.../validation/ConfigValidationError.java:34-46 Medium

5. Merge Recommendation

Conclusion: Ready to merge after fixes

  1. Blockers — must be fixed
    • None that block merging this specific diff on safety/compatibility grounds — the change is purely additive, execute() and existing CLI behavior are untouched, and every new test is deterministic. I would not block merge on Issues 1-4 alone.
  2. Recommended fixes — non-blocking, but should land before any real adapter (CLI JSON flag, MCP tool, REST endpoint) starts depending on this model
    • Populate optionPath from the existing "option: <key>" message shape (Issue 1), and either fix or clearly document that location/plugin are effectively static-phase-null today.
    • Add a DryRun.CONNECT test through validateResult() so the one code path where location/plugin extraction can work is actually verified (Issue 1).
    • Decide whether the static phase should also go through DryRunConnectFailureMessageSanitizer (or an equivalent) before this model's "sanitized message" claim is accurate for the default phase (Issue 2).
    • Consider a RuleCategory enum instead of magic strings, to reduce the free-text classification's blast radius as an alternative worth discussing (Issue 3).
    • Add field-level Javadoc to ConfigValidationError's five fields (Issue 4).

Overall assessment: this is a low-risk, well-scoped, well-tested first step that correctly avoids touching any existing behavior while laying groundwork for the AI-assisted config-generation effort described in #10651. My main concern is that the PR's own description promises a schema with five populated, sanitized, stable per-error fields, and my trace shows that in the default/primary validation phase only two of those five (ruleCategory, message) are reliably populated, and message is not actually sanitized in that phase either — so the feature does not yet fully solve the "not machine-readable" problem the linked issue set out to fix, specifically for the phase (static) that essentially all current users exercise. I don't see a fundamentally better alternative architecture here — post-hoc classification of legacy exception text is a reasonable bridge given the underlying validation code doesn't throw structured exceptions today — but I'd like a maintainer to weigh in on whether landing this now (as an inert, unconsumed foundation to iterate on) or holding for the fixes above is the preferred sequencing, since I have read-only review rights and cannot make that merge-timing call myself.

On CI: as of this review, the apache-side Build check was still in_progress (dereferenced to the fork run CryoThrust/seatunnel run 33826236750); the unit-test jobs that exercise this PR's new tests had not yet completed. Please confirm those finish green before merge. mergeStateStatus shows BLOCKED/mergeable: MERGEABLE, which reflects branch protection requiring an approving review (reviewDecision: REVIEW_REQUIRED), not a merge conflict.

Welcome to the Apache SeaTunnel community, and thank you for the clear PR description, the honest scoping note about what this intentionally does not do yet, and for grounding the design in a real roadmap issue rather than adding speculative surface area. Looking forward to seeing this connected to an actual adapter.

@CryoThrust

Copy link
Copy Markdown
Contributor Author

Addressed review feedback in 409b1d4: structured results now sanitize messages for all validation phases, extract option/options paths, use a closed internal category enum, and document all error fields. Added CONNECT-phase coverage for location/plugin extraction and sensitive-message sanitization. Spotless applied. Local Maven test execution remains blocked by the existing seatunnel-config-shade shaded Config test-compile issue; no new compilation error was reported before that reactor failure.

@CryoThrust

Copy link
Copy Markdown
Contributor Author

The review-requested changes are now on head 409b1d4 and the Build workflow is still running with no failures reported so far. When convenient, could you please take another look at the updated structured-result model?

@CryoThrust

Copy link
Copy Markdown
Contributor Author

Correction to my earlier note: the current PR head is 9fa47a4d6, which removes the over-broad option-path assertion that caused the prior test failure. The new Build workflow is now running on this exact commit.

@DanielLeens DanielLeens left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What Problem Does This PR Solve?

Per the linked design issue (#10651), SeaTunnel wants to let AI generate task config files and validate them through an official path, but today seatunnel.sh --check/--dry-run only produces plain, unstructured exception text — there is no reusable, machine-readable validation result that a future CLI JSON flag, an MCP tool, or a REST adapter could consume without screen-scraping human-oriented error strings.

This PR adds two new value classes (ConfigValidationResult, ConfigValidationError) plus a new SeaTunnelConfValidateCommand.validateResult() method that runs the existing execute() validation, catches the resulting ConfigCheckException, and repackages it into a small versioned model with valid/phase/errors[], where each error carries location/plugin/optionPath/ruleCategory/message.

This is my second full pass. My first review was at f7f0824 (CHANGES_REQUESTED). Two new commits landed on top of it since: 409b1d427 ("Enrich structured validation errors" — addresses my feedback) and 9fa47a4d6 ("Avoid assuming option path for type failures" — a follow-up correction to the first). I re-diffed the entire PR against current dev from scratch and separately diffed f7f0824 → current head to see exactly what changed in response to my review, rather than trusting the author's own summary of what was addressed.

1. Code Change Review

1.1 Core Logic Analysis

Confirming execute() and existing --check behavior are still untouched: re-diffed SeaTunnelConfValidateCommand.java against dev directly — the entire pre-existing execute() method is still byte-for-byte unchanged; all new code is additive (new fields, new validateResult() method and its private helpers). validateResult() has no other caller anywhere in the repo besides the new tests.

Round-1 Issue 2 (Medium) — "sanitized message" only true for the connect phase: FIXED. Previously, execute()'s catch block only sanitized the message via DryRunConnectFailureMessageSanitizer.sanitize(...) when DryRun.CONNECT; the default static phase carried raw exception text straight into the new model despite the PR description promising a sanitized field. I traced the new code at SeaTunnelConfValidateCommand.java:212-214: validateResult() now calls DryRunConnectFailureMessageSanitizer.sanitize(message) unconditionally, before calling toValidationError(...), regardless of which phase produced the failure. I also read the sanitizer itself (DryRunConnectFailureMessageSanitizer.java, unmodified, pre-existing) to confirm it's safe to apply broadly: it only masks JDBC URLs and key: value/key=value pairs whose key matches a fixed credential-like word list (password|token|secret|...), plus a 2048-char truncation — no risk of stripping legitimate static-phase diagnostic content. A new test, testValidationResultSanitizesConnectFailure, exercises this against a pre-existing test fixture (DryRunTestSourceFactory's sensitive_connection_failure option, already on dev, not added by this PR) that throws a message containing secret-password/secret-token, and asserts neither leaks through. Confirmed fixed.

Round-1 Issue 1 (Medium) — location/plugin only for the untested connect phase, optionPath never populated: PARTIALLY FIXED, and honestly so.

  • location/plugin: a new test, testValidationResultIncludesConnectFailureLocationAndPlugin, now drives validateResult() through DryRun.CONNECT against DryRunTestSourceFactory's fail_connection option and asserts location == "source[0](DryRunTestSource)" and plugin == "DryRunTestSource". This closes the "completely unverified" half of my original finding — the one code path where this extraction can work is now actually tested.
  • optionPath: now populated via a new OPTION_PATH_PATTERN = Pattern.compile("(?m)^\\s*options?:\\s*([^\\r\\n]+)") (SeaTunnelConfValidateCommand.java:80-83), which I traced against OptionUtil.formatError/formatOptionsError's pre-existing "%s: %s\n type: %s\n constraint: %s" template (OptionUtil.java:36,99-104) — this correctly extracts the option key for missing-required-option and unknown-key failures, which is genuine, verified progress over "never populated at all."
  • However, I traced a third failure shape this PR's own test suite exercises — testValidationResultClassifiesOptionFailure against config/invalid_option_type.json (env.parallelism: "invalid_string_value", a type-coercion failure) — and that message never goes through OptionUtil.formatError at all. I followed the call chain: validateOptionTypes() → ReadonlyConfig.getOptional(option) → ConfigUtil.convertValue(rawValue, option) (seatunnel-api/.../util/ConfigUtil.java:44-83), which on a Jackson deserialization failure throws IllegalArgumentException("Json parsing exception, value '%s', and expected type '%s'", ...) — a completely different message shape with no "option: " line, so OPTION_PATH_PATTERN cannot match it and optionPath stays null for this case too. The last commit (9fa47a4d6) is the author correctly discovering this and removing an assertion (Assertions.assertNotNull(result.getErrors().get(0).getOptionPath())) that no longer held for this test — I want to call this out as the right engineering move (fixing the test to match verified behavior rather than leaving a wrong assertion in), but the net effect is that optionPath is still incomplete: populated for OptionUtil-formatted option-rule violations, still null for type-coercion/Jackson conversion failures, even though both are classified under the same "option" ruleCategory bucket. Worth being explicit about this in the PR description/Javadoc rather than leaving it implicit.
  • Location: seatunnel-core/seatunnel-core-starter/.../command/SeaTunnelConfValidateCommand.java:226-260 (helper), seatunnel-api/.../util/ConfigUtil.java:44-83 (root cause of the gap).

Round-1 Issue 3 (Low) — ruleCategory as free-text magic strings: FIXED. toValidationError() now returns a private ValidationRuleCategory enum (PARSE/OPTION/PLUGIN/VALIDATION, each carrying its external string value) instead of raw String literals internally (SeaTunnelConfValidateCommand.java:255-268). The classification logic itself (lower-cased substring matching against messages from unrelated, unowned subsystems) is unchanged and still carries the drift risk I described in round 1 — the enum only removes the internal magic-string typo risk, it doesn't make the classification itself more robust against upstream message wording changes. That's consistent with what I asked for (a smaller, non-blocking ask), not a claim that the deeper structural concern is resolved.

Round-1 Issue 4 (Medium) — missing field-level Javadoc on ConfigValidationError's five fields: FIXED. All five fields (location, plugin, optionPath, ruleCategory, message) now carry field-level Javadoc describing their format and when they're populated (ConfigValidationError.java:27-40). Confirmed by direct diff read.

Minor bonus fix, not something I'd asked for: the location/option-path regexes moved from being recompiled on every call (Pattern.compile(...) inline) to private static final Pattern fields (SeaTunnelConfValidateCommand.java:80-83) — resolves the minor, non-blocking perf note from my round-1 §1.3.

1.2 Compatibility Impact

Fully compatible, unchanged from round 1 — re-verified execute() is still byte-for-byte identical to dev. No config option, default value, protocol, or serialization format is touched. validateResult() still has zero callers outside this PR's own tests.

1.3 Performance / Side-Effect Analysis

All new code runs once per CLI invocation of validateResult(), not on any hot path, not called anywhere in production code yet. The Pattern hoisting noted above is a small genuine improvement. No new threads, connections, or unbounded collections.

1.4 Error Handling and Logging

Exceptions caught by validateResult() are not logged, which is appropriate — the caller owns how to surface the structured result, mirroring execute()'s existing throw-rather-than-log contract. The sanitizer fix (Issue 2 above) closes the one real gap I had here.

2. Code Quality Assessment

2.1 Coding Standards

All five ConfigValidationError fields now have Javadoc (Issue 4, fixed). New private helper methods remain short and self-explanatory; validateResult() keeps its Javadoc. New files carry the ASF license header. No new coding-standard gaps found in the delta since my last review.

2.2 Test Coverage and Test Stability

Two new tests exercise the fixes directly: testValidationResultIncludesConnectFailureLocationAndPlugin and testValidationResultSanitizesConnectFailure, both using buildConnectArgsFromPath (pre-existing helper) and DryRunTestSourceFactory (pre-existing test fixture, unmodified by this PR — I confirmed via direct diff that this file has zero changes in this PR despite being central to the new tests, so nothing was added purely to make a new test trivially pass).

Stability rating: Stable. Both new tests are synchronous, deterministic, operate on local temp files and in-memory objects — no Thread.sleep, no timing assumptions, no shared mutable static state. Same rating as round 1 for the rest of the suite; nothing in the delta changes that assessment.

The one test-suite change worth calling out explicitly (already discussed in 1.1): testValidationResultClassifiesOptionFailure had an assertNotNull(getOptionPath()) assertion removed in the last commit. I traced this myself rather than taking the commit message at face value — it is a correct fix, not a weakening. The type-coercion failure this test exercises never populates optionPath under either the old or new code (see 1.1); the assertion was simply wrong from the moment optionPath extraction was introduced in the previous commit, and removing it makes the test assert only what the code actually and correctly does. I don't consider this a test-severity regression.

Independent CI verification for the current head: I dereferenced the fork's own Actions run for the exact current head (9fa47a4d6ed321d5d77f5b1faeec51cf8c2cb9e0) — CryoThrust/seatunnel run 33838680596. As of this review the overall run is still in_progress (the author's last comment, posted 3 minutes before I checked, says the same), but I pulled job-level status rather than waiting on the aggregate: Code style = success, License header = success, and all four unit-test matrix jobs (JDK 8/11 × ubuntu/windows) = success, which is exactly the job set that compiles and runs this PR's new/changed test code. The remaining in-progress jobs are unrelated integration-test modules. Please let the full workflow finish before merge, but the module-relevant jobs are genuinely green on this head, not just claimed green by the author.

2.3 Documentation Updates

Not required — validateResult() still isn't wired to any CLI flag or user-visible output; nothing changed here since round 1.

3. Architectural Soundness

3.1 Elegance of the Solution

Unchanged assessment from round 1: the overall shape (small, versioned, immutable value model, fully decoupled from execute()'s control flow) is clean and low-risk. The classification/extraction mechanism (toValidationError()) is still a temporary workaround — retrofitting structure onto free-text exception messages after the fact — and this round's fixes make that workaround measurably better (three of four documented fields now genuinely populated for the primary static phase, with the fourth's remaining gap now precisely scoped and honestly reflected in the test suite) rather than changing its fundamental nature. That's the right amount of investment for a first, inert, unconsumed foundation stone.

3.2 Maintainability

Improved since round 1: the ValidationRuleCategory enum and the field-level Javadoc both make this easier for the next person to extend correctly. The underlying coupling to unowned subsystems' message wording (round-1 Issue 3) is unchanged in nature, as expected — I didn't ask for that to be solved in this round.

3.3 Extensibility

Same as round 1 — SCHEMA_VERSION and the List<ConfigValidationError> shape anticipate future growth. One thing I noticed on this pass that I hadn't flagged before: ConfigValidationResult.failure(phase, error) only ever accepts a single ConfigValidationError (ConfigValidationResult.java:56-58), so today's model can carry at most one error per validation run even though the field is a List. Not a defect — execute() itself only ever throws the first failure it hits — just noting it as a low-priority forward-compatibility gap for whoever wires up an adapter that wants to report every problem in one pass, not something to hold this PR on.

3.4 Historical-Version Compatibility

Fully preserved — purely additive, no interaction with existing config options, serialization formats, or checkpoint/savepoint state.

4. Issue Summary

# Issue Location Severity Status
1 optionPath populated for OptionUtil-formatted option-rule violations, but still null for type-coercion/Jackson conversion failures (both bucketed under the same "option" category) SeaTunnelConfValidateCommand.java:226-260; root cause in seatunnel-api/.../util/ConfigUtil.java:44-83 Low (down from Medium — genuinely narrower now, and the gap is honestly reflected in the test suite rather than papered over) Carryover, partially fixed
2 "Sanitized message" only true for connect phase SeaTunnelConfValidateCommand.java:181-214 Medium Fixed
3 ruleCategory magic strings SeaTunnelConfValidateCommand.java:226-268 Low Fixed (enum added; underlying substring-classification risk unchanged, as expected)
4 Missing field-level Javadoc on ConfigValidationError ConfigValidationError.java:27-40 Medium Fixed
5 (new, informational) ConfigValidationResult.failure() only ever carries a single error despite the List<ConfigValidationError> shape ConfigValidationResult.java:56-58 Low New observation, non-blocking

5. Merge Recommendation

Conclusion: Ready to merge

  1. Blockers — must be fixed

    • None. As in round 1, nothing here blocks merge on safety/compatibility grounds — the change remains purely additive and execute()/existing CLI behavior are untouched. Please confirm the fork's Build workflow finishes fully green (module-relevant jobs — Code style, all four unit-test matrix jobs — are already green as of this review; the rest is unrelated IT modules still running).
  2. Recommended fixes — non-blocking

    • Issue 1: if/when a real adapter starts depending on optionPath, extend extraction to also cover the ConfigUtil.convertValue type-coercion message shape, or document in the field-level Javadoc that optionPath is currently OptionUtil-format-only.
    • Issue 5: consider whether ConfigValidationResult should eventually collect multiple errors per run, or document that it's single-error-by-design for now.

Overall assessment: this round directly and honestly addressed three of my four round-1 findings (sanitization now applies to both phases, location/plugin extraction is now actually tested via a real DryRun.CONNECT path, magic strings became a real enum, and the missing Javadoc was added), and the fourth (optionPath) went from "never populated" to "populated for the primary option-rule-violation shape, with the remaining gap precisely identified and the test suite corrected to reflect what the code truly does rather than what was hoped it did." That last part — catching your own wrong assertion and fixing the test instead of leaving it green-by-luck — is exactly the kind of engineering diligence I want to see, and it's a stronger signal to me than a green checkmark would have been on its own. Nothing here blocks merging this as the inert, unconsumed foundation it's designed to be. Thank you for the thorough, evidence-based turnaround — welcome to the Apache SeaTunnel community, and I'm looking forward to seeing this connected to a real adapter down the line.

@DanielLeens

Copy link
Copy Markdown
Contributor

Quick CI update since my last review only knew the Build run was still in progress: it has since finished, and it failed — but on a job that has nothing to do with this PR's diff.

The failing job is Run / transform-v2-it-part-1 (8, ubuntu-latest), specifically TestFilterRowKindIT.testFilterRowKindMultiTable on the Spark 2/3 engines (expected: <0> but was: <1> on a row-count assertion). This PR only touches seatunnel-api's reusable config-validation-result plumbing — it never goes near the FilterRowKind transform or the Spark multi-table pipeline this test exercises, so this looks like a pre-existing, environment-timing-sensitive E2E flake rather than something introduced by this change. All of the module-relevant jobs (Code style, all four unit-test matrix combinations) are green.

Since this isn't tied to your diff, I don't think there's anything to change in the PR itself — a CI rerun on the failing job should be enough to get a clean signal. My "Ready to merge" conclusion from the last review stands either way.

@CryoThrust

Copy link
Copy Markdown
Contributor Author

The failed workflow was rerun on the fork with --failed after confirming that the only failure was the unrelated transform-v2-it-part-1 (JDK 8) integration test (TestFilterRowKindIT.testFilterRowKindMultiTable). The module-relevant unit-test and code-style jobs were already green; I will follow the rerun result before taking any further action.

@DanielLeens

Copy link
Copy Markdown
Contributor

Thanks for confirming — that matches what I found: transform-v2-it-part-1 (JDK 8)'s TestFilterRowKindIT.testFilterRowKindMultiTable failure is unrelated to this PR's diff, and the module-relevant jobs (Code style, all four unit-test matrix jobs) are already green. No action needed on your end beyond the rerun; my "Ready to merge" conclusion stands.

@davidzollo davidzollo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
LGTM

Thanks for your contribution! This is a solid addition to the project, and your attention to detail really shows. Contributors like you are what keep this community moving forward. Feel free to keep the PRs coming — we're always glad to have you here!

Please ignore the following message if you have joined in the Slack~

To facilitate focused communication among contributors, we’ve created a brand-new dedicated #contributors channel in official Slack workspace.
Slack workspace link: https://s.apache.org/seatunnel-slack

This channel is designed for technical discussions, feature design brainstorming, roadmap sync and direct communication with project core Contributors. Everyone who contributes code, docs, test cases or community ideas is warmly invited to join, share thoughts and collaborate together under The Apache Way. Welcome to join 😄

@davidzollo
davidzollo added this pull request to the merge queue Sep 4, 2026
Merged via the queue into apache:dev with commit ce4cde5 Sep 4, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved core SeaTunnel core module First-time contributor First-time contributor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Discussion] Support AI generation for SeaTunnel task config files

3 participants