Repository navigation
[fix](fe) Fix prepared statement streaming, forwarding and cursor framing - #69314
Conversation
### What problem does this PR solve?
Issue Number: N/A
Problem Summary: Streaming parameters with server-side prepared statements send
COM_STMT_SEND_LONG_DATA before COM_STMT_EXECUTE. The command previously fell
through to the unsupported-command handler, emitting an unread ERR packet and
leaving subsequent responses out of sync. Uploaded parameter values were also
not retained or bound during execution.
Handle uploads without a response, accumulate chunks per statement and parameter,
and defer upload errors until execution. Bind uploaded values without consuming
inline parameter bytes, preserve BLOB bytes as VARBINARY, enforce the parameter
max_allowed_packet limit, and clear upload state after execute or reset. Inline
uploaded values and cached types in the existing forwarding buffer so the master
can bind them without connection-local upload state. Use the parser's current
prepare context when planning a forwarded PREPARE.
### Release note
Fix streaming parameters in server-side prepared statements and prevent extra
response packets from desynchronizing MySQL connections.
### Check List (For Author)
- Test: Unit Test / Regression test / Manual test
- FE build and Checkstyle via build.sh --fe
- run-fe-ut.sh --run: MysqlConnectProcessorLongDataTest, MysqlConnectProcessorCursorFetchTest, ConnectContextTest
- run-regression-test.sh --run -d prepared_stmt_p0 -s send_long_data; generate expected output with -genOut
- Raw protocol silence, deferred errors, binary chunks, empty values, reset, close, and connection reuse
- MariaDB C API mysql_stmt_send_long_data and Connector/J 8.0.28 observer-to-master forwarding with BE 4.1.3
- Behavior changed: Yes, accept streamed parameters without replying to upload commands
- Does this need documentation: No
…atements ### What problem does this PR solve? Issue Number: N/A Related PR: apache#48689 Problem Summary: Connector/J can omit unchanged parameter types when reusing a server-side prepared statement. A non-master FE retains those types locally but forwards the original EXECUTE buffer. The master prepares the SQL again and has no cached types, so repeated execution and rewritten cached INSERT batches fail with NoSuchElementException: No value present. Direct master connections work. Extend the existing long-data forwarding buffer normalization to executions that omit parameter types. Include the cached type codes, NULL bitmap and original value bytes so each forwarded execution can be decoded independently. Test fresh untyped proxy placeholders, unsigned types, NULL transitions and type updates. ### Release note Fix repeated server-side prepared statements and cached batch INSERTs on non-master FEs. ### Check List (For Author) - Test: Unit Test / Manual test - build.sh --fe and Checkstyle passed - run-fe-ut.sh: 48 tests passed, including 12 long-data/type-forwarding tests - Original Connector/J 8.0.27 reproducer passed on master and observer - QA Connector/J 8.0.27/8.0.33/8.4.0/9.7.0: 76 passed, 12 existing TLS cases skipped because enable_ssl=false; cached INSERT batches, INSERT SELECT, stored values and long-data forwarding passed on every driver - BE 4.1.3-rc02 was used with enable_local_shuffle_planner=false during stored-data validation because it lacks the new LOCAL_EXCHANGE_NODE; the original setting was restored afterward - Behavior changed: Yes; forwarded executions include cached parameter types - Does this need documentation: No
### What problem does this PR solve? Issue Number: N/A Related PR: apache#67520 Problem Summary: With useCursorFetch and a positive fetch size, older Connector/J releases consume an empty result's sole end marker while probing for a cursor and wait for another packet. Sending an extra marker instead causes newer drivers to return an empty result prematurely and leaves unread packets on the connection, particularly when connection attributes are disabled or rewritten by a proxy. Return a valid empty cursor for empty results with CLIENT_DEPRECATE_EOF and complete COM_STMT_FETCH with LAST_ROW_SENT. Track cursor state per prepared statement, clear it on execution/reset/close, and preserve it on the client-facing FE after forwarding. Nonempty results retain immediate result delivery. Keep the previous framing for requests from older FEs that cannot complete FETCH, using an optional request capability checked against maintained Thrift branches. ### Release note Fix queries hanging or desynchronizing JDBC connections when useCursorFetch is enabled, including empty results and clients with disabled connection attributes. Nonempty result pagination remains unsupported. ### Check List (For Author) - Test: Unit Test / Regression test / Manual test - build.sh --fe passed with zero Checkstyle violations - run-fe-ut.sh: 66 relevant tests passed; after the compatibility adjustment, 30 adapter/sender/forwarding tests passed - run-regression-test.sh: cursor_fetch_empty_result passed - QA Connector/J 8.0.27/8.0.33/8.4.0/9.7.0: 92 passed, zero failed; 12 TLS cases skipped because enable_ssl=false - Proxy-rewritten Connector/J 8.0.27 reproduced the hang before the fix and passed afterward; anonymous Connector/J 9.7.0 and raw cursor lifecycle passed - BE 4.1.3-rc02 used be_exec_version=10 and enable_local_shuffle_planner=false; an undersized 1 GB BE memory limit caused initial test failures, and the successful rerun used 2 GB - Behavior changed: Yes; empty cursor requests return CURSOR_EXISTS and complete on COM_STMT_FETCH - Does this need documentation: No
### What problem does this PR solve? Issue Number: N/A Related PR: apache#67520 Problem Summary: Normalizing COM_STMT_SEND_LONG_DATA parameters creates a full EXECUTE payload on the connection adapter. Clearing uploads or closing/resetting the statement does not clear that independent copy, so an idle pooled connection can retain the entire streamed value until another EXECUTE or disconnect. Clear the command-only execution buffer in afterStatement, which runs after response finalization and synchronous forwarding. Add a test through real upload and EXECUTE dispatch that verifies complete normalized bytes remain available during execution and are released after the response. ### Release note Reduce FE heap retention on pooled connections after executing statements with streamed parameters. ### Check List (For Author) - Test: Unit Test - run-fe-ut.sh: 32 long-data, protocol adapter and forwarding tests passed - Checkstyle: both modified Java files checked, zero violations - Behavior changed: Yes; command-only parameter buffers are released after response completion - Does this need documentation: No
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
FE UT Coverage ReportIncrement line coverage |
TPC-H: Total hot run time: 28986 ms |
TPC-DS: Total hot run time: 150143 ms |
ClickBench: Total hot run time: 24.35 s |
Local pipeline review — ✅ PASSschema: doris-repo-review/v1
status: PASS
pr: apache/doris#69314
commit: 3fedd6185eefae5bf968733b8bd20c7954838073
base: 01e735b29c8b8803b9e2df10b46b9b02db1e6251
reviewed_at: 2026-10-20T14:22+08:00
reviewer: CalvinKirs
model: gpt-6.1-sol
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: trueNotes for maintainers
Reviewed locally with the |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
FE Regression Coverage ReportIncrement line coverage |
|
PR approved by anyone and no changes requested. |
|
PR approved by at least one committer and no changes requested. |
…ranch-4.2 (#69463) ### What problem does this PR solve? Related PR: #69314 Problem Summary: - `COM_STMT_SEND_LONG_DATA` returns an unexpected response and does not bind uploaded chunks, leaving streaming clients out of sync. Accept chunks without replying, defer upload errors until EXECUTE, and clear uploaded data after execution or RESET. - Reusing cached server-prepared statements through a non-master FE can fail because the master prepares each forwarded execution independently. Forward complete parameter types and values, including streamed parameters, and use the active prepare-stage context during proxy planning. - Empty results with `useCursorFetch=true` can hang or leave unread packets, including when connection attributes are absent or rewritten. Return an empty cursor, complete FETCH with `LAST_ROW_SENT`, and preserve framing for older FEs through an optional peer capability. - Completed EXECUTE payloads can remain attached to idle connections. Release command payload buffers after response finalization and forwarding finish. Backport adaptations for branch-4.2: - Keep protocol state in the existing `ConnectContext`/`ConnectProcessor` architecture and preserve branch-4.2's expansion of cached parameter types only when forwarding. - Regenerate regression expectations with the standard test script to retain branch-4.2's existing VARBINARY function semantics. ### Release note Fix streamed prepared parameters, repeated prepared execution through non-master FEs, and empty cursor fetch hangs. ### Check List (For Author) - Test - [x] Regression test: `prepared_stmt_p0/send_long_data` and `prepared_stmt_p0/cursor_fetch_empty_result` passed with `run-regression-test.sh`. - [x] Unit Test: 91 FE tests passed with `run-fe-ut.sh`, with zero failures, errors or skips. - [x] Manual test: Connector/J 8.0.27, 8.0.33, 8.4.0 and 9.7.0 each passed 428 checks against the built branch-4.2 FE and BE 4.1.3. Connect explicitly to master and follower endpoints; verify streamed bytes, empty streams and statement reuse; run 50 batches of three rows per version and compare all stored values; alternate empty/nonempty cursor results with and without connection attributes, including forwarded execution and connection reuse. - [x] Full FE build passed via `build.sh --fe`; Checkstyle passed with zero violations. - Behavior changed: - [x] Yes. Long-data uploads have no response; forwarded parameters are self-contained; empty cursors complete through FETCH; completed command buffers are released. - Does this need documentation? - [x] No. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
### What problem does this PR solve? Related PR: #69314 JDBC `setBytes()` and Connector/ODBC binary bindings can send `MYSQL_TYPE_VAR_STRING` / `MYSQL_TYPE_STRING` rather than BLOB. FE decoded those bytes with Java's replacement UTF-8 decoder, silently changing arbitrary binary data. For example, `0080FF4100C328` became `00EFBFBDEFBFBD4100EFBFBD28` in `SELECT HEX(?)`. The BLOB handling added by #69314 did not cover these wire types. Decode string parameters strictly and retain non-UTF-8 payloads as binary literals. Bind those payloads through a BE-side cast to the original string category, preserving string function and comparison typing without replacing bytes. Valid UTF-8 retains the existing text literal types, and BLOB types retain their existing binary behavior. Use value-based binary literal child hashes so equivalent casts remain equal after rebinding and forwarding. ### Release note Preserve non-UTF-8 bytes in MySQL server-prepared string parameters, including JDBC `setBytes()` and Connector/ODBC binary bindings. ### Check List (For Author) - Test: - [x] Regression test: generated and replayed `prepared_stmt_p0/binary_parameters`; replayed `prepared_stmt_p0/send_long_data`. Covers all byte values, malformed UTF-8, parameter reuse, stored HEX/MD5, string expression predicates, NULL/empty values and ordinary Unicode text. - [x] Unit Test: 17 tests passed across `MysqlParameterLiteralTest` and `MysqlConnectProcessorLongDataTest`, including string wire aliases, cached types, uploads and forwarding. - [x] Manual test: Connector/J 8.2.0 server-prepared `setBytes`, `setBinaryStream` and `setBlob` preserve `0080FF4100C328`. - [x] Manual test: Linux Connector/ODBC 5.3.14 Unicode, 17 cases passed, including 1-MiB binary/Unicode inputs. An additional 17-case proxy-rewritten VARCHAR wire-type probe also passed. - [x] FE build and Checkstyle passed. - Behavior changed: - [x] Yes. Non-UTF-8 string-coded parameters preserve the input bytes instead of substituting U+FFFD. Valid UTF-8 text behavior is unchanged. - Does this need documentation? - [x] No. Runtime regression checks used the newly built master FE with an existing 4.1.4 BE and condition cache disabled. The predicate regression uses `CONCAT(value, '') = ?`; direct constant predicate pushdown on a current master BE was not validated. No BE, storage-format or protocol-field changes are included.
…meter types (#70150) ### What problem does this PR solve? Issue Number: None Related PR: #68162, #69314 Problem Summary: **In short.** This ports `MysqlPreparedStatementForwardingTest` from branch-4.1 #68162 to master. Master already has the fix the test guards, through #69314. The port adds what master's own tests do not cover: a NULL bitmap of two bytes, and the forward request that `FEOpExecutor` builds and Thrift carries. The PR changes tests only. **Background.** A client that runs a server-side prepared statement again may leave out the parameter types (`new_params_bind_flag` = 0) and have the server reuse the ones it cached. A follower forwards every COM_STMT_EXECUTE to the master. The master prepares the statement again for each forwarded execution, so it has no cached types and cannot bind the parameters. #68162 fixed this on branch-4.1, together with the Lance prepared vector search it was written for. Master fixed it independently in #69314: when the client reuses cached types, `MysqlConnectProcessor.handleExecute` writes the forwarded packet again, with the NULL bitmap, `new_params_bind_flag` = 1, every type code including the unsigned flag, and the values. #69314 also has the prepare stage plan with the context of the re-parsed statement, which is the other generic part of #68162. Its tests in `MysqlConnectProcessorLongDataTest` cover cached types, unsigned flags, changing NULLs, changing types, all-NULL executions and the prepare context. **What those tests leave out.** They use at most four parameters, so the NULL bitmap always fits in one byte. They also give the master the buffer kept on the follower's context, rather than the request `FEOpExecutor.buildStmtForwardParams` builds. **What this PR adds.** The test runs a ten-parameter statement five times on a follower connection: - The parameter types are sent with the first and the fourth execution; the other executions reuse the cached ones. - One execution has a NULL parameter in each byte of the NULL bitmap. - An unsigned INT parameter becomes a BIGINT in the fourth execution. After each execution, the test builds the forward request with `FEOpExecutor`, passes it through a Thrift round trip and executes it on a new proxy context whose placeholders have no types. It then checks three things: every parameter binds to the same value as on the follower, the unsigned flag carries over, and the client's packet is read to its end and left unchanged. The code is the 4.1 test adapted to master's structure: master's `PrepareCommand`, query id and audit setup, as in `MysqlConnectProcessorLongDataTest`, and expressions compared directly. The 4.1 assertion that the follower keeps the client's buffer was dropped, because master writes a separate buffer for cached types by design. **Results.** | `handleExecute` on master | `MysqlConnectProcessorLongDataTest` (14) | `MysqlPreparedStatementForwardingTest` (1) | |---|---|---| | as is | pass | pass | | no rewrite of the forwarded packet for cached types (the bug #68162 fixed) | 3 fail | fails: the master binds untyped placeholders (`No value present`) | | the rewrite keeps only the first byte of the NULL bitmap | pass | fails: the master reads the packet one byte off | Co-authored-by: Gabriel <[email protected]>
What problem does this PR solve?
Problem Summary:
COM_STMT_SEND_LONG_DATAincorrectly returns an error packet, leaving streaming clients out of sync. Accept parameter chunks without replying, defer upload errors until EXECUTE, preserve binary bytes, and clear upload state on EXECUTE/RESET/CLOSE.NoSuchElementException: No value present: the driver omits unchanged parameter types, while the master prepares each forwarded execution independently. Include cached types, the NULL bitmap and complete values in forwarded EXECUTE payloads.useCursorFetch=truecan hang or leave unread packets when client connection attributes are disabled or rewritten. Return a valid empty cursor, complete FETCH withLAST_ROW_SENT, retain cursor state on the client-facing FE, and preserve existing framing for older FEs. Nonempty results continue to use buffered delivery; nonempty cursor pagination is not added.Release note
Check List (For Author)
prepared_stmt_p0/send_long_dataandprepared_stmt_p0/cursor_fetch_empty_resultpassed during implementation.Check List (For Reviewer who merge this PR)