Repository navigation
Commit 958aaf4
authored
[fix](protocol) Prevent Connector/J cursor fetch from hanging on empty results (apache#67520)
### What problem does this PR solve?
Issue Number: None
Related PR: apache#61050, apache#61062
After `CLIENT_DEPRECATE_EOF` is negotiated, affected Connector/J clients
using `useCursorFetch=true` and a positive fetch size can hang when a
server-prepared statement returns an empty result. The driver consumes
the first terminator after column definitions while checking whether a
server cursor was created. With no rows, this consumes the only
result-end marker, so the driver waits for a packet that Doris will
never send.
The reported Connector/J 8.2.0 case is covered by real-driver tests.
This PR preserves the final result marker for affected clients and
avoids inserting the compatibility marker for identified Connector/J
9.5+ and MariaDB clients. It also preserves this behavior through FE
forwarding and fixes related packet/capability regressions.
### What is changed?
1. **Cursor result boundaries.** Read `CURSOR_TYPE_READ_ONLY` from each
`COM_STMT_EXECUTE`, retain it in the request context, and classify
driver behavior using connection attributes. For affected cursor clients
with deprecated EOF enabled, insert a compatibility ResultSet OK after
metadata so the real final marker remains available. Identified modern
clients and ordinary non-cursor requests keep the standard sequence.
2. **FE forwarding.** Forward cursor intent and negotiated capabilities
through optional Thrift fields. The master restores them before
serializing results. A new follower adapts buffered results from an
older master that has not confirmed protocol adaptation:
insert/remove/convert metadata boundaries as appropriate and convert the
final legacy EOF when needed. Row payloads are retained; confirmed
results are not adapted twice.
3. **Complete OK information.** Preserve the original forwarded DML/DDL
OK and ERR packets rather than rebuilding a partial successful OK.
Retain affected rows, warnings, info, load labels and transaction IDs.
Preserve warnings and server status when producing/converting result-set
terminators. SQL is not re-executed by this adaptation.
4. **Legacy protocol compatibility.** Configure the serializer using the
intersection of client/server capabilities, rather than server defaults.
Limit the empty OK-info compatibility byte to the applicable negotiated
format, preserving legacy OK behavior.
5. **Arrow follower compatibility.** Access MySQL-channel capabilities
only for MySQL connections, so the shared forwarding path does not call
the unsupported MySQL-channel getter on an Arrow Flight SQL context.
6. **Clients without connection attributes.** Remove the new fail-fast
rejection of anonymous cursor clients. When client behavior is unknown,
prioritize the older Connector/J compatibility sequence. This retains
the older-client target but cannot simultaneously guarantee
compatibility with anonymous clients requiring the standard sequence.
7. **File-load compatibility.** In both LOAD handlers, require
`CLIENT_LOCAL_FILES` only for client-side `LOCAL INFILE` uploads.
FE-side file reads do not require a client-upload capability and remain
usable when that bit is absent.
Full PR diff against the merged master, at `32d01e5df19`:
| Category | Files | Added lines | Removed lines | Net added lines |
|---|---:|---:|---:|---:|
| Production code, including Thrift IDL | 12 | 218 | 40 | 178 |
| Tests and expected output | 12 | 787 | 13 | 774 |
The test total includes 11 automatically generated expected-output
lines. Environment configuration and review documents are excluded.
### Compatibility boundaries
- This preserves streamed results; it does not implement MySQL
server-side cursors or `COM_STMT_FETCH` batching.
- **New follower → old master** is covered by the mixed-version client
matrix. **Old follower → new master** may have already lost cursor
intent; the master cannot reconstruct it. The historical cursor hang on
that route remains until the client-facing follower is upgraded.
Existing ordinary prepared-statement behavior is retained.
- Identified Connector/J 9.5+ and MariaDB clients use the standard path.
Anonymous clients requiring that sequence remain outside the
older-client fallback guarantee.
- The OIDC provider and authentication routing implementation are not
changed by this PR.
### Release note
Fix affected Connector/J cursor queries hanging on empty prepared
results. Preserve negotiated MySQL packet formats, forwarded DML/DDL OK
information, Arrow Flight SQL forwarding and FE-side file loading.
Upgrade the client-facing follower to preserve cursor intent during an
FE rolling upgrade.
### Check List (For Author)
- Test:
- [x] Unit Test: **113 targeted FE tests passed, 0 failures/errors**,
before the final master/JUnit 5 merge. Coverage includes driver
classification, cursor boundaries, forwarding, capabilities, packet
serialization, authentication routing and both LOAD handlers.
- [x] Regression test: `prepared_stmt_p0/cursor_fetch_empty_result`,
`prepared_stmt_p0/prepared_show`, and `arrow_flight_sql_p0/test_ddl`
through a real follower passed. Cursor expected output was generated by
the regression runner.
- [x] Manual test: **120 successful real-JDBC driver/endpoint/transport
runs** across baseline/candidate direct endpoints, a same-version
follower and a new follower targeting an old master, with plaintext/TLS
modes. Each driver JAR runs in its own JVM to avoid classpath version
collisions.
- MySQL Connector/J: **5.1.49, 8.0.28, 8.0.33, 8.2.0, 8.4.0, 9.0.0,
9.4.0, 9.5.0, 9.6.0**. MariaDB Connector/J: **3.5.6**.
- Each run exercises 12 requested combinations of cursor/server-prepare
switches and fetch sizes **0/1/10000**, repeated
Statement/PreparedStatement empty→nonempty→empty results on reused
connections, and wrong-password rejection. Drivers may couple cursor and
server-prepare settings internally. Connection/read timeouts bound
failures; a timeout is not counted as success.
- Additional checks passed: explicit TLS 1.2/1.3 (16 groups), MySQL
Shell 8/9 ordinary authentication (16), legacy EOF via PyMySQL (8),
SQL-error connection reuse (8), and suppressed connection attributes
with older drivers (12).
- Actual baseline/candidate FE-file and client-file imports passed;
forwarded INSERT retained label/status/txnId. Real LDAP/local-password
routing, rejected credentials and connection reuse passed 20
baseline/candidate checks across plaintext/TLS.
- Independent FE-only raw-protocol tests passed 12 configurations
covering multi-result status, cursor transitions, RESET/CLOSE and reuse
after errors. These are packet tests, not additional real JDBC runs or
proof of mixed-version multi-statement forwarding.
- FE build passed before the final master merge. After the merge,
standalone `mvn checkstyle:check -pl fe-core -Dcheckstyle.skip=false`
passed with **0 violations**. Compilation and unit tests were not rerun
after the JUnit 5 migration.
- The multi-version JDBC matrix was executed locally with retained
scripts/logs; it is not an automatic all-version CI matrix.
- Review and validation limits:
- The latest independent review of the complete final diff found **0
Blocker / 0 Major production findings**. It noted two deterministic
SQL-result assertions that still use `assertEquals` rather than
regression `qt` output, and missing dedicated tests for errors occurring
after partial metadata or rows have been buffered. Those error paths
were inspected in source, not fault-injected; the OK/ERR preservation
unit test does not cover buffered intermediate errors.
- Connector/J 6.0.6 failed handshakes on both baseline and candidate in
six runs; it is not counted as passing or newly supported.
- Four broader prepared suites failed identically on baseline/candidate
with the available older BE because it cannot execute the plan.
Authentication-integration metadata regression also remains incomplete
with that BE. Four independent SELECT configurations without a BE were
blocked and not counted as passing.
- Full product OIDC-provider login/new TLS-extension E2E and the
customer's exact SmartBI environment were not validated. Ordinary
password/TLS and authentication unit tests do not replace these checks.
- Behavior changed:
- [x] Yes. Correct cursor packet boundaries, preserve complete forwarded
OK information and use negotiated capabilities, with the compatibility
boundaries above.
- Does this need documentation?
- [x] No.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label1 parent 0d36c66 commit 958aaf4
26 files changed
Lines changed: 1120 additions & 55 deletions
File tree
- fe/fe-core/src
- main/java/org/apache/doris
- mysql
- nereids/trees/plans/commands
- load
- qe
- test/java/org/apache/doris
- mysql
- nereids/trees/plans/commands
- qe
- gensrc/thrift
- regression-test
- data/prepared_stmt_p0
- suites
- arrow_flight_sql_p0
- prepared_stmt_p0
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
58 | 58 | | |
59 | 59 | | |
60 | 60 | | |
61 | | - | |
62 | | - | |
63 | | - | |
64 | | - | |
65 | | - | |
66 | | - | |
67 | | - | |
68 | | - | |
69 | | - | |
70 | | - | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
71 | 66 | | |
72 | 67 | | |
73 | 68 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
183 | 183 | | |
184 | 184 | | |
185 | 185 | | |
186 | | - | |
187 | | - | |
188 | | - | |
189 | | - | |
190 | 186 | | |
191 | 187 | | |
192 | 188 | | |
| |||
208 | 204 | | |
209 | 205 | | |
210 | 206 | | |
211 | | - | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
212 | 212 | | |
213 | 213 | | |
214 | 214 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
40 | 40 | | |
41 | 41 | | |
42 | 42 | | |
| 43 | + | |
43 | 44 | | |
44 | 45 | | |
45 | 46 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
496 | 496 | | |
497 | 497 | | |
498 | 498 | | |
499 | | - | |
| 499 | + | |
500 | 500 | | |
501 | 501 | | |
502 | 502 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
219 | 219 | | |
220 | 220 | | |
221 | 221 | | |
222 | | - | |
| 222 | + | |
223 | 223 | | |
224 | 224 | | |
225 | 225 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
287 | 287 | | |
288 | 288 | | |
289 | 289 | | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
290 | 293 | | |
291 | 294 | | |
292 | 295 | | |
| |||
513 | 516 | | |
514 | 517 | | |
515 | 518 | | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
516 | 527 | | |
517 | 528 | | |
518 | 529 | | |
| |||
1008 | 1019 | | |
1009 | 1020 | | |
1010 | 1021 | | |
| 1022 | + | |
1011 | 1023 | | |
1012 | 1024 | | |
1013 | 1025 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
54 | 54 | | |
55 | 55 | | |
56 | 56 | | |
| 57 | + | |
57 | 58 | | |
58 | 59 | | |
59 | 60 | | |
| |||
754 | 755 | | |
755 | 756 | | |
756 | 757 | | |
757 | | - | |
758 | | - | |
759 | | - | |
760 | | - | |
761 | | - | |
762 | | - | |
763 | 758 | | |
764 | 759 | | |
765 | 760 | | |
| 761 | + | |
766 | 762 | | |
767 | 763 | | |
768 | 764 | | |
| |||
848 | 844 | | |
849 | 845 | | |
850 | 846 | | |
| 847 | + | |
851 | 848 | | |
852 | 849 | | |
853 | 850 | | |
| |||
883 | 880 | | |
884 | 881 | | |
885 | 882 | | |
| 883 | + | |
| 884 | + | |
| 885 | + | |
| 886 | + | |
| 887 | + | |
| 888 | + | |
| 889 | + | |
| 890 | + | |
| 891 | + | |
| 892 | + | |
| 893 | + | |
| 894 | + | |
| 895 | + | |
| 896 | + | |
| 897 | + | |
| 898 | + | |
| 899 | + | |
| 900 | + | |
886 | 901 | | |
887 | 902 | | |
888 | 903 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
29 | 33 | | |
30 | 34 | | |
31 | 35 | | |
| |||
46 | 50 | | |
47 | 51 | | |
48 | 52 | | |
| 53 | + | |
49 | 54 | | |
50 | 55 | | |
51 | 56 | | |
| |||
215 | 220 | | |
216 | 221 | | |
217 | 222 | | |
| 223 | + | |
218 | 224 | | |
219 | 225 | | |
220 | 226 | | |
| |||
231 | 237 | | |
232 | 238 | | |
233 | 239 | | |
| 240 | + | |
234 | 241 | | |
235 | 242 | | |
236 | 243 | | |
| |||
264 | 271 | | |
265 | 272 | | |
266 | 273 | | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
267 | 344 | | |
268 | 345 | | |
269 | 346 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
63 | 63 | | |
64 | 64 | | |
65 | 65 | | |
| 66 | + | |
66 | 67 | | |
67 | 68 | | |
68 | 69 | | |
| |||
128 | 129 | | |
129 | 130 | | |
130 | 131 | | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
131 | 136 | | |
132 | 137 | | |
133 | 138 | | |
134 | 139 | | |
135 | | - | |
136 | | - | |
137 | | - | |
138 | 140 | | |
139 | 141 | | |
140 | 142 | | |
| |||
208 | 210 | | |
209 | 211 | | |
210 | 212 | | |
211 | | - | |
212 | | - | |
| 213 | + | |
| 214 | + | |
213 | 215 | | |
214 | 216 | | |
215 | 217 | | |
| |||
0 commit comments