[Bug] Arrow Flight SQL: PL/SQL CALL returns INTERNAL and executes its DML side effect four times #67369

Activity

1. Doris-Breakwater commented on Sep 1, 2026

   

   Doris-Breakwater

   on Sep 1, 2026

   Collaborator

   More actions

   Breakwater-GitHub-Analysis-Slot: slot_d813a37c5b64

   Preliminary maintainer analysis

   Judgment: This is a valid, high-severity FE / Arrow Flight SQL correctness defect. The exact 4x multiplication is not yet proven from the supplied evidence, but the reported INTERNAL error-after-side-effect is directly explained by the 4.1.3-rc02 source. Until fixed, PL/SQL stored-procedure CALL should be treated as unsafe over Flight SQL. The issue currently has no labels; please add the repository's bug and Arrow Flight SQL / FE area labels.

   Verified from commit 31263df4dc1d4d3a27517d264802cd4d6b92c874

   1. Flight SQL executes the SQL during GetFlightInfo: DorisFlightSqlProducer.executeQueryStatement() calls FlightSqlConnectProcessor.handleQuery() and turns an error in ConnectContext into an INTERNAL response (source).
   2. A stored-procedure call reaches CallProcedure.run(), which invokes PlSqlOperation.execute() (source). There is no Arrow Flight protocol gate in CallCommand before that invocation.
   3. PlSqlOperation.execute() evaluates the procedure via exec.parseAndEval(args) first, then unconditionally executes ctx.getMysqlChannel().reset() in finally (source). On a Flight session, FlightSqlConnectContext.getMysqlChannel() always throws exactly getMysqlChannel not in mysql connection (source). The exception is caught and converted to an error state only after the procedure body may already have committed its DML. This confirms the partial-success ambiguity and the observed client error.
   4. This is broader than the final reset(): PlsqlResult also sends rows, metadata, and EOF through MysqlChannel (source). Removing or guarding only the final reset would not establish Flight SQL support for PL/SQL results.
   5. Doris defaults max_query_retry_time to 3, which makes "initial attempt + 3 retries = 4" a plausible clue (source). It is not sufficient proof of the four executions: this particular MySQL-channel exception is caught inside PlSqlOperation and returned as query state, while StmtExecutor.queryRetry() retries only specific thrown failures. Static inspection therefore does not establish whether the repeated execution is caused by an outer Flight/ADBC GetFlightInfo replay, an independent retry while running the inner INSERT, or another layer.

   Information needed to prove the 4x path

   • Exact versions of adbc_driver_flightsql, adbc_driver_manager, and pyarrow (for example, the output of python -m pip freeze | grep -E 'adbc|pyarrow'). The DB-API path prepares statements automatically in current ADBC implementations, so the driver version matters.
   • FE topology and mode: cloud vs. non-cloud, number of FEs, whether 127.0.0.1:41070 is the master FE, and the effective max_query_retry_time value.
   • FE audit records and the surrounding fe.log for one fresh unique key, including the outer CALL, inner INSERT, query IDs, timestamps, and the full server stack trace. On an isolated test cluster, enabling enable_print_request_before_execution would make attempt counting clearer.
   • A client/RPC trace that counts CreatePreparedStatement, GetFlightInfoPreparedStatement, and/or GetFlightInfoStatement requests for the single cur.execute(). Please also compare the DB-API prepared path with a low-level ADBC statement executed without calling prepare().
   • On an isolated test cluster, repeat once with max_query_retry_time = 0. A change from four rows to one would implicate a Doris retry path; no change would shift attention to Flight/ADBC replay. Do not use this as a production mitigation because one write can still commit before the error.

   The audit pattern will localize the replay boundary:

   • Four outer CALL audit entries plus four inner INSERT entries: the Flight request is being replayed/resubmitted.
   • One outer CALL entry plus four inner INSERT entries: retry occurs inside procedure/DML execution.
   • One outer and one inner entry despite four rows: investigate the storage/result-counting path and capture transaction/load labels.

   Recommended next steps

   1. Immediate containment: reject stored-procedure CALL on ConnectType.ARROW_FLIGHT_SQL before PlSqlOperation.execute() is entered. This is the only currently evidenced way to guarantee no side effect before an unsupported-protocol error. Keep the documented MySQL/JDBC workaround and the warning not to retry ambiguous Flight errors.
   2. If Flight SQL support is intended: make PL/SQL result delivery and finalization protocol-aware end to end; do not fix only getMysqlChannel().reset(). Define how repeated GetFlightInfo requests are deduplicated or otherwise prevented from re-running side-effecting statements.
   3. Add regression coverage for both prepared and direct Flight SQL paths. Each must assert either (a) a pre-execution unsupported error with zero inserted rows, or (b) success with exactly one row. Also add a focused FE test proving rejection happens before the PL/SQL executor is invoked.
   4. Correlate the outer and inner query IDs before selecting the final retry fix. The current evidence is enough for containment, but not enough to claim the root cause of the exact four attempts.
2. morningman commented on Sep 4, 2026

   

   morningman

   on Sep 4, 2026

   ContributorAuthor

   More actions

   Scope: this only affects branch-4.0, branch-4.1 and branch-3.1. Not reproducible on master.

   PL/SQL was removed from master by #58700 (ec4014ec268, 2025-12-08), which deleted the whole
   org.apache.doris.plsql package, the PLLexer.g4 / PLParser.g4 grammars, CallProcedure, and the
   CREATE|DROP|SHOW CREATE PROCEDURE grammar rules. branch-4.0, branch-4.1 and branch-3.1 still
   carry that code, so the bug lives only there.

   Verified on a master build (both protocols, MySQL on 9033 and Arrow Flight SQL on 8077):

   statement	MySQL	Arrow Flight SQL
   CREATE [OR REPLACE] PROCEDURE ...	errCode = 2, no viable alternative at input 'CREATE OR REPLACE PROCEDURE'(line 1, pos 18)	same syntax error, wrapped in get flight info statement failed
   DROP PROCEDURE ...	no viable alternative at input 'DROP PROCEDURE'	same
   SHOW CREATE PROCEDURE ...	no viable alternative at input 'SHOW CREATE PROCEDURE'	same
   CALL p(444,'x')	do not support call function P	same, no side effect
   SHOW PROCEDURE STATUS / SHOW FUNCTION STATUS	empty result set	empty result set
   SELECT * FROM information_schema.routines	0 rows	0 rows

   The last two rows are the deliberate compatibility shims restored by #66659 — the SQL entry points parse
   and return empty, the stored-procedure functionality itself stays removed. On master CALL only accepts
   the built-ins EXECUTE_STMT and FLUSH_AUDIT_LOG; anything else is rejected in
   CallFunc.getFunc() before anything is executed, so no DML can be duplicated.

   Root cause on 4.x

   PlSqlOperation.execute() ends with an unconditional

   ctx.getMysqlChannel().reset();

   in its finally block, and FlightSqlConnectContext.getMysqlChannel() throws
   getMysqlChannel not in mysql connection. The INSERT has already run by then, the throw is swallowed
   into an ERR state, executeQueryStatement turns that into after executeQueryStatement handleQuery,
   and the client sees INTERNAL — which matches the report exactly: the error is raised after the side
   effect.

   The deeper reason is that PL/SQL assumes a MySQL channel throughout:
   PlsqlQueryExecutor.executeQuery() runs every inner statement through
   ConnectContext.get().cloneContext(), and cloneContext() builds a plain ConnectContext
   (connectType = MYSQL) while copying mysqlChannel — which is null on a Flight SQL connection.

   About "four times"

   I could not find a server-side loop that would execute the body four times:
   getFlightInfoStatement executes the statement once per GetFlightInfo; StmtExecutor.queryRetry()
   only retries the three SystemInfoService.NEED_REPLAN_ERRORS messages; handleQueryWithRetry()
   additionally requires RpcException and ConnectType.MYSQL; and the retryTimes loop in
   InsertIntoTableCommand.initPlan() only re-plans before the transaction begins. If anyone reproduces
   this on 4.x, fe.audit.log settles it: four CALL entries means the client retried GetFlightInfo,
   while one CALL plus four INSERTs would mean a server-side loop.

   Next step

   A guard for branch-4.1 is on the way: reject CALL <procedure> on a non-MySQL connection in
   CallProcedure.create(), i.e. before exec.parseAndEval() runs, so the statement fails cleanly with
   no side effect. branch-4.0 / branch-3.1 can take the same cherry-pick.

   /cc — please set the affected-version labels (4.0 / 4.1 / 3.1) and drop any master label.
3. morningman commented on Sep 4, 2026

   

   morningman

   on Sep 4, 2026

   ContributorAuthor

   More actions

   Correction to the "next step" in my previous comment: there will be no fix on the 4.x branches. #67521 has been closed — PL/SQL is not being developed further on the release branches, and it is already removed from master by #58700.

   The workaround stands for branch-4.0 / branch-4.1 / branch-3.1: run PL/SQL CALL over the MySQL protocol, and do not auto-retry on getMysqlChannel not in mysql connection, because the server may already have applied the side effect.
4. morningman commented on Sep 4, 2026

   

   morningman

   on Sep 4, 2026

   ContributorAuthor

   More actions

   Closing as not planned. PL/SQL was removed from master by #58700, so this is not reproducible there, and the release branches (branch-4.0 / branch-4.1 / branch-3.1) will not get a fix — see the comments above for the root cause and the workaround.
5. morningman

   closed this as not plannedon Sep 4, 2026