Skip to content

Commit 1250443

Browse files
authored
[fix](sqlcache) Do not replay the MySQL sql cache on an Arrow Flight connection (#67381)
### What problem does this PR solve? Issue Number: close #67364 Related PR: #65182, #65615 Problem Summary: Reading anything over Arrow Flight SQL fails with `INTERNAL` / `IllegalStateException` as soon as the query hits the FE sql cache. The FE sql cache is keyed by `<catalog>.<db>:<user>:<sql text>` (`NereidsSqlCacheManager.generateCacheKey`) and is shared by every protocol, but its rows are MySQL wire protocol packets that `StmtExecutor.sendCachedValues` replays through a `MysqlChannel`. An Arrow Flight SQL connection has no channel, so an entry created by an identical MySQL query makes `handleQueryStmt` take the cached-plan branch and fail `Preconditions.checkState(connectType == MYSQL)` in `sendFields()`. The client sees: ``` INTERNAL: get flight info statement failed, after executeQueryStatement handleQuery, error code: ERR_UNKNOWN_ERROR, error msg: IllegalStateException, msg: null ``` The `CacheAnalyzer` branch right below it is already gated on `channel != null` with a `// TODO support arrow flight sql`; only this cached-plan replay was left unguarded. **This is not about `HLL` / `QUANTILE_STATE`.** The issue was reported on raw aggregate-state columns, but both the FE schema helper and the BE map `HLL` / `BITMAP` / `QUANTILE_STATE` to Arrow `binary` and have Arrow writers for them, and they read back correctly once the query is actually executed. They only looked special because that sql text was the one primed through the MySQL control session; `select 1` fails exactly the same way. Two conditions have to line up, which is why this is not seen more often: 1. The same sql text must have been run on a MySQL connection first (an Arrow Flight connection never populates the cache). 2. Both sessions must agree on every session variable the cache compares (`NereidsSqlCacheManager.usedVariablesChanged` over the `affectQueryResult*` set). The MySQL **JDBC driver** adds `STRICT_TRANS_TABLES` to `sql_mode` at connect time while the Arrow Flight JDBC driver does not, so a JDBC control session masks the bug -- a `mysql` CLI session, or any client that leaves `sql_mode` alone, does not. ### What is changed - `ConnectProcessor.executeQuery`: look the sql cache up only for a MySQL connection. Any other protocol re-executes the query and gets its result from the BE. Such a connection never populates the cache either, so this only removes a broken read path; it does not change MySQL behaviour. - `StmtExecutor.handleQueryStmt`: assert the channel in the cached-plan branch, so a future regression names the protocol instead of throwing a bare `IllegalStateException`. - `SessionVariable`: mark `return_object_data_as_binary` as `affectQueryResultInExecution`. It is forwarded to the BE and decides whether the MySQL result writer serializes `HLL` / `BITMAP` / `QUANTILE_STATE` as their raw bytes or as NULL, so it changes the cached rows and must take part in the cache key comparison. Without it, a session that turns it on is served the NULLs cached by a session that had it off -- a separate, pure-MySQL-protocol correctness bug in the same family.
1 parent da1eb72 commit 1250443

5 files changed

Lines changed: 296 additions & 2 deletions

File tree

‎fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -280,7 +280,15 @@ public void executeQuery(String originStmt) throws Exception {
280280
ctx.setSqlHash(sqlHash);
281281

282282
SessionVariable sessionVariable = ctx.getSessionVariable();
283-
boolean wantToParseSqlFromSqlCache = CacheAnalyzer.canUseSqlCache(sessionVariable);
283+
// The sql cache keeps the result rows in MySQL wire format and replays them through a
284+
// MysqlChannel (StmtExecutor.sendCachedValues -> sendFields), which only exists on a MySQL
285+
// connection. An Arrow Flight SQL connection has no channel and needs Arrow batches built by
286+
// the BE, and the cached rows would be wrong for it anyway (object types such as HLL /
287+
// BITMAP / QUANTILE_STATE were serialized as NULL under return_object_data_as_binary=false).
288+
// So a non-MySQL connection must always re-execute the query instead of replaying the cache.
289+
// The cache is never populated by such a connection either, see StmtExecutor.handleQueryStmt.
290+
boolean wantToParseSqlFromSqlCache = connectType.equals(ConnectType.MYSQL)
291+
&& CacheAnalyzer.canUseSqlCache(sessionVariable);
284292
List<StatementBase> stmts = null;
285293
long parseSqlStartTime = System.currentTimeMillis();
286294
List<StatementBase> cachedStmts = null;

‎fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1992,7 +1992,12 @@ public void setMaxJoinNumberOfReorder(int maxJoinNumberOfReorder) {
19921992
@VarAttrDef.VarAttr(name = GLOBAL_PARTITION_TOPN_THRESHOLD)
19931993
private double globalPartitionTopNThreshold = 100;
19941994

1995-
@VarAttrDef.VarAttr(name = RETURN_OBJECT_DATA_AS_BINARY)
1995+
// Forwarded to the BE as a query option and read by the MySQL result writer: when it is false
1996+
// the object types (HLL / BITMAP / QUANTILE_STATE) are serialized as NULL instead of their raw
1997+
// bytes. It therefore changes the result rows the sql cache stores, and must take part in the
1998+
// cache key, otherwise a session that turns it on replays the NULLs cached by a session that
1999+
// had it off. It only affects execution, not the plan, so it does not force forwarding.
2000+
@VarAttrDef.VarAttr(name = RETURN_OBJECT_DATA_AS_BINARY, affectQueryResultInExecution = true)
19962001
private boolean returnObjectDataAsBinary = false;
19972002

19982003
@VarAttrDef.VarAttr(name = BLOCK_ENCRYPTION_MODE, affectQueryResultInPlan = true)

‎fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1455,6 +1455,12 @@ private void handleQueryStmt() throws Exception {
14551455
LogicalPlanAdapter logicalPlanAdapter = (LogicalPlanAdapter) parsedStmt;
14561456
LogicalPlan logicalPlan = logicalPlanAdapter.getLogicalPlan();
14571457
if (logicalPlan instanceof org.apache.doris.nereids.trees.plans.algebra.SqlCache) {
1458+
// sendCachedValues replays MySQL protocol packets, so it needs a MysqlChannel.
1459+
// ConnectProcessor.executeQuery only looks the sql cache up for a MySQL connection,
1460+
// so a cached plan must never reach another protocol here.
1461+
Preconditions.checkState(channel != null,
1462+
"sql cache can only be replayed on a MySQL connection, but connect type is %s",
1463+
context.getConnectType());
14581464
NereidsPlanner nereidsPlanner = (NereidsPlanner) planner;
14591465
PhysicalSqlCache physicalSqlCache = (PhysicalSqlCache) nereidsPlanner.getPhysicalPlan();
14601466
sendCachedValues(channel, physicalSqlCache.getCacheValues(), logicalPlanAdapter, false, true);
Lines changed: 168 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,168 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
import org.apache.doris.regression.util.JdbcUtils
19+
20+
// Regression for https://github.com/apache/doris/issues/67364
21+
//
22+
// The FE sql cache is shared by every protocol, but its rows are MySQL wire protocol packets that
23+
// StmtExecutor.sendCachedValues replays through a MysqlChannel, and an Arrow Flight SQL connection
24+
// has none. Replaying an entry created by an identical MySQL query used to fail
25+
// Preconditions.checkState(connectType == MYSQL) in StmtExecutor.sendFields() and reach the client
26+
// as "INTERNAL ... IllegalStateException, msg: null", for any result type. The issue was reported
27+
// on raw HLL / QUANTILE_STATE columns only because that sql text happened to be the one primed
28+
// through the MySQL control session.
29+
//
30+
// Two setup details decide whether this test can reproduce the bug at all -- get either wrong and
31+
// it stays green on a broken FE:
32+
//
33+
// 1. The flight statements are sent on the raw flight connection. Suite.arrow_flight_sql()
34+
// prepends "USE <db>;" to the statement, which changes the sql text and therefore the cache
35+
// key (NereidsSqlCacheManager.generateCacheKey is "<catalog>.<db>:<user>:<sql text>").
36+
// 2. Both sessions must agree on every session variable the cache compares
37+
// (NereidsSqlCacheManager.usedVariablesChanged compares the whole affectQueryResult* set).
38+
// The MySQL JDBC driver adds STRICT_TRANS_TABLES to sql_mode at connect time while the Arrow
39+
// Flight JDBC driver does not, and sql_mode is affectQueryResultInPlan, so an unaligned
40+
// sql_mode alone makes every flight lookup miss.
41+
suite("test_sql_cache_over_arrow_flight") {
42+
def mysqlConn = context.getConn()
43+
def flightConn = context.getArrowFlightSqlConnection()
44+
45+
def runOnMysql = { String stmt ->
46+
def (result, meta) = JdbcUtils.executeToList(mysqlConn, stmt)
47+
return result
48+
}
49+
def runOnFlight = { String stmt ->
50+
def (result, meta) = JdbcUtils.executeToList(flightConn, stmt)
51+
return result
52+
}
53+
54+
def hasSqlCache = { String stmt ->
55+
def (explainRows, meta) = JdbcUtils.executeToList(mysqlConn, "explain physical plan " + stmt)
56+
return explainRows.collect { row -> row.get(0).toString() }.join("\n").contains("PhysicalSqlCache")
57+
}
58+
59+
// Create the cache entry on the MySQL connection, and wait until an identical statement is
60+
// actually served from it, so the flight query below really runs against a populated cache.
61+
def primeSqlCacheOnMysql = { String stmt ->
62+
for (int i = 0; i < 60; ++i) {
63+
runOnMysql(stmt)
64+
if (hasSqlCache(stmt)) {
65+
return
66+
}
67+
sleep(1000)
68+
}
69+
throw new IllegalStateException("failed to create sql cache for: " + stmt)
70+
}
71+
72+
// JdbcUtils renders a binary column as an "0x.." hex string, but falls back to the raw object
73+
// when the driver does not implement getBytes().
74+
def isNonEmptyBinary = { value ->
75+
if (value == null) {
76+
return false
77+
}
78+
if (value instanceof byte[]) {
79+
return ((byte[]) value).length > 0
80+
}
81+
return value.toString().length() > "0x".length()
82+
}
83+
84+
withGlobalLock("cache_last_version_interval_second") {
85+
runOnMysql "ADMIN SET ALL FRONTENDS CONFIG ('cache_last_version_interval_second' = '0')"
86+
87+
def dbName = context.dbName
88+
runOnMysql "USE `${dbName}`"
89+
runOnFlight "USE `${dbName}`"
90+
runOnMysql "set enable_sql_cache=true"
91+
runOnFlight "set enable_sql_cache=true"
92+
// See note 2 above: without this the flight lookup always misses and the test is toothless.
93+
runOnMysql "set sql_mode='ONLY_FULL_GROUP_BY'"
94+
runOnFlight "set sql_mode='ONLY_FULL_GROUP_BY'"
95+
96+
// The cache key is the catalog, the database, the user and the sql text, and the lookup
97+
// additionally compares the session variables that affect the result. The statements below
98+
// are byte identical on both connections, so assert the rest of the inputs match too.
99+
assertEquals(runOnMysql("select database()")[0][0], runOnFlight("select database()")[0][0])
100+
assertEquals(runOnMysql("select current_user()")[0][0], runOnFlight("select current_user()")[0][0])
101+
assertEquals(runOnMysql("select @@sql_mode")[0][0], runOnFlight("select @@sql_mode")[0][0])
102+
103+
// 1. A constant result, cached in the FE itself (PhysicalOneRowRelation.computeResultInFe
104+
// -> tryAddFeSqlCache). This replays through the resultSet branch of sendCachedValues,
105+
// needs no table and no quiet window, and is the cheapest way to hit the bug.
106+
def constantSql = "select 1 as c, 'x' as s"
107+
primeSqlCacheOnMysql(constantSql)
108+
def constantOnFlight = runOnFlight(constantSql)
109+
assertEquals(1, constantOnFlight.size())
110+
assertEquals(1, constantOnFlight[0][0] as int)
111+
assertEquals("x", constantOnFlight[0][1].toString())
112+
113+
def tblName = "test_sql_cache_over_arrow_flight_tbl"
114+
runOnMysql "DROP TABLE IF EXISTS ${tblName}"
115+
runOnMysql """
116+
CREATE TABLE ${tblName} (
117+
k INT,
118+
h HLL HLL_UNION,
119+
q QUANTILE_STATE QUANTILE_UNION
120+
) AGGREGATE KEY(k)
121+
DISTRIBUTED BY HASH(k) BUCKETS 1
122+
PROPERTIES("replication_num"="1")
123+
"""
124+
runOnMysql "INSERT INTO ${tblName} SELECT 1, HLL_HASH('x'), TO_QUANTILE_STATE(1, 2048)"
125+
126+
// 2. A plain scalar result read from a table, cached on the BE. The failure was protocol
127+
// specific, not type specific.
128+
def scalarSql = "select k from ${tblName} order by k"
129+
primeSqlCacheOnMysql(scalarSql)
130+
def scalarOnFlight = runOnFlight(scalarSql)
131+
assertEquals(1, scalarOnFlight.size())
132+
assertEquals(1, scalarOnFlight[0][0] as int)
133+
134+
// 3. The raw aggregate state columns from the issue. HLL and QUANTILE_STATE are carried as
135+
// arrow binary (be/src/format/arrow/arrow_row_batch.cpp), so flight returns the serialized
136+
// state, while the MySQL protocol keeps showing NULL under
137+
// return_object_data_as_binary=false. Asserting both at once also proves the flight result
138+
// is produced by the BE rather than replayed from the MySQL rows sitting in the cache.
139+
def rawStateSql = "select h, q from ${tblName}"
140+
primeSqlCacheOnMysql(rawStateSql)
141+
def rawStateOnMysql = runOnMysql(rawStateSql)
142+
assertEquals(1, rawStateOnMysql.size())
143+
assertNull(rawStateOnMysql[0][0])
144+
assertNull(rawStateOnMysql[0][1])
145+
def rawStateOnFlight = runOnFlight(rawStateSql)
146+
assertEquals(1, rawStateOnFlight.size())
147+
assertTrue(isNonEmptyBinary(rawStateOnFlight[0][0]),
148+
"expect a non empty HLL state over arrow flight, but got: " + rawStateOnFlight[0][0])
149+
assertTrue(isNonEmptyBinary(rawStateOnFlight[0][1]),
150+
"expect a non empty QUANTILE_STATE over arrow flight, but got: " + rawStateOnFlight[0][1])
151+
152+
// 4. The server side conversions the issue used as a workaround.
153+
def convertedSql = "select hll_cardinality(h) as c, quantile_percent(q, 0.5) as p from ${tblName}"
154+
primeSqlCacheOnMysql(convertedSql)
155+
def convertedOnFlight = runOnFlight(convertedSql)
156+
assertEquals(1, convertedOnFlight.size())
157+
assertEquals(1L, convertedOnFlight[0][0] as long)
158+
assertEquals(1.0d, convertedOnFlight[0][1] as double, 1e-9)
159+
160+
// The flight queries must not have consumed the cache: a cached plan reaching a non MySQL
161+
// connection is exactly the crash this test guards against, and the entries must still be
162+
// there for the MySQL session afterwards.
163+
assertTrue(hasSqlCache(constantSql))
164+
assertTrue(hasSqlCache(scalarSql))
165+
assertTrue(hasSqlCache(rawStateSql))
166+
assertTrue(hasSqlCache(convertedSql))
167+
}
168+
}
Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
import org.apache.doris.regression.util.JdbcUtils
19+
20+
// return_object_data_as_binary decides whether the BE's MySQL result writer serializes HLL /
21+
// BITMAP / QUANTILE_STATE as their raw bytes or as NULL, so it changes the very rows the sql cache
22+
// stores. It must therefore take part in the cache key comparison
23+
// (NereidsSqlCacheManager.usedVariablesChanged over SessionVariable.affectQueryResultFields),
24+
// otherwise a session that turns it on replays the NULLs cached by a session that had it off.
25+
suite("sql_cache_object_type") {
26+
def conn = context.getConn()
27+
def run = { String stmt ->
28+
def (result, meta) = JdbcUtils.executeToList(conn, stmt)
29+
return result
30+
}
31+
def hasSqlCache = { String stmt ->
32+
def (rows, meta) = JdbcUtils.executeToList(conn, "explain physical plan " + stmt)
33+
return rows.collect { row -> row.get(0).toString() }.join("\n").contains("PhysicalSqlCache")
34+
}
35+
def primeSqlCache = { String stmt ->
36+
for (int i = 0; i < 60; ++i) {
37+
run(stmt)
38+
if (hasSqlCache(stmt)) {
39+
return
40+
}
41+
sleep(1000)
42+
}
43+
throw new IllegalStateException("failed to create sql cache for: " + stmt)
44+
}
45+
def isNonEmpty = { value ->
46+
if (value == null) {
47+
return false
48+
}
49+
if (value instanceof byte[]) {
50+
return ((byte[]) value).length > 0
51+
}
52+
return !value.toString().isEmpty()
53+
}
54+
55+
withGlobalLock("cache_last_version_interval_second") {
56+
run "ADMIN SET ALL FRONTENDS CONFIG ('cache_last_version_interval_second' = '0')"
57+
run "set enable_sql_cache=true"
58+
59+
def tblName = "sql_cache_object_type_tbl"
60+
run "DROP TABLE IF EXISTS ${tblName}"
61+
run """
62+
CREATE TABLE ${tblName} (
63+
k INT,
64+
h HLL HLL_UNION,
65+
b BITMAP BITMAP_UNION
66+
) AGGREGATE KEY(k)
67+
DISTRIBUTED BY HASH(k) BUCKETS 1
68+
PROPERTIES("replication_num"="1")
69+
"""
70+
run "INSERT INTO ${tblName} SELECT 1, HLL_HASH('x'), TO_BITMAP(1)"
71+
72+
def objectSql = "select h, b from ${tblName}"
73+
74+
// With the default (false) the object columns come back as NULL, and that is what lands in
75+
// the cache.
76+
run "set return_object_data_as_binary=false"
77+
primeSqlCache(objectSql)
78+
def asNull = run(objectSql)
79+
assertEquals(1, asNull.size())
80+
assertNull(asNull[0][0])
81+
assertNull(asNull[0][1])
82+
83+
// Turning it on must not be served the cached NULLs: it is a different result, so it is a
84+
// different cache key and the query has to be executed again.
85+
run "set return_object_data_as_binary=true"
86+
assertFalse(hasSqlCache(objectSql),
87+
"return_object_data_as_binary=true must not reuse the entry cached with it off")
88+
def asBinary = run(objectSql)
89+
assertEquals(1, asBinary.size())
90+
assertTrue(isNonEmpty(asBinary[0][0]),
91+
"expect the raw HLL bytes, but got: " + asBinary[0][0])
92+
assertTrue(isNonEmpty(asBinary[0][1]),
93+
"expect the raw BITMAP bytes, but got: " + asBinary[0][1])
94+
95+
// The two settings keep their own entries, and each still serves its own result.
96+
primeSqlCache(objectSql)
97+
def asBinaryCached = run(objectSql)
98+
assertTrue(isNonEmpty(asBinaryCached[0][0]))
99+
assertTrue(isNonEmpty(asBinaryCached[0][1]))
100+
101+
run "set return_object_data_as_binary=false"
102+
assertTrue(hasSqlCache(objectSql))
103+
def asNullAgain = run(objectSql)
104+
assertNull(asNullAgain[0][0])
105+
assertNull(asNullAgain[0][1])
106+
}
107+
}

0 commit comments

Comments
 (0)