Skip to content

Commit d4b69e5

Browse files
morningmanclaude
andcommitted
[test](fluss) Check that enable_jni_heap_admission leaves fluss rows unchanged
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: enable_jni_heap_admission has the fluss and paimon connectors declare, on every range whose JNI reader holds much of BE's JVM heap, how much it will hold, and BE opens those readers only while what it admitted fits its budget. It is off by default. The unit tests cover the estimates and the gate; nothing ran a real fluss scan with the variable on. The new suite reads the fluss_test fixtures with the variable on and records the rows: primary-key buckets read whole (PK_FULL ranges, which declare), a union read of a primary-key table whose lake half the paimon connector plans and whose tail is a PK_TAIL range (both declare), and a log read, which declares nothing. Each query is also run with the variable off and the two results are compared in the suite, since what it asserts is that admission may only make a reader wait and never changes what it returns. It also records that the variable is off by default. The suite forces the v2 file scanner, the only one with a fluss reader, because the pipeline's fuzzy mode randomises the choice. ### Release note None ### Check List (For Author) - Test: Regression test - external_table_p0/fluss: all 18 suites pass, serially, against a local cluster and the fluss 1.0.0 docker environment; the .out was generated with -genOut and then checked by the full run - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 7269c10 commit d4b69e5

2 files changed

Lines changed: 130 additions & 0 deletions

File tree

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
-- This file is automatically generated. You should know what you did if you want to edit this
2+
-- !default_off --
3+
enable_jni_heap_admission false false 0
4+
5+
-- !pk_rows --
6+
1 k1 1.5
7+
2 k2-updated 22.5
8+
4 k4 4.5
9+
10+
-- !pk_count --
11+
3
12+
13+
-- !pk_part --
14+
1 q1a 20260101
15+
2 q1b-updated 20260101
16+
3 q2a 20260102
17+
18+
-- !union_rows --
19+
1 m1
20+
10 m10-hot
21+
2 m2-hot
22+
3 m3
23+
4 m4
24+
5 m5-lake
25+
6 m6
26+
8 m8
27+
9 m9
28+
29+
-- !union_count --
30+
9
31+
32+
-- !log_rows --
33+
1 alice 10.10
34+
2 bob 20.20
35+
3 carol 30.30
36+
Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
// enable_jni_heap_admission has the connectors declare, on every range whose JNI reader holds much of
19+
// BE's JVM heap, how much it will hold, and BE opens those readers only while what it admitted fits
20+
// its budget. It is off by default. Turned on it may make a reader wait for room, and nothing else: the
21+
// rows must come back exactly as they do with it off.
22+
//
23+
// The reads that declare are here: fluss primary-key buckets read whole (PK_FULL), and a union read of a
24+
// primary-key table, whose tail is a PK_TAIL range and whose lake half the paimon connector plans. A log
25+
// read declares nothing and is here as the case that must not change either. Each query is recorded
26+
// with the variable on, and compared with the same query with it off; the comparison stays in the code
27+
// because what it asserts is the agreement.
28+
//
29+
// Fixtures come from docker/thirdparties/docker-compose/fluss/sql/init.sql and init-lake-tail.sql, and
30+
// are static: this suite never writes.
31+
suite("test_fluss_jni_heap_admission", "p0,external") {
32+
String enabled = context.config.otherConfigs.get("enableFlussTest")
33+
if (enabled == null || !enabled.equalsIgnoreCase("true")) {
34+
return
35+
}
36+
37+
String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
38+
String coordinatorPort = context.config.otherConfigs.get("fluss_coordinator_port")
39+
String minioPort = context.config.otherConfigs.get("fluss_minio_port")
40+
String bootstrapServers = "${externalEnvIp}:${coordinatorPort}"
41+
String catalogName = "test_fluss_jni_heap_admission"
42+
43+
// Off unless a statement asks for it.
44+
qt_default_off """show variables like 'enable_jni_heap_admission'"""
45+
46+
sql """drop catalog if exists ${catalogName}"""
47+
// required: a union read that quietly fell back to fluss alone would never plan a lake split, and
48+
// the paimon half of the declaration would go untested.
49+
sql """
50+
create catalog ${catalogName} properties (
51+
"type" = "fluss",
52+
"fluss.bootstrap.servers" = "${bootstrapServers}",
53+
"fluss.lake.paimon.s3.endpoint" = "http://${externalEnvIp}:${minioPort}",
54+
"fluss.lake.paimon.s3.access-key" = "minioadmin",
55+
"fluss.lake.paimon.s3.secret-key" = "minioadmin",
56+
"fluss.union_read.mode" = "required"
57+
);
58+
"""
59+
sql """switch ${catalogName}"""
60+
sql """use fluss_test"""
61+
// The C++ glue exists only for the v2 file scanner, and the session variable that picks between
62+
// them is randomised by the fuzzy mode this pipeline runs.
63+
sql """set enable_file_scanner_v2 = true"""
64+
65+
def rowsOf = { String query -> sql(query).collect { row -> row.collect { it.toString() } } }
66+
def sameWithAdmissionOff = { String query ->
67+
sql """set enable_jni_heap_admission = false"""
68+
def off = rowsOf(query)
69+
sql """set enable_jni_heap_admission = true"""
70+
def on = rowsOf(query)
71+
assertEquals(off, on, "enable_jni_heap_admission changed the rows of: ${query}\noff=${off}\non=${on}")
72+
}
73+
74+
sql """set enable_jni_heap_admission = true"""
75+
76+
// --- primary-key buckets read whole -------------------------------------
77+
order_qt_pk_rows """select id, name, score from pk_basic"""
78+
order_qt_pk_count """select count(*) from pk_basic"""
79+
order_qt_pk_part """select id, name, dt from pk_part"""
80+
sameWithAdmissionOff("select id, name, score from pk_basic order by id")
81+
sameWithAdmissionOff("select id, name, dt from pk_part order by id, dt")
82+
83+
// --- a union read: the lake half from paimon, the tail a PK_TAIL --------
84+
order_qt_union_rows """select id, name from lake_pk_multi"""
85+
order_qt_union_count """select count(*) from lake_pk_multi"""
86+
sameWithAdmissionOff("select id, name from lake_pk_multi order by id")
87+
88+
// --- a log read declares nothing ----------------------------------------
89+
order_qt_log_rows """select id, name, price from log_basic"""
90+
sameWithAdmissionOff("select id, name, price from log_basic order by id")
91+
92+
sql """set enable_jni_heap_admission = false"""
93+
sql """switch internal"""
94+
}

0 commit comments

Comments
 (0)