Skip to content

Commit 82ff679

Browse files
morningmanclaude
andcommitted
[fix](test)(file-cache) sum file_cache_statistics metrics across cache paths to deflake hit/read count checks
information_schema.file_cache_statistics reports one row per (cache_path, metric_name): the BE iterates every cache instance and a given data file is routed to exactly ONE instance via hash(filename) % num_paths. The tests read each metric with a bare `... limit 1`, so they inspect a single arbitrary path's counter, which need not be the path the query's data file routed to. With >1 file_cache_path this made total_hit_counts/total_read_counts assertions flaky (the inspected path never moved while the routed path did), e.g. test_file_cache_statistics failing with "total_hit_counts did not increase" (4499 -> 4499) on a 2-path backend. - test_file_cache_statistics: read every metric via SUM(CAST(METRIC_VALUE AS DOUBLE)) across all paths instead of `limit 1`, and replace the fixed Thread.sleep loops with Awaitility polling (total_*_counts are live bvar reads needing no wait; hits_ratio*/queue-curr are republished by the background monitor and are polled up to a couple of intervals). The counter-increase check re-runs the just-cached query inside the poll so a transient miss/eviction self-heals. - test_file_cache_query_limit: sum total_hit_counts/total_read_counts across paths via a cacheMetricSum helper. Its normal_queue/queryCacheCapacity logic is intentionally left per-instance, as file_cache_query_limit_percent is derived from a single path's brpc file_cache_capacity. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
1 parent 0675e24 commit 82ff679

2 files changed

Lines changed: 152 additions & 122 deletions

File tree

‎regression-test/suites/external_table_p0/cache/test_file_cache_query_limit.groovy‎

Lines changed: 22 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,17 @@ suite("test_file_cache_query_limit", "external_docker,hive,external_docker_hive,
9191
}
9292
}
9393

94+
// Sum a file_cache_statistics metric across ALL cache paths. file_cache_statistics reports one
95+
// row per (cache_path, metric_name); a single data file routes to exactly one path, so reading
96+
// a single arbitrary path's row with "limit 1" can miss a counter that moved on another path.
97+
// Used for cluster-wide absolute counters (total_hit_counts / total_read_counts). METRIC_VALUE
98+
// is a numeric string (std::to_string(double)), so CAST(... AS DOUBLE) is safe.
99+
def cacheMetricSum = { String metricName ->
100+
def r = sql """select sum(cast(METRIC_VALUE as double)) from information_schema.file_cache_statistics
101+
where METRIC_NAME = '${metricName}';"""
102+
return (r.size() == 0 || r[0][0] == null) ? null : Double.valueOf(r[0][0].toString())
103+
}
104+
94105
sql """drop catalog if exists ${catalog_name} """
95106

96107
sql """CREATE CATALOG ${catalog_name} PROPERTIES (
@@ -306,18 +317,14 @@ suite("test_file_cache_query_limit", "external_docker,hive,external_docker_hive,
306317
"elements: ${initialNormalQueueMaxElements}")
307318

308319
// ===== Hit And Read Counts Metrics Check =====
309-
// Get initial values for hit and read counts
310-
def initialTotalHitCountsResult = sql """select METRIC_VALUE from information_schema.file_cache_statistics
311-
where METRIC_NAME = 'total_hit_counts' limit 1;"""
312-
logger.info("Initial total_hit_counts result: " + initialTotalHitCountsResult)
313-
314-
def initialTotalReadCountsResult = sql """select METRIC_VALUE from information_schema.file_cache_statistics
315-
where METRIC_NAME = 'total_read_counts' limit 1;"""
316-
logger.info("Initial total_read_counts result: " + initialTotalReadCountsResult)
317-
318-
// Store initial values
319-
double initialTotalHitCounts = Double.valueOf(initialTotalHitCountsResult[0][0])
320-
double initialTotalReadCounts = Double.valueOf(initialTotalReadCountsResult[0][0])
320+
// total_hit_counts / total_read_counts are cluster-wide LIVE counters reported per cache-path
321+
// (one row per path). A data file routes to exactly one path, so a bare "limit 1" may inspect a
322+
// path the query never touched and miss the increment (this is what made the sibling
323+
// test_file_cache_statistics flaky). Sum across all paths so the totals always include whichever
324+
// path(s) the query's files routed to.
325+
double initialTotalHitCounts = cacheMetricSum('total_hit_counts')
326+
double initialTotalReadCounts = cacheMetricSum('total_read_counts')
327+
logger.info("Initial total_hit_counts (sum): ${initialTotalHitCounts}, total_read_counts (sum): ${initialTotalReadCounts}")
321328

322329
// Set backend configuration parameters for file_cache_query_limit test 1
323330
setBeConfigTemporary([
@@ -376,18 +383,9 @@ suite("test_file_cache_query_limit", "external_docker,hive,external_docker_hive,
376383
assertTrue((updatedNormalQueueCurrSize as Long) <= queryCacheCapacity,
377384
NORMAL_QUEUE_CURR_SIZE_GREATER_THAN_QUERY_CACHE_CAPACITY_MSG)
378385

379-
// Get updated values for hit and read counts after cache operations
380-
def updatedTotalHitCountsResult = sql """select METRIC_VALUE from information_schema.file_cache_statistics
381-
where METRIC_NAME = 'total_hit_counts' limit 1;"""
382-
logger.info("Updated total_hit_counts result: " + updatedTotalHitCountsResult)
383-
384-
def updatedTotalReadCountsResult = sql """select METRIC_VALUE from information_schema.file_cache_statistics
385-
where METRIC_NAME = 'total_read_counts' limit 1;"""
386-
logger.info("Updated total_read_counts result: " + updatedTotalReadCountsResult)
387-
388-
// Check if updated values are greater than initial values
389-
double updatedTotalHitCounts = Double.valueOf(updatedTotalHitCountsResult[0][0])
390-
double updatedTotalReadCounts = Double.valueOf(updatedTotalReadCountsResult[0][0])
386+
// Get updated values for hit and read counts after cache operations (summed across paths)
387+
double updatedTotalHitCounts = cacheMetricSum('total_hit_counts')
388+
double updatedTotalReadCounts = cacheMetricSum('total_read_counts')
391389

392390
logger.info("Total hit and read counts comparison - hit counts: ${initialTotalHitCounts} -> " +
393391
"${updatedTotalHitCounts} , read counts: ${initialTotalReadCounts} -> ${updatedTotalReadCounts}")

0 commit comments

Comments
 (0)