Skip to content

Commit 130ee02

Browse files
committed
Address scan I/O review and extract independent fixes
1 parent e32e4b5 commit 130ee02

9 files changed

Lines changed: 455 additions & 359 deletions

File tree

‎docs/source/user-guide/latest/metrics.md‎

Lines changed: 44 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -79,11 +79,50 @@ Here is a guide to some of the native metrics.
7979
| `spilled_bytes` | Actual bytes written to native shuffle spill files on disk. |
8080
| `memory_spilled_bytes` | Uncompressed Arrow backing-buffer and partition-index memory spilled. |
8181

82+
### Native Parquet scans
83+
84+
Native Parquet scans expose these counters in the Spark SQL metric map as well as native
85+
execution metrics. Counters accumulate per scan operator; they do not instrument individual rows.
86+
87+
| Metric | Description |
88+
| ------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
89+
| `scan_io_data_bytes` | Bytes returned to the Parquet reader for projected data-page ranges. |
90+
| `scan_io_metadata_bytes` | Bytes returned for footer prefetches, page indexes, and Bloom filters. A footer prefetch can also contain unused data bytes. |
91+
| `scan_io_footer_reads` | Storage reads of complete serialized footer payloads, counted once per metadata open. Plaintext payloads must decode successfully; encrypted payloads are counted before key retrieval/decryption, even if those later fail. |
92+
| `scan_io_footer_bytes` | Serialized footer payload bytes, excluding the final eight-byte trailer. These bytes are already included in `scan_io_metadata_bytes`. |
93+
| `scan_io_object_store_get_calls` | Nonempty GET operations at the native remote `ObjectStore` API, after range coalescing. Does not count transport retries or HEAD requests. |
94+
| `scan_io_object_store_get_requested_bytes` | Requested coalesced range bytes at that API. A failed or partly consumed request can contribute requested bytes without the same number of response bytes. |
95+
| `scan_io_object_store_response_bytes_read` | Response bytes actually consumed at that API, including bytes fetched between coalesced ranges. Not HTTP wire bytes. |
96+
| `scan_io_metadata_cache_hits` | Successful, cache-eligible metadata opens requiring no storage reads. |
97+
| `scan_io_metadata_cache_misses` | Successful, cache-eligible metadata opens requiring storage reads. Failed opens and encrypted opens, which bypass this shared cache, increment neither cache counter. |
98+
99+
Reader-level and object-store bytes are two views of the same reads; do not add them together.
100+
Likewise, footer bytes are a subset of metadata bytes, not a third reader-level category. A warm
101+
metadata-cache hit contributes no new metadata or footer I/O. A valid plaintext footer followed by
102+
a page-index failure still contributes footer bytes; an invalid plaintext footer does not.
103+
104+
Remote counters follow the backend selected during object-store construction, including native
105+
S3 (`s3`/`s3a`), GCS (`gs`), Azure (`az`, `adl`, `azure`, `abfs`, `abfss`), and HTTP(S) stores.
106+
Local files, in-memory stores, and HDFS/custom backends (including cloud-looking schemes selected
107+
through `fs.comet.libhdfs.schemes`) retain reader-level counters but have zero remote counters.
108+
The native cloud wrapper observes default range coalescing; custom `get_ranges` implementations
109+
require an explicit accounting contract before being composed with that wrapper.
110+
111+
For data-bearing scans, comparing remote response bytes with reader data bytes can reveal
112+
coalescing and metadata overhead. For a metadata-only scan, data bytes are zero: report the
113+
metadata and remote totals instead of dividing by zero. Cancellation can leave late asynchronous
114+
work outside the final metric snapshot; these counters are not a guarantee of complete network
115+
traffic accounting after cancellation.
116+
82117
## Task-Level Input Metrics on Spark 4.1+
83118

84-
Comet's native scans set `inputMetrics.bytesRead` to the actual file IO performed by the
85-
DataFusion parquet reader (`bytes_scanned`). This is the truthful number you would see at the
86-
filesystem layer.
119+
Comet's native scans populate `inputMetrics.bytesRead` from the existing `bytes_scanned`
120+
counter. It counts requested data/Bloom-filter ranges through the Parquet reader's byte-read
121+
methods, not all filesystem I/O. Footer and page-index reads through metadata loading bypass
122+
this counter, and range coalescing can fetch more bytes than the logical ranges request. The
123+
additional scan I/O metrics above expose those differences without changing `bytes_scanned`.
124+
The native `scan_efficiency_ratio` still uses `bytes_scanned` as its numerator and has the same
125+
blind spots; it is not the remote read-amplification ratio described above.
87126

88127
Spark 4.1 changed its own parquet reader to pre-open the `SeekableInputStream` and read the file
89128
footer outside the `FileScanRDD.compute()` thread. Spark's `inputMetrics.bytesRead` is updated
@@ -100,4 +139,5 @@ unaffected and remains exactly equal between Comet and Spark.
100139

101140
If you compare Comet's `bytesRead` against vanilla Spark's on Spark 4.1+ (via the Spark UI or
102141
the REST API), expect Comet's number to be substantially larger for small files, and closer to
103-
Spark's for large files. Comet's value reflects what the storage layer actually delivered.
142+
Spark's for large files in that workload. Neither metric should be interpreted as complete
143+
filesystem or network traffic accounting.

‎native/core/src/execution/jni_api.rs‎

Lines changed: 1 addition & 84 deletions
Original file line numberDiff line numberDiff line change
@@ -95,7 +95,6 @@ use std::time::{Duration, Instant};
9595
use std::{sync::Arc, task::Poll};
9696
use tokio::runtime::{Handle, Runtime};
9797
use tokio::sync::mpsc;
98-
use tokio::task::JoinHandle;
9998

10099
use crate::execution::memory_pools::{
101100
create_memory_pool, handle_task_shared_pool_release, parse_memory_pool_config, MemoryPoolConfig,
@@ -325,7 +324,6 @@ struct ExecutionContext {
325324
pub stream: Option<SendableRecordBatchStream>,
326325
/// Receives batches from a spawned tokio task (async I/O path)
327326
pub batch_receiver: Option<mpsc::Receiver<DataFusionResult<RecordBatch>>>,
328-
pub batch_producer: Option<JoinHandle<()>>,
329327
/// Native metrics
330328
pub metrics: Arc<Global<JObject<'static>>>,
331329
// The interval in milliseconds to update metrics
@@ -539,7 +537,6 @@ pub unsafe extern "system" fn Java_org_apache_comet_Native_createPlan(
539537
input_sources,
540538
stream: None,
541539
batch_receiver: None,
542-
batch_producer: None,
543540
metrics,
544541
metrics_update_interval,
545542
metrics_last_update_time: Instant::now(),
@@ -838,7 +835,7 @@ pub unsafe extern "system" fn Java_org_apache_comet_Native_executePlan(
838835
// decreasing to 1 would serialize production and consumption.
839836
let (tx, rx) = mpsc::channel(2);
840837
let mut stream = stream;
841-
let producer = get_runtime().spawn(async move {
838+
get_runtime().spawn(async move {
842839
let result = std::panic::AssertUnwindSafe(async {
843840
while let Some(batch) = stream.next().await {
844841
if tx.send(batch).await.is_err() {
@@ -865,7 +862,6 @@ pub unsafe extern "system" fn Java_org_apache_comet_Native_executePlan(
865862
}
866863
});
867864
exec_context.batch_receiver = Some(rx);
868-
exec_context.batch_producer = Some(producer);
869865
} else {
870866
exec_context.stream = Some(stream);
871867
}
@@ -970,11 +966,6 @@ pub extern "system" fn Java_org_apache_comet_Native_releasePlan(
970966
try_unwrap_or_throw(&e, |env| unsafe {
971967
let execution_context = get_execution_context(exec_context);
972968

973-
execution_context.batch_receiver.take();
974-
if let Some(producer) = execution_context.batch_producer.take() {
975-
stop_batch_producer(producer);
976-
}
977-
978969
// Update metrics
979970
update_metrics(env, execution_context)?;
980971

@@ -998,17 +989,6 @@ pub extern "system" fn Java_org_apache_comet_Native_releasePlan(
998989
})
999990
}
1000991

1001-
fn stop_batch_producer(producer: JoinHandle<()>) {
1002-
producer.abort();
1003-
let deadline = Instant::now() + Duration::from_millis(100);
1004-
while !producer.is_finished() {
1005-
if Instant::now() >= deadline {
1006-
return;
1007-
}
1008-
std::thread::sleep(Duration::from_millis(1));
1009-
}
1010-
}
1011-
1012992
fn update_metrics(env: &mut Env, exec_context: &mut ExecutionContext) -> CometResult<()> {
1013993
if let Some(native_query) = &exec_context.root_op {
1014994
let metrics = exec_context.metrics.as_obj();
@@ -1395,66 +1375,3 @@ pub unsafe extern "system" fn Java_org_apache_comet_Native_columnarToRowClose(
13951375
Ok(())
13961376
})
13971377
}
1398-
1399-
#[cfg(test)]
1400-
mod tests {
1401-
use super::*;
1402-
1403-
#[test]
1404-
fn waits_for_background_batch_producer_shutdown() {
1405-
let (sender, receiver) = std::sync::mpsc::channel::<()>();
1406-
let producer = get_runtime().spawn(async move {
1407-
let _sender = sender;
1408-
futures::future::pending::<()>().await;
1409-
});
1410-
1411-
stop_batch_producer(producer);
1412-
1413-
assert_eq!(
1414-
receiver.try_recv(),
1415-
Err(std::sync::mpsc::TryRecvError::Disconnected)
1416-
);
1417-
}
1418-
1419-
#[test]
1420-
fn does_not_wait_indefinitely_for_blocked_batch_producer() {
1421-
let (started_sender, started_receiver) = std::sync::mpsc::channel();
1422-
let (release_sender, release_receiver) = std::sync::mpsc::channel();
1423-
let producer = get_runtime().spawn(async move {
1424-
started_sender.send(()).unwrap();
1425-
release_receiver.recv().unwrap();
1426-
});
1427-
started_receiver.recv().unwrap();
1428-
1429-
let release_thread = std::thread::spawn(move || {
1430-
std::thread::sleep(Duration::from_millis(500));
1431-
release_sender.send(()).unwrap();
1432-
});
1433-
let started = Instant::now();
1434-
stop_batch_producer(producer);
1435-
let elapsed = started.elapsed();
1436-
release_thread.join().unwrap();
1437-
1438-
assert!(elapsed < Duration::from_millis(400));
1439-
}
1440-
1441-
#[test]
1442-
fn stops_finished_batch_producer_with_exhausted_runtime_budget() {
1443-
let (sender, receiver) = std::sync::mpsc::channel();
1444-
1445-
get_runtime().spawn(async move {
1446-
let producer = get_runtime().spawn(async {});
1447-
while !producer.is_finished() {
1448-
tokio::task::yield_now().await;
1449-
}
1450-
while tokio::task::coop::has_budget_remaining() {
1451-
tokio::task::coop::consume_budget().await;
1452-
}
1453-
1454-
stop_batch_producer(producer);
1455-
sender.send(()).unwrap();
1456-
});
1457-
1458-
assert_eq!(receiver.recv_timeout(Duration::from_millis(500)), Ok(()));
1459-
}
1460-
}

‎native/core/src/execution/operators/parquet_writer.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -311,7 +311,7 @@ impl ParquetWriterExec {
311311
#[cfg(feature = "hdfs-opendal")]
312312
{
313313
// Use prepare_object_store_with_configs to create and register the object store
314-
let (_object_store_url, object_store_path) = prepare_object_store_with_configs(
314+
let (_object_store_url, object_store_path, _) = prepare_object_store_with_configs(
315315
_runtime_env,
316316
output_file_path.to_string(),
317317
object_store_options,

‎native/core/src/execution/planner.rs‎

Lines changed: 9 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -94,7 +94,7 @@ use iceberg::expr::Bind;
9494
use crate::execution::operators::ExecutionError::GeneralError;
9595
use crate::execution::shuffle::{CometPartitioning, CompressionCodec};
9696
use crate::execution::spark_plan::SparkPlan;
97-
use crate::parquet::parquet_support::{is_hdfs_scheme, prepare_object_store_with_configs};
97+
use crate::parquet::parquet_support::prepare_object_store_with_configs;
9898
use datafusion::common::scalar::ScalarStructBuilder;
9999
use datafusion::common::{
100100
tree_node::{Transformed, TransformedResult, TreeNode, TreeNodeRecursion, TreeNodeRewriter},
@@ -1631,13 +1631,12 @@ impl PhysicalPlanner {
16311631
.iter()
16321632
.map(|(k, v)| (k.clone(), v.clone()))
16331633
.collect();
1634-
let is_hdfs_object_store = url::Url::parse(&one_file)
1635-
.is_ok_and(|url| is_hdfs_scheme(&url, &object_store_options));
1636-
let (object_store_url, _) = prepare_object_store_with_configs(
1637-
self.session_ctx.runtime_env(),
1638-
one_file,
1639-
&object_store_options,
1640-
)?;
1634+
let (object_store_url, _, object_store_backend) =
1635+
prepare_object_store_with_configs(
1636+
self.session_ctx.runtime_env(),
1637+
one_file,
1638+
&object_store_options,
1639+
)?;
16411640

16421641
// Get files for this partition
16431642
let files = self.get_partitioned_files(partition_files)?;
@@ -1648,7 +1647,7 @@ impl PhysicalPlanner {
16481647
Some(data_schema),
16491648
Some(partition_schema),
16501649
object_store_url,
1651-
is_hdfs_object_store,
1650+
object_store_backend,
16521651
file_groups,
16531652
Some(projection_vector),
16541653
Some(data_filters?),
@@ -1686,7 +1685,7 @@ impl PhysicalPlanner {
16861685
.and_then(|f| f.partitioned_file.first())
16871686
.map(|f| f.file_path.clone())
16881687
.ok_or(GeneralError("Failed to locate file".to_string()))?;
1689-
let (object_store_url, _) = prepare_object_store_with_configs(
1688+
let (object_store_url, _, _) = prepare_object_store_with_configs(
16901689
self.session_ctx.runtime_env(),
16911690
one_file,
16921691
&object_store_options,

0 commit comments

Comments
 (0)