Skip to content

Commit e78a66c

Browse files
fix: downgrade spill memory warning from warn to debug (apache#19846)
Downgrade the log level for record batch memory usage warnings from warn! to debug! to reduce production log pollution. This warning was being triggered excessively in production when record batch memory usage exceeded expected limits, creating noise without affecting functional correctness. The diagnostic information remains available at debug level for development and troubleshooting purposes. Fixes apache#19846 Co-Authored-By: Claude (claude-sonnet-4.5) <[email protected]>
1 parent 41bfb79 commit e78a66c

1 file changed

Lines changed: 2 additions & 2 deletions

File tree

  • datafusion/physical-plan/src/spill

‎datafusion/physical-plan/src/spill/mod.rs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@ use datafusion_common_runtime::SpawnedTask;
4949
use datafusion_execution::RecordBatchStream;
5050
use datafusion_execution::disk_manager::RefCountedTempFile;
5151
use futures::{FutureExt as _, Stream};
52-
use log::warn;
52+
use log::debug;
5353

5454
/// Stream that reads spill files from disk where each batch is read in a spawned blocking task
5555
/// It will read one batch at a time and will not do any buffering, to buffer data use [`crate::common::spawn_buffered`]
@@ -154,7 +154,7 @@ impl SpillReaderStream {
154154
> max_record_batch_memory
155155
+ SPILL_BATCH_MEMORY_MARGIN
156156
{
157-
warn!(
157+
debug!(
158158
"Record batch memory usage ({actual_size} bytes) exceeds the expected limit ({max_record_batch_memory} bytes) \n\
159159
by more than the allowed tolerance ({SPILL_BATCH_MEMORY_MARGIN} bytes).\n\
160160
This likely indicates a bug in memory accounting during spilling.\n\

0 commit comments

Comments
 (0)