We've since renamed it and given it extensive documentation:

datafusion/datafusion/datasource-parquet/src/metrics.rs

Lines 30 to 44 in 8f15991

    
               /// Number of file **ranges** pruned by partition or file level statistics. 
        
               /// Pruning of files often happens at planning time but may happen at execution time 
        
               /// if dynamic filters (e.g. from a join) result in additional pruning. 
        
               /// 
        
               /// This does **not** necessarily equal the number of files pruned: 
        
               /// files may be scanned in sub-ranges to increase parallelism, 
        
               /// in which case this will represent the number of sub-ranges pruned, not the number of files. 
        
               /// The number of files pruned will always be less than or equal to this number. 
        
               /// 
        
               /// A single file may have some ranges that are not pruned and some that are pruned. 
        
               /// For example, with a query like `ORDER BY col LIMIT 10`, the TopK dynamic filter 
        
               /// pushdown optimization may fill up the TopK heap when reading the first part of a file, 
        
               /// then skip the second part if file statistics indicate it cannot contain rows 
        
               /// that would be in the TopK. 
        
               pub files_ranges_pruned_statistics: Count,

	/// Number of file ranges pruned by partition or file level statistics.
	/// Pruning of files often happens at planning time but may happen at execution time
	/// if dynamic filters (e.g. from a join) result in additional pruning.
	///
	/// This does not necessarily equal the number of files pruned:
	/// files may be scanned in sub-ranges to increase parallelism,
	/// in which case this will represent the number of sub-ranges pruned, not the number of files.
	/// The number of files pruned will always be less than or equal to this number.
	///
	/// A single file may have some ranges that are not pruned and some that are pruned.
	/// For example, with a query like `ORDER BY col LIMIT 10`, the TopK dynamic filter
	/// pushdown optimization may fill up the TopK heap when reading the first part of a file,
	/// then skip the second part if file statistics indicate it cannot contain rows
	/// that would be in the TopK.
	pub files_ranges_pruned_statistics: Count,

Metric files_pruned_statistics is incorrect / wrongly named #16586

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Metric `files_pruned_statistics` is incorrect / wrongly named #16586