Greptile Summary

This PR introduces a significant performance optimization for WAL (Write-Ahead Log) parquet file scanning by implementing time-range-aware directory filtering. The key enhancement is the addition of two new functions in src/config/src/utils/file.rs:

1. wal_dir_datetime_filter_builder: Creates datetime-based path filters that can parse directory structures with embedded dates (YYYY/MM/DD/HH format) and determine if directories fall within a specified time range
2. scan_files_filtered: An async directory walker that applies filters during traversal, allowing early pruning of directory branches that don't match the criteria

The core problem being solved is that the existing scan_files function was gathering all nested file paths before applying any filtering, causing 8+ second delays when searching through 130K+ files even for recent data queries like "last 15 minutes". The new implementation leverages OpenObserve's hierarchical date-based WAL storage structure to skip entire directory trees that fall outside the query time range.

In src/service/search/grpc/wal.rs, the get_file_list_inner function now uses scan_files_filtered instead of the original scan_files, applying datetime filters when time ranges are present in the query. When no time range is specified, it falls back to an unfiltered scan, maintaining backward compatibility.

The filtering logic works by parsing date components from directory paths, handling partial paths gracefully, and using breadth-first traversal to make early pruning decisions. This approach is particularly effective for time-series data storage patterns where files are organized chronologically in the filesystem.

Confidence score: 4/5

• This PR addresses a clear performance bottleneck with a well-architected solution that should significantly improve search times
• The implementation correctly handles edge cases like partial date paths and maintains backward compatibility
• The async implementation and proper error handling demonstrate good engineering practices, though the PathBuf cloning in the async context could be monitored for performance impact

_{2 files reviewed, no comments}

_{Edit Code Review Bot Settings | Greptile}