• Cargo.lock is excluded by !**/*.lock

• Cargo.toml (1 hunks)
• src/config/src/config.rs (2 hunks)
• src/handler/http/request/status/mod.rs (2 hunks)
• src/infra/src/cache/mod.rs (1 hunks)
• src/service/search/datafusion/exec.rs (8 hunks)
• src/service/search/datafusion/storage/file_list.rs (2 hunks)
• src/service/search/datafusion/storage/file_statistics_cache.rs (1 hunks)
• src/service/search/datafusion/storage/mod.rs (1 hunks)

• Cargo.toml
• src/config/src/config.rs
• src/infra/src/cache/mod.rs

src/service/search/datafusion/storage/mod.rs (1)

23-23: Successfully added file_statistics_cache module to enhance modularity.

This change promotes better separation of concerns and encapsulation within the system architecture.

src/service/search/datafusion/storage/file_statistics_cache.rs (1)

26-140: Implement robust caching mechanism with FileStatisticsCache.

The use of DashMap for concurrency and VecDeque for eviction policy is well thought out. Ensure that the cache eviction logic is thoroughly tested to prevent potential data loss or inconsistencies.

src/handler/http/request/status/mod.rs (1)

313-316: Enhance monitoring capabilities by adding file_statistics_cache statistics.

The integration of file_statistics_cache statistics into the cache_status function is a valuable addition for better visibility and monitoring of cache performance.

src/service/search/datafusion/exec.rs (5)

1240-1247: The with_collect_stat option has been added to the listing options for both JSON and PARQUET file types. Ensure this change aligns with the intended data collection and performance metrics.

This change should help in better statistics collection for performance tuning.

---

27-27: Ensure the new import schema_ext::SchemaExt is utilized effectively in the code.

Verification successful

---

The SchemaExt trait is indeed utilized in various parts of the codebase, including the file src/service/search/datafusion/exec.rs. This confirms that the new import schema_ext::SchemaExt is effectively used.

Scripts Executed

The following scripts were executed for the analysis:

---

Script:

#!/bin/bash
# Description: Verify the usage of `schema_ext::SchemaExt` throughout the codebase.

# Test: Search for the usage of `SchemaExt`. Expect: At least one usage in the code.
ast-grep --lang rust --pattern $'use $_ { $$$, SchemaExt, $$$ }'

Length of output: 65

---

---

Script:

#!/bin/bash
# Description: Verify the usage of `SchemaExt` throughout the codebase.

# Test: Search for any references to `SchemaExt`. Expect: At least one usage in the code.
rg 'SchemaExt'

Length of output: 2064

---

Line range hint 1141-1176: Review the configuration of the RuntimeEnv. Ensure that the memory pool settings and cache manager configurations are correctly applied based on the system's configuration.

Verification successful

---

The recent script outputs provide useful insights into the configuration of the RuntimeEnv, specifically regarding memory pool settings and cache manager configurations. Here are the key findings:

• Memory Pool Settings:

  • The memory_pool references in src/service/search/datafusion/exec.rs indicate that both GreedyMemoryPool and FairSpillPool are being configured based on the CONFIG.memory_cache.datafusion_memory_pool setting.
  • The datafusion_memory_pool configuration is defined in src/config/src/config.rs.

• Cache Manager Settings:

  • The cache_manager references in src/service/search/datafusion/exec.rs show that the CacheManagerConfig is being used and applied to rn_config.

These findings confirm that the memory pool settings and cache manager configurations are being correctly applied based on the system's configuration.

• src/service/search/datafusion/exec.rs:

  • Memory pool settings are configured with GreedyMemoryPool and FairSpillPool.
  • Cache manager settings are applied using CacheManagerConfig.

• src/config/src/config.rs:

  • The datafusion_memory_pool configuration is defined.

Based on this information, the review comment is verified.

Scripts Executed

The following scripts were executed for the analysis:

---

Script:

#!/bin/bash
# Description: Verify the correct application of memory pool settings and cache manager configurations.

# Test: Search for the usage of `RuntimeEnv` configuration. Expect: Correct settings based on system configuration.
ast-grep --lang rust --pattern $'let $_ = RuntimeEnv::new($_)'

Length of output: 63

---

---

Script:

#!/bin/bash
# Description: Verify the correct application of memory pool settings and cache manager configurations.

# Test: Search for the usage of `RuntimeEnv` configuration and related settings.

# Search for any instantiation or configuration of RuntimeEnv
ast-grep --lang rust --pattern $'let $_ = RuntimeEnv::new($_)'

# Search for any method calls on RuntimeEnv that might configure it
rg 'RuntimeEnv::with_'

# Search for any references to memory pool settings
rg 'memory_pool'

# Search for any references to cache manager settings
rg 'cache_manager'

Length of output: 1016

---

Line range hint 1256-1291: The handling of different storage types (Memory, Wal, Tmpfs) in the register_table function has been updated. Ensure that these changes do not affect the existing functionality and that the schema key generation and file list management are correctly implemented.

Verification successful

---

The register_table function in src/service/search/datafusion/exec.rs handles different storage types (Memory, Wal, Tmpfs) and ensures schema key generation and file list management are correctly implemented. The function also includes error handling for unsupported file types and URL parsing errors.

• The function correctly sets the prefix based on the storage type.
• It manages the file list for Memory and Wal storage types.
• It handles schema inference and caching based on configuration.

The changes appear to be correctly implemented without affecting existing functionality.

Scripts Executed

The following scripts were executed for the analysis:

---

Script:

#!/bin/bash
# Description: Verify the handling of different storage types in the `register_table` function.

# Test: Search for the usage of storage type handling in `register_table`. Expect: Correct implementation without affecting existing functionality.
ast-grep --lang rust --pattern $'let $_ = register_table($_, $_, $_, $_, $_, $_)'

Length of output: 82

---

---

Script:

#!/bin/bash
# Search for the definition of the `register_table` function in the codebase
ast-grep --lang rust --pattern 'fn register_table($_, $_, $_, $_, $_, $_) -> $_ { $$$ }'

# Search for the usage of the `register_table` function in the codebase
rg 'register_table\('

Length of output: 3508

---

---

Script:

#!/bin/bash
# Extract the definition of the `register_table` function from `src/service/search/datafusion/exec.rs`
ast-grep --lang rust --pattern 'pub async fn register_table($_, $_, $_, $_, $_, $_) -> $_ { $$$ }' src/service/search/datafusion/exec.rs

Length of output: 6298

---

45-45: The addition of CacheManagerConfig should be aligned with the overall caching strategy. Confirm that this configuration is properly integrated and utilized.