fix: optimise file list fetch from db #4573
Conversation
WalkthroughThe changes introduce new configuration fields to the Changes
Possibly related PRs
Suggested labels
Recent review detailsConfiguration used: CodeRabbit UI Files selected for processing (4)
Files skipped from review as they are similar to previous changes (4)
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
Outside diff range and nitpick comments (2)
src/infra/src/file_list/sqlite.rs (1)
Line range hint
315-345: Approve the changes to conditionally apply an upper bound to themax_tsvalue.The changes introduce a new configuration option
use_upper_bound_for_max_tsto conditionally apply an upper bound to themax_tsvalue in the SQL query. This provides more flexibility in querying data based on the configuration settings and can potentially improve the accuracy of the retrieved data.Suggestions:
- Add unit tests to cover the new functionality and ensure the expected behavior.
- Document the new configuration option and its impact on the querying behavior.
src/config/src/config.rs (1)
971-976: Consider making the default value configurable based on usage patterns.The new
upper_bound_for_max_tsfield looks good overall. The naming, type, and help text are all appropriate.The default value of
60minutes for the buffer seems like a reasonable starting point. However, consider whether this default should be made configurable based on the system's usage patterns and requirements.
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (4)
- src/config/src/config.rs (1 hunks)
- src/infra/src/file_list/mysql.rs (3 hunks)
- src/infra/src/file_list/postgres.rs (19 hunks)
- src/infra/src/file_list/sqlite.rs (3 hunks)
Additional context used
Path-based instructions (4)
src/infra/src/file_list/sqlite.rs (1)
Pattern
**/*.rs: You are a smart rustlang pull request reviewer.
You are going to review all the rustlang files.
Be concise, and add a brief explanation to your suggestionsMake sure the responses are not too verbose and keep the suggestions to the point i.e. actionable.
src/infra/src/file_list/postgres.rs (1)
Pattern
**/*.rs: You are a smart rustlang pull request reviewer.
You are going to review all the rustlang files.
Be concise, and add a brief explanation to your suggestionsMake sure the responses are not too verbose and keep the suggestions to the point i.e. actionable.
src/infra/src/file_list/mysql.rs (1)
Pattern
**/*.rs: You are a smart rustlang pull request reviewer.
You are going to review all the rustlang files.
Be concise, and add a brief explanation to your suggestionsMake sure the responses are not too verbose and keep the suggestions to the point i.e. actionable.
src/config/src/config.rs (1)
Pattern
**/*.rs: You are a smart rustlang pull request reviewer.
You are going to review all the rustlang files.
Be concise, and add a brief explanation to your suggestionsMake sure the responses are not too verbose and keep the suggestions to the point i.e. actionable.
Additional comments not posted (29)
src/infra/src/file_list/postgres.rs (24)
20-20: LGTM!The
get_configfunction is imported correctly from theconfigmodule.
Line range hint
1-1:
Line range hint
1-1:
107-112: Looks good!The error handling for
sqlx::Error::RowNotFoundis now more clearly structured with braces, improving code readability. The logic remains unaltered.
Line range hint
1-1:
189-197: Looks good!The SQL query in
batch_remove_deletedis reformatted with line breaks for better readability. The query logic remains unaltered.
200-205: Looks good!The error handling for
sqlx::Error::RowNotFoundis now more clearly structured with braces, improving code readability. The logic remains unaltered.
233-243: Looks good!The SQL query in
getis reformatted with line breaks and indentation for better readability. The query logic remains unaltered.
Line range hint
1-1:
Line range hint
1-1:
Line range hint
1-1:
304-347: New configuration-dependent behavior introduced in querying logic.The changes introduce a new configuration check to conditionally apply an upper bound filter on the
max_tstimestamp during queries. This behavior could impact how data is retrieved and processed.Please ensure that:
- The upper bound filter is applied correctly and produces the expected results.
- The changes are thoroughly tested under different configuration settings to validate the behavior.
If no issues are found during testing, the changes can be approved.
370-377: Looks good!The SQL query in
query_deletedis reformatted with line breaks and indentation for better readability. The query logic remains unaltered.
Line range hint
1-1:
Line range hint
1-1:
443-443: Looks good!The additional line break in the SQL query string improves readability without altering the query logic.
Line range hint
1-1:
Line range hint
1-1:
553-570: Looks good!The SQL query in
set_stream_statsis reformatted with line breaks and indentation for better readability. The query logic remains unaltered.
587-591: Looks good!The SQL query in
reset_stream_statsis reformatted with line breaks and indentation for better readability. The query logic remains unaltered.
Line range hint
1-1:
Line range hint
1-1:
Line range hint
1-1:
Line range hint
1-1:src/infra/src/file_list/mysql.rs (4)
20-20: LGTM!The import statement for the
get_configfunction looks good.
304-322: The configuration-based query optimization looks good!The code segment introduces a configuration-driven optimization to the SQL query by applying an upper bound to the
max_tstimestamp when specified. The upper bound calculation and the corresponding SQL query modifications are implemented correctly.The original query logic is preserved when the upper bound is not required, ensuring backward compatibility.
336-336: LGTM!The closing brace is correctly placed to complete the
ifblock.
305-307: The configuration retrieval and usage look good!The code segment correctly retrieves the configuration using the
get_config()function and checks thelimit.use_upper_bound_for_max_tsflag to determine whether to apply the upper bound optimization.The
limit.upper_bound_for_max_tsconfiguration value is used to calculate the upper bound in microseconds.src/config/src/config.rs (1)
965-970: LGTM!The new boolean field
use_upper_bound_for_max_tslooks good. The naming, type, default value, and help text are all appropriate.
oasisk
force-pushed
the
fix-optimize-file-list-fetch-from-db
branch
from
381c8b4 to
5698fff
Compare
Summary by CodeRabbit
New Features
Bug Fixes
Documentation