perf: disable repartition when sort by timestamp desc #3918

haohuaijin · 2024-07-08T02:08:56Z

Summary by CodeRabbit

Chores
- Updated Rust toolchain setup in .github/workflows/unit-tests.yml and rust-toolchain.toml.
Dependencies
- Updated serde version in Cargo.toml.
- Updated gxhash version specification in src/config/Cargo.toml.
Internal Improvements
- Enhanced context creation and data fusion preparation for improved performance and flexibility.
- Improved file group splitting logic based on CPU configurations for optimized resource usage.

coderabbitai · 2024-07-08T02:09:04Z

Walkthrough

This update involves significant changes, primarily focused on modifying function signatures to incorporate additional parameters, such as sorting options, and refactoring import statements for clarity. Notable updates include a revised handling of file group splitting based on CPU configuration and adjustments to DataFusion context preparation. Changes also extend to dependency upgrades, specifically in serde and gxhash, as well as updates in Rust toolchain versions.

Changes

File(s)	Change Summary
`src/.../grpc/storage.rs`, `src/.../grpc/wal.rs`	Added parameters to `create_context` and `prepare_datafusion_context` function calls.
`src/.../exec.rs`	Added `sort_by_timestamp_desc` parameter to `prepare_datafusion_context` and modified related functions. Adjusted `register_table` function to use `sort_key`.
`src/.../request/search/mod.rs`, `src/.../multi_streams.rs`	Reformatted import statements related to HTTP request handling functions.
`src/.../table_provider/mod.rs`	Updated file group splitting logic based on CPU configuration. Added new functions `repartition_sorted_groups` and `find_max_group_index`.
`.github/.../unit-tests.yml`, `rust-toolchain.toml`	Changed Rust toolchain version from `nightly-2024-07-08` to `nightly-2024-03-02`.
`Cargo.toml`, `src/config/Cargo.toml`	Updated version specifications for `serde` and `gxhash` dependencies.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai generate interesting stats about this repository and render them as a table.
- @coderabbitai show all the console.log statements in this repository.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

coderabbitai

Actionable comments posted: 0

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 4bff0be and 722293d.

Files selected for processing (3)

src/service/promql/search/grpc/storage.rs (1 hunks)
src/service/promql/search/grpc/wal.rs (2 hunks)
src/service/search/datafusion/exec.rs (10 hunks)

Files skipped from review due to trivial changes (1)

src/service/promql/search/grpc/storage.rs

Additional context used

Path-based instructions (2)

src/service/promql/search/grpc/wal.rs (1)

Pattern **/*.rs: You are a smart rustlang pull request reviewer.
You are going to review all the rustlang files.
Be concise, and add a brief explanation to your suggestions

Make sure the responses are not too verbose and keep the suggestions to the point i.e. actionable.

src/service/search/datafusion/exec.rs (1)

Pattern **/*.rs: You are a smart rustlang pull request reviewer.
You are going to review all the rustlang files.
Be concise, and add a brief explanation to your suggestions

Make sure the responses are not too verbose and keep the suggestions to the point i.e. actionable.

Additional comments not posted (5)

src/service/promql/search/grpc/wal.rs (1)

190-191: LGTM! The function call update is correct.

The addition of the empty array &[] as an argument in the register_table function call is appropriate.

src/service/search/datafusion/exec.rs (4)

118-118: LGTM! The function call update is correct.

The addition of the &sql.meta.order_by parameter in the register_table function call is appropriate.

1217-1218: LGTM! The function signature update is correct.

The addition of the sort_by_timestamp_desc parameter in the prepare_datafusion_context function signature is appropriate.

1125-1128: LGTM! The function signature update is correct.

The addition of the sort_by_timestamp_desc parameter in the create_session_config function signature is appropriate.

1268-1274: LGTM! The function signature update is correct.

The addition of the sort_key parameter in the register_table function signature is appropriate.

coderabbitai

Actionable comments posted: 1

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 722293d and ec58083.

Files selected for processing (3)

src/handler/http/request/search/mod.rs (1 hunks)
src/handler/http/request/search/multi_streams.rs (1 hunks)
src/service/search/datafusion/exec.rs (10 hunks)

Files skipped from review due to trivial changes (2)

src/handler/http/request/search/mod.rs
src/handler/http/request/search/multi_streams.rs

Additional context used

Path-based instructions (1)

src/service/search/datafusion/exec.rs (1)

Pattern **/*.rs: You are a smart rustlang pull request reviewer.
You are going to review all the rustlang files.
Be concise, and add a brief explanation to your suggestions

Make sure the responses are not too verbose and keep the suggestions to the point i.e. actionable.

Additional comments not posted (4)

src/service/search/datafusion/exec.rs (4)
Line range hint 1125-1142:
LGTM! Changes align with the PR objective.

The changes to include the sort_by_timestamp_desc parameter and conditionally disable repartition and batch coalescing look good.

1218-1220: LGTM! Changes align with the PR objective.

The changes to include the sort_by_timestamp_desc parameter and pass it to create_session_config look good.

1269-1280: LGTM! Changes align with the PR objective.

The changes to include the sort_key parameter and determine if sorting by timestamp in descending order is needed look good.

122-127: LGTM! But verify the new parameter usage.

The change to include the sort_by_timestamp_desc parameter in the prepare_datafusion_context function call looks good.

However, ensure that all function calls to prepare_datafusion_context in the codebase match the new signature.

Verification successful

To ensure all calls to prepare_datafusion_context match the new signature, let's extract the relevant lines from the mentioned files and verify their correctness.

All calls to prepare_datafusion_context match the new signature.

The sort_by_timestamp_desc parameter has been correctly integrated into all relevant function calls.

src/service/search/datafusion/exec.rs

src/service/promql/search/grpc/wal.rs
Scripts executed

The following scripts were executed for the analysis:

Script:
#!/bin/bash
# Description: Verify all function calls to `prepare_datafusion_context` match the new signature.

# Test: Search for the function usage. Expect: Only occurrences of the new signature.
rg --type rust $'prepare_datafusion_context'
Length of output: 934

Script:
#!/bin/bash
# Extract lines with `prepare_datafusion_context` function definition and calls to verify the new signature.

# Extract function definition and calls in `src/service/search/datafusion/exec.rs`
rg --type rust --context 5 'prepare_datafusion_context' src/service/search/datafusion/exec.rs

# Extract function definition and calls in `src/service/promql/search/grpc/wal.rs`
rg --type rust --context 5 'prepare_datafusion_context' src/service/promql/search/grpc/wal.rs
Length of output: 3452

src/service/search/datafusion/exec.rs

coderabbitai

Actionable comments posted: 0

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between ec58083 and a579edc.

Files selected for processing (1)

src/service/search/datafusion/table_provider/mod.rs (3 hunks)

Additional context used

Path-based instructions (1)

src/service/search/datafusion/table_provider/mod.rs (1)

Pattern **/*.rs: You are a smart rustlang pull request reviewer.
You are going to review all the rustlang files.
Be concise, and add a brief explanation to your suggestions

Make sure the responses are not too verbose and keep the suggestions to the point i.e. actionable.

Additional comments not posted (4)

src/service/search/datafusion/table_provider/mod.rs (4)

37-37: Import statement is necessary for new functionality.

The get_config import is required for retrieving CPU configuration in the newly added logic.

252-264: New logic for splitting file groups based on CPU number is appropriate.

The logic aligns with the PR objective of improving performance by optimizing partitioning based on available CPU resources.

342-381: Function split_groups_by_cpu_num effectively splits file groups based on CPU number.

The function uses a straightforward even-odd strategy to split larger groups into smaller ones until the desired number of partitions is achieved.

383-394: Function find_max_group_index correctly identifies the group with the most files.

The function is a necessary helper for split_groups_by_cpu_num to determine which group to split next.

coderabbitai

Actionable comments posted: 0

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between a579edc and ec3628b.

Files selected for processing (1)

src/service/search/datafusion/table_provider/mod.rs (3 hunks)

Files skipped from review as they are similar to previous changes (1)

src/service/search/datafusion/table_provider/mod.rs

coderabbitai

Actionable comments posted: 0

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between ec3628b and 28eab0b.

Files selected for processing (1)

src/service/search/datafusion/exec.rs (10 hunks)

Files skipped from review as they are similar to previous changes (1)

src/service/search/datafusion/exec.rs

coderabbitai

Actionable comments posted: 0

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 28eab0b and 270e03d.

Files selected for processing (6)

.github/workflows/unit-tests.yml (1 hunks)
Cargo.toml (1 hunks)
rust-toolchain.toml (1 hunks)
src/config/Cargo.toml (1 hunks)
src/service/search/datafusion/exec.rs (10 hunks)
src/service/search/datafusion/table_provider/mod.rs (2 hunks)

Files skipped from review due to trivial changes (4)

.github/workflows/unit-tests.yml
Cargo.toml
rust-toolchain.toml
src/config/Cargo.toml

Files skipped from review as they are similar to previous changes (2)

src/service/search/datafusion/exec.rs
src/service/search/datafusion/table_provider/mod.rs

perf: disable repartition when sort by timestamp desc

fb471ac

Merge branch 'main' into config_change

722293d

coderabbitai bot reviewed Jul 8, 2024

View reviewed changes

haohuaijin added 2 commits July 8, 2024 10:27

retrigger ci

6901196

fix: disable coalesce

ec58083

coderabbitai bot reviewed Jul 8, 2024

View reviewed changes

src/service/search/datafusion/exec.rs Outdated Show resolved Hide resolved

perf: split group to meet cpu_num

a579edc

coderabbitai bot reviewed Jul 8, 2024

View reviewed changes

fix

ec3628b

coderabbitai bot reviewed Jul 8, 2024

View reviewed changes

Merge branch 'main' into config_change

28eab0b

coderabbitai bot reviewed Jul 8, 2024

View reviewed changes

fix: format code

270e03d

coderabbitai bot reviewed Jul 8, 2024

View reviewed changes

hengfeiyang approved these changes Jul 8, 2024

View reviewed changes

hengfeiyang merged commit 5572cb3 into main Jul 8, 2024

hengfeiyang deleted the config_change branch July 8, 2024 11:05

hengfeiyang added a commit that referenced this pull request Jul 8, 2024

feat: sync the PR #3918 #3917

5ffb27d

taimingl pushed a commit that referenced this pull request Jul 12, 2024

perf: disable repartition when sort by timestamp desc (#3918)

91bb740

coderabbitai bot mentioned this pull request Jan 6, 2025

feat: batch read data to avoid metrics OOM #5584

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

perf: disable repartition when sort by timestamp desc #3918

perf: disable repartition when sort by timestamp desc #3918

Uh oh!

haohuaijin commented Jul 8, 2024 •

edited by coderabbitai bot

Loading

Uh oh!

coderabbitai bot commented Jul 8, 2024 •

edited

Loading

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (`.coderabbit.yaml`)

Documentation and Community

Uh oh!

coderabbitai bot left a comment

Uh oh!

coderabbitai bot left a comment

Uh oh!

Uh oh!

coderabbitai bot left a comment

Uh oh!

coderabbitai bot left a comment

Uh oh!

coderabbitai bot left a comment

Uh oh!

coderabbitai bot left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

perf: disable repartition when sort by timestamp desc #3918

perf: disable repartition when sort by timestamp desc #3918

Uh oh!

Conversation

haohuaijin commented Jul 8, 2024 • edited by coderabbitai bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

coderabbitai bot commented Jul 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (.coderabbit.yaml)

Documentation and Community

Uh oh!

coderabbitai bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

coderabbitai bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai bot left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

haohuaijin commented Jul 8, 2024 •

edited by coderabbitai bot

Loading

coderabbitai bot commented Jul 8, 2024 •

edited

Loading

CodeRabbit Configration File (`.coderabbit.yaml`)