feat: secondary index #3870

hengfeiyang · 2024-07-01T14:52:46Z

New ENV

ZO_ENABLE_INVERTED_INDEX=true: enabled inverted index by default.
ZO_COMPACT_STRATEGY=file_time: file merge strategy, support: file_size and file_time, default is file_time.
ZO_FEATURE_QUERY_WITHOUT_INDEX=false: for debugging, we can set it to true
ZO_INVERTED_INDEX_OLD_FORMAT=false: enable it will use inverted index v1 stream name format.

API changes:

PUT /api/{org}/streams/{stream}/settings

{
	"partition_keys": [],
	"full_text_search_keys": [
		"content"
	],
	"index_fields": [
		"f3"
	],
	"bloom_filter_fields": [],
	"defined_schema_fields": [],
	"data_retention": 3000
}

We added index_fields as the secondary index fields.

TODO

Need add secondary index on stream setting page.
It is same to inverted index, but you can't both choose inverted index and secondary index, others can multiple select.

How it works

with secondary index and enabled ZO_ENABLE_INVERTED_INDEX=true we will build an extra index file for logs stream, then when we search for where idx_field='abc' we will use index to accelerate the query.

Limitation

Only support for text field. data_type: Utf8
Only support for new stream, because old data have no index data, enabled index field on old stream then old data can't be searched.

Summary by CodeRabbit

Dependency Updates
- Updated versions and repositories for various dependencies including datafusion, arrow, arrow-json, arrow-schema, and parquet.
New Features
- Added support for Binary data type conversion from hex strings to binary data.
- Introduced configuration flags for enabling/disabling inverted indexes with old format support.
- Added functionality for creating and handling segment_ids in various components.
Improved Error Handling
- Enhanced error logging and response generation for query parsing in HTTP requests.
Performance Improvements
- Refactored logic in several modules to optimize schema handling and avoid duplicate inserts.
Bug Fixes
- Resolved potential issues with duplicate configuration inserts and improved metadata population processes.
Tests
- Added new test cases for schema comparisons and hash key assertions.

coderabbitai · 2024-07-01T14:53:10Z

Walkthrough

The recent changes introduced new features and made several updates throughout the codebase. Key updates include dependency version bumps, enhanced configuration handling, additional fields in data structures, and modifications to indexing and schema processing. Some functions and configurations were refactored to handle the new inverted indexes and segment IDs more efficiently. These enhancements collectively aim to improve performance, flexibility, and maintainability.

Changes

Files/Modules	Change Summary
`Cargo.toml`	Updated versions and repositories for dependencies like `datafusion`, `arrow`, `parquet`, etc.
`src/common/utils/stream.rs`	Updated `populate_file_meta` to accept `min_field` and `max_field` as optional parameters instead of accessing configuration directly.
`src/config/**`	Added `bitvec` dependency; updated configurations in `Common` struct and `MemoryTable` settings; introduced `bitvec` module and updated stream settings.
`src/handler/**`	Enhanced request handling for gRPC and HTTP with additional logging, error handling, and new field initializations.
`src/infra/**`	Refactored schema-related functions to handle `StreamSettings` and updated imports accordingly.
`src/ingester/**`	Modified batching and partition implementations to incorporate stream settings, indexing fields, and segment IDs.
`src/job/files/**`	Adjusted file meta population, added constants, and improved schema handling during file processing.
`src/proto/proto/cluster/common.proto`	Added the `segment_ids` field to the `FileKey` message.
`src/service/**`	Updated compact and storage services with new field implementations, restructured imports, and enhanced data processing functions.
`src/utils/**`	Improved record batch conversion from JSON, added test modules for schema hash.

Sequence Diagram(s)

New Flow with Segment IDs and Inverted Indexes

sequenceDiagram
    participant User
    participant HTTPHandler
    participant FileService
    participant SchemaService
    participant IngestService

    User->>HTTPHandler: Make Request
    HTTPHandler->>FileService: getFileMeta(min_field, max_field)
    FileService->>SchemaService: getSchemaSettings(schema)
    SchemaService-->>FileService: Return Settings
    FileService-->>HTTPHandler: Return File Meta
    HTTPHandler-->>User: Send Response
    
    User->>IngestService: Send Data
    IngestService->>SchemaService: getStreamSettings(stream_type)
    SchemaService-->>IngestService: Return Stream Settings
    IngestService->>IngestService: pop_time_range(batch, min_field, max_field)
    IngestService-->>User: Acknowledge Data Ingestion

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai generate interesting stats about this repository and render them as a table.
- @coderabbitai show all the console.log statements in this repository.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

src/service/search/cluster/mod.rs

hengfeiyang added 3 commits June 30, 2024 21:04

feat: extend inverted index to support secondary index

42ea251

feat: table provider support segment ids

d9454c2

feat: add search with segments

e43cba0

github-actions bot added the ✏️ Feature label Jul 1, 2024

Merge branch 'main' into feat/secondary-index

d18514f

feat: add const variable for _all field

a4c3428

hengfeiyang requested review from ansrivas, haohuaijin and taimingl and removed request for haohuaijin July 1, 2024 15:01

hengfeiyang marked this pull request as draft July 1, 2024 15:01

Merge branch 'main' into feat/secondary-index

ee6cd95

taimingl reviewed Jul 2, 2024

View reviewed changes

src/service/search/cluster/mod.rs Outdated Show resolved Hide resolved

hengfeiyang added 10 commits July 2, 2024 10:52

fix: sort index file list by _timestamp

4d0b3ec

feat: optimize for complex query conditions

f83317a

chore: format code

0810839

feat: support other stream create index

ed21581

Merge branch 'main' into feat/secondary-index

e93a3f0

fix: check using inverted index

24bbe11

fix: index generate wrong

8b26ba8

Merge branch 'main' into feat/secondary-index

a40033c

fix: branch confilict

495a5d1

Merge branch 'main' into feat/secondary-index

e79fbd6

hengfeiyang mentioned this pull request Jul 2, 2024

Query performance improvements #3852

Closed

10 tasks

hengfeiyang added 4 commits July 2, 2024 22:36

Merge branch 'main' into feat/secondary-index

79f2328

fix: inverted index search

3fb1fc2

fix: search with inverted index

b90db87

fix: only log stream support user defined schema

b179e67

hengfeiyang merged commit 46a5af1 into main Jul 10, 2024

hengfeiyang deleted the feat/secondary-index branch July 10, 2024 07:08

taimingl pushed a commit that referenced this pull request Jul 12, 2024

feat: secondary index (#3870)

813ea68

This was referenced Oct 9, 2024

feat: Pipeline integration v2 #4732

Merged

perf: use private merge function for basic stream type #4783

Merged

This was referenced Oct 22, 2024

fix: can't query data in the case with multiple memtable buckets #4867

Merged

feat: SearchEventContext for search usage reports #4934

Merged

This was referenced Oct 30, 2024

feat: add jobs for compact old data #4952

Merged

feat: refactor merge file on compactor #4971

Merged

coderabbitai bot mentioned this pull request Nov 7, 2024

feat: Integrate tantivy as an inverted index substitute #4733

Merged

7 tasks

This was referenced Nov 25, 2024

fix: add index_condition when do not have index file #5137

Merged

feat: add custom tokenizer #5177

Merged

This was referenced Dec 2, 2024

fix: no need to create tantivy index if the data is empty #5202

Merged

feat: index for super cluster #5221

Merged

This was referenced Dec 19, 2024

feat: streaming agg query #5412

Merged

fix: add limit for full text search field must by text #5422

Merged

feat: support cross stream type join #5452

Merged

This was referenced Jan 9, 2025

feat: add option for extra default index fields #5631

Merged

feat: impl join right table only match once #5661

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: secondary index #3870

feat: secondary index #3870

Uh oh!

hengfeiyang commented Jul 1, 2024 •

edited by coderabbitai bot

Loading

Uh oh!

coderabbitai bot commented Jul 1, 2024 •

edited

Loading

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (`.coderabbit.yaml`)

Documentation and Community

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

feat: secondary index #3870

feat: secondary index #3870

Uh oh!

Conversation

hengfeiyang commented Jul 1, 2024 • edited by coderabbitai bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

New ENV

API changes:

TODO

How it works

Limitation

Summary by CodeRabbit

Uh oh!

coderabbitai bot commented Jul 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

New Flow with Segment IDs and Inverted Indexes

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (.coderabbit.yaml)

Documentation and Community

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

hengfeiyang commented Jul 1, 2024 •

edited by coderabbitai bot

Loading

coderabbitai bot commented Jul 1, 2024 •

edited

Loading

CodeRabbit Configration File (`.coderabbit.yaml`)