Greptile Overview

Greptile Summary

Fixed a critical tokenizer inconsistency where o2_collect_tokens wasn't applying the same length filters (min 2 chars, max 64 chars) that are applied during indexing, causing search queries with single-character tokens to fail with "not found" results.

Key Changes:

• Replaced direct instantiation of SimpleTokenizer/O2Tokenizer in o2_collect_tokens with a call to o2_tokenizer_build()
• Now both indexing and search operations use identical tokenization logic with RemoveShortFilter and RemoveLongFilter
• Ensures queries like match_all('INFO actix_web...') work correctly by filtering single-char tokens during search, matching the index behavior

Impact:

• Resolves search failures when queries contain single-character tokens
• Maintains consistency between index creation and search query processing

Confidence Score: 5/5

• This PR is safe to merge with minimal risk
• The fix is a simple, well-targeted change that solves a clear tokenizer inconsistency bug. It replaces manual tokenizer instantiation with a call to the existing o2_tokenizer_build() function, ensuring search and indexing use identical token processing. The change is minimal (7 lines removed, 1 line added), has no side effects, and improves correctness without introducing new complexity or risks.
• No files require special attention

Important Files Changed

File Analysis

Sequence Diagram

sequenceDiagram
    participant User
    participant SearchAPI
    participant o2_collect_tokens
    participant o2_tokenizer_build
    participant RemoveShortFilter
    participant RemoveLongFilter
    participant Index

    User->>SearchAPI: match_all('INFO actix_web...')
    SearchAPI->>o2_collect_tokens: Tokenize search query
    
    Note over o2_collect_tokens: BEFORE: Used SimpleTokenizer/O2Tokenizer directly<br/>(no length filtering)
    Note over o2_collect_tokens: AFTER: Uses o2_tokenizer_build()<br/>(applies length filtering)
    
    o2_collect_tokens->>o2_tokenizer_build: Build tokenizer with filters
    o2_tokenizer_build->>RemoveShortFilter: Apply min_token_length >= 2
    RemoveShortFilter->>RemoveLongFilter: Apply max_token_length <= 64
    RemoveLongFilter-->>o2_collect_tokens: Configured tokenizer
    
    o2_collect_tokens->>o2_collect_tokens: Process tokens with filters
    Note over o2_collect_tokens: Single char tokens removed
    
    o2_collect_tokens-->>SearchAPI: Filtered tokens
    SearchAPI->>Index: Query with filtered tokens
    Index-->>User: Consistent results

_{1 file reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}