Skip to content

SPEC.md

Latest commit

 

History

History
99 lines (69 loc) · 2.47 KB

SPEC.md

File metadata and controls

99 lines (69 loc) · 2.47 KB
Raw

Open Fast Lookup Standard Specification

Version: 1.0.0 (Draft)

1. Terminology

  • Source: A searchable collection of data
  • Index: An acceleration structure built from source data
  • Query: A structured search request
  • SearchOutput: The canonical response object
  • Engine: The search implementation
  • Connector: A module that interfaces with a specific source type
  • Segment: An immutable unit of index storage
  • Generation: A version number incremented on each index update

2. Source Model

2.1 Source Types

  • filesystem-metadata - File and directory metadata
  • records-json - JSON record collections
  • records-ndjson - NDJSON record collections
  • records-csv - CSV record collections (optional)

2.2 Source Descriptor

Each source must have: unique ID, name, type, connector, location, schema, capabilities.

2.3 Capabilities

Sources declare supported operations: exact lookup, prefix, range, text search, streaming, etc.

3. Query Model

3.1 Query Object

  • version: Query schema version
  • text: Free text search
  • fields: Fields to search
  • filters: Array of filter conditions
  • sources: Source selection
  • sort: Sort specification
  • limit: Maximum results
  • cursor: Pagination cursor
  • options: Query options

3.2 Filter Operators

eq, neq, gt, gte, lt, lte, range, prefix, suffix, contains, starts_with_path, exists, missing, in, not_in, tag, not_tag

3.3 Boolean Composition

and, or, not

4. Output Model

See OUTPUT.md for SearchOutput specification.

5. Error Model

Structured errors with: code, message, retryable, source_id, details.

6. Index Model

6.1 Requirements

  • Deterministic
  • Rebuildable from source
  • Versioned
  • Checksummed (SHA-256)
  • Segment-based
  • Crash-safe
  • Atomic commits

6.2 Index Types

  • Exact index (hash map)
  • Prefix index (trie/radix tree)
  • Range index (sorted array)
  • Bitmap/tag index
  • Inverted text index
  • Trigram index
  • Path/hierarchy index

7. Performance Classes

  • Hot: Engine running, index loaded. Target: p50 <1ms, p99 <2ms
  • Warm: Engine running, some disk reads. Target: usually <10ms
  • Cold: Fresh process start. No specific targets.

8. Conformance Rules

Implementations must:

  • Return deterministic results for same query and index state
  • Use structured SearchOutput
  • Support error codes
  • Report timing
  • Support pagination cursors
  • Verify checksums on index open