Note

Ensure byte offsets are written for empty tokens to fix cross-field highlighting; update forward index token API and add highlight tests.

• Core indexing/highlighting:
  • Always write byte offsets during tokenization, including for empty tokens, via writeByteOffsets in src/document.c to preserve correct cross-field highlighting.
  • Move byte-offset writing responsibility out of forwardIndexTokenFunc into caller; compute indexesEmpty near token loop.
  • Change forwardIndexTokenFunc signature to int forwardIndexTokenFunc(ForwardIndexTokenizerCtx *tokCtx, const Token *tokInfo) and add detailed docs; remove internal allOffsets writes.
• Headers:
  • Update declaration in src/forward_index.h to match new signature.
• Tests:
  • Add tests/pytests/test_highlight.py covering highlighting across empty fields, arrays, field order, INDEXEMPTY, and empty-string queries.

^{Written by Cursor Bugbot for commit 975346a. This will update automatically on new commits. Configure here.}