Skip to content

Add support for latest txtai indexing options #61

@davidmezzetti

Description

@davidmezzetti

Make the following improvements to better integrate with the latest txtai indexing options.

  • Only tokenize text if scoring is set. Transformers models don't need text tokenization and it can hurt performance.
  • Only filter sections when indexing if scoring is set. This isn't necessary for Transformers models.
  • Check the return type from embeddings.search and handle both dict content=true and tuple content=false return types.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions