Skip to content

Iterators refactor - Replace Old Iterators implementation with the new ones - [MOD-8850]#6572

Merged
GuyAv46 merged 36 commits intomasterfrom
guyav-replace_old_iterators
Aug 5, 2025
Merged

Iterators refactor - Replace Old Iterators implementation with the new ones - [MOD-8850]#6572
GuyAv46 merged 36 commits intomasterfrom
guyav-replace_old_iterators

Conversation

@GuyAv46
Copy link
Collaborator

@GuyAv46 GuyAv46 commented Jul 30, 2025

Describe the changes in the pull request

Replacing old implementation usage with the new one, and deleting the old implementation + any now-dead code.

Additional changes

  1. Move Optimus and Hybrid iterators to the iterators directory, and align their flow to the new API (mainly changing the struct and adding the new Revalidate API)
  2. Remove all the old micro benchmarks and their mock helpers
  3. Removing the old "OnReopen" concurrent mechanism
  4. Delete cursed index.c file

Mark if applicable

  • This PR introduces API changes
  • This PR introduces serialization changes

@GuyAv46 GuyAv46 added enforce:coverage Run coverage flow even on draft pull request enforce:sanitize Run sanitizer flow even on draft pull request labels Jul 30, 2025
@GuyAv46 GuyAv46 force-pushed the guyav-replace_old_iterators branch from 32aee8a to b00ed89 Compare July 30, 2025 12:37
@GuyAv46 GuyAv46 requested a review from JoanFM August 5, 2025 12:07
@GuyAv46 GuyAv46 enabled auto-merge August 5, 2025 12:07
JoanFM
JoanFM previously approved these changes Aug 5, 2025
@GuyAv46 GuyAv46 added this pull request to the merge queue Aug 5, 2025
Copy link
Collaborator

@alonre24 alonre24 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

few more small questions

Comment on lines +687 to +691
if (term && sctx) {
// compute IDF based on num of docs in the header
term->idf = CalculateIDF(sctx->spec->docs.size, idx->numDocs); // FIXME: docs.size starts at 1???
term->bm25_idf = CalculateIDF_BM25(sctx->spec->stats.numDocuments, idx->numDocs);
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need that for tags? they are not supposed to be scorable

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But they are today. Maybe because there was no dedicated API for them or another reason, but I don't intend to change that on this PR.
We can remove this part in the future.

@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Aug 5, 2025
alonre24
alonre24 previously approved these changes Aug 5, 2025
@GuyAv46 GuyAv46 added this pull request to the merge queue Aug 5, 2025
@GuyAv46 GuyAv46 removed this pull request from the merge queue due to a manual request Aug 5, 2025
@GuyAv46 GuyAv46 added this pull request to the merge queue Aug 5, 2025
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Aug 5, 2025
@GuyAv46 GuyAv46 added this pull request to the merge queue Aug 5, 2025
Merged via the queue into master with commit 7615b92 Aug 5, 2025
16 of 17 checks passed
@GuyAv46 GuyAv46 deleted the guyav-replace_old_iterators branch August 5, 2025 19:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants