Skip to content

[MOD-9372 , MOD-9733] Stop indexing OOM - Add wait before OOM#6114

Merged
lerman25 merged 47 commits intomasterfrom
omerL-Wait-After-OOM
May 20, 2025
Merged

[MOD-9372 , MOD-9733] Stop indexing OOM - Add wait before OOM#6114
lerman25 merged 47 commits intomasterfrom
omerL-Wait-After-OOM

Conversation

@lerman25
Copy link
Collaborator

@lerman25 lerman25 commented May 11, 2025

This PR adds a wait-before-OOM option, extending the work in #5778.
In Enterprise we want to give the Resource Manager time to raise memory limits when background indexing hits OOM.
If 2 consecutive OOM were reached - the scan stops.

We therefore introduce the BG_INDEX_OOM_PAUSE_TIME config, which sets how long the background-indexing thread sleeps before it restarts the scan.

  • Enterprise default: 5 s
  • OSS default: 0 s (no pause, no restart)

While the scan is paused, a Redis log entry is written for visibility.

For testing, 2 new command are introduced for the BG_SCAN_CONTROLLER (#5672 ) - SET_PAUSE_BEFORE_OOM_RESET and DEBUG_SCANNER_UPDATE_CONFIG.
The first - pauses before we wait and restart the scan
The latter - updates the debug indexes scanner if a config was changed after it's initialization.
Added scanner status PAUSED_BEFORE_OOM_RESET to support the above.

@github-actions github-actions bot added size:L and removed size:M labels May 12, 2025
@lerman25 lerman25 added enforce:coverage Run coverage flow even on draft pull request size:M and removed size:L labels May 12, 2025
@lerman25 lerman25 marked this pull request as ready for review May 12, 2025 15:23
@lerman25 lerman25 marked this pull request as draft May 12, 2025 15:23
Copy link
Collaborator

@alonre24 alonre24 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well done

@lerman25 lerman25 added this pull request to the merge queue May 19, 2025
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks May 19, 2025
@lerman25 lerman25 added this pull request to the merge queue May 19, 2025
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks May 19, 2025
@lerman25 lerman25 added this pull request to the merge queue May 19, 2025
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks May 19, 2025
@lerman25 lerman25 added this pull request to the merge queue May 20, 2025
Merged via the queue into master with commit cde5f79 May 20, 2025
14 checks passed
@lerman25 lerman25 deleted the omerL-Wait-After-OOM branch May 20, 2025 08:14
BenGoldberger pushed a commit that referenced this pull request May 21, 2025
* Add config

* insert oom_scan field to scanner, remove check for oom in debug pause on oom

* fix comment

* create basis for function

* handle last scanned key

* change sleep function name

* Add pause before and after reset

* Add pause before and after statuses code

* Add pause before and after reset to bg scan

* wait if config >0

* adjust default sleep time

* fix last scanned key

* temp tests

* bsaic test strcuture

* styling

* Add scanner status strings

* Add first test

* Add update option for dbug scanner and more tests

* remove scanner cancleation

* Alter and Drop tests

* spellcheck

* skip cluster in tests

* Add config test for new config

* shortern tests

* styling

* remove pause on OOM from drop test

* Add tests, style

* style

* style

* Remove unused pause after OOM reset

* debug commands tests and cluster tests

* Naming, styling, formatting

* rename, change structure for simplicity

* improve test robustness

* remove unused and move to better location

* Ben comments round1

* Add 0 thresh test

* comment

* ADd skip cluster

* change config and other Alon's comments

* small tess

* remove pause after
Remove duplicates in tests

* more test compression

* fix assert
BenGoldberger pushed a commit that referenced this pull request May 21, 2025
* Add config

* insert oom_scan field to scanner, remove check for oom in debug pause on oom

* fix comment

* create basis for function

* handle last scanned key

* change sleep function name

* Add pause before and after reset

* Add pause before and after statuses code

* Add pause before and after reset to bg scan

* wait if config >0

* adjust default sleep time

* fix last scanned key

* temp tests

* bsaic test strcuture

* styling

* Add scanner status strings

* Add first test

* Add update option for dbug scanner and more tests

* remove scanner cancleation

* Alter and Drop tests

* spellcheck

* skip cluster in tests

* Add config test for new config

* shortern tests

* styling

* remove pause on OOM from drop test

* Add tests, style

* style

* style

* Remove unused pause after OOM reset

* debug commands tests and cluster tests

* Naming, styling, formatting

* rename, change structure for simplicity

* improve test robustness

* remove unused and move to better location

* Ben comments round1

* Add 0 thresh test

* comment

* ADd skip cluster

* change config and other Alon's comments

* small tess

* remove pause after
Remove duplicates in tests

* more test compression

* fix assert
BenGoldberger pushed a commit that referenced this pull request May 21, 2025
* Add config

* insert oom_scan field to scanner, remove check for oom in debug pause on oom

* fix comment

* create basis for function

* handle last scanned key

* change sleep function name

* Add pause before and after reset

* Add pause before and after statuses code

* Add pause before and after reset to bg scan

* wait if config >0

* adjust default sleep time

* fix last scanned key

* temp tests

* bsaic test strcuture

* styling

* Add scanner status strings

* Add first test

* Add update option for dbug scanner and more tests

* remove scanner cancleation

* Alter and Drop tests

* spellcheck

* skip cluster in tests

* Add config test for new config

* shortern tests

* styling

* remove pause on OOM from drop test

* Add tests, style

* style

* style

* Remove unused pause after OOM reset

* debug commands tests and cluster tests

* Naming, styling, formatting

* rename, change structure for simplicity

* improve test robustness

* remove unused and move to better location

* Ben comments round1

* Add 0 thresh test

* comment

* ADd skip cluster

* change config and other Alon's comments

* small tess

* remove pause after
Remove duplicates in tests

* more test compression

* fix assert
JoanFM pushed a commit that referenced this pull request May 27, 2025
* Add config

* insert oom_scan field to scanner, remove check for oom in debug pause on oom

* fix comment

* create basis for function

* handle last scanned key

* change sleep function name

* Add pause before and after reset

* Add pause before and after statuses code

* Add pause before and after reset to bg scan

* wait if config >0

* adjust default sleep time

* fix last scanned key

* temp tests

* bsaic test strcuture

* styling

* Add scanner status strings

* Add first test

* Add update option for dbug scanner and more tests

* remove scanner cancleation

* Alter and Drop tests

* spellcheck

* skip cluster in tests

* Add config test for new config

* shortern tests

* styling

* remove pause on OOM from drop test

* Add tests, style

* style

* style

* Remove unused pause after OOM reset

* debug commands tests and cluster tests

* Naming, styling, formatting

* rename, change structure for simplicity

* improve test robustness

* remove unused and move to better location

* Ben comments round1

* Add 0 thresh test

* comment

* ADd skip cluster

* change config and other Alon's comments

* small tess

* remove pause after
Remove duplicates in tests

* more test compression

* fix assert
JoanFM pushed a commit that referenced this pull request May 27, 2025
* Add config

* insert oom_scan field to scanner, remove check for oom in debug pause on oom

* fix comment

* create basis for function

* handle last scanned key

* change sleep function name

* Add pause before and after reset

* Add pause before and after statuses code

* Add pause before and after reset to bg scan

* wait if config >0

* adjust default sleep time

* fix last scanned key

* temp tests

* bsaic test strcuture

* styling

* Add scanner status strings

* Add first test

* Add update option for dbug scanner and more tests

* remove scanner cancleation

* Alter and Drop tests

* spellcheck

* skip cluster in tests

* Add config test for new config

* shortern tests

* styling

* remove pause on OOM from drop test

* Add tests, style

* style

* style

* Remove unused pause after OOM reset

* debug commands tests and cluster tests

* Naming, styling, formatting

* rename, change structure for simplicity

* improve test robustness

* remove unused and move to better location

* Ben comments round1

* Add 0 thresh test

* comment

* ADd skip cluster

* change config and other Alon's comments

* small tess

* remove pause after
Remove duplicates in tests

* more test compression

* fix assert
lerman25 added a commit that referenced this pull request May 27, 2025
* Add config

* insert oom_scan field to scanner, remove check for oom in debug pause on oom

* fix comment

* create basis for function

* handle last scanned key

* change sleep function name

* Add pause before and after reset

* Add pause before and after statuses code

* Add pause before and after reset to bg scan

* wait if config >0

* adjust default sleep time

* fix last scanned key

* temp tests

* bsaic test strcuture

* styling

* Add scanner status strings

* Add first test

* Add update option for dbug scanner and more tests

* remove scanner cancleation

* Alter and Drop tests

* spellcheck

* skip cluster in tests

* Add config test for new config

* shortern tests

* styling

* remove pause on OOM from drop test

* Add tests, style

* style

* style

* Remove unused pause after OOM reset

* debug commands tests and cluster tests

* Naming, styling, formatting

* rename, change structure for simplicity

* improve test robustness

* remove unused and move to better location

* Ben comments round1

* Add 0 thresh test

* comment

* ADd skip cluster

* change config and other Alon's comments

* small tess

* remove pause after
Remove duplicates in tests

* more test compression

* fix assert
github-merge-queue bot pushed a commit that referenced this pull request May 28, 2025
* test config set doesnt effect other configs

* change config setters and getters

* handle isInverted

* fix partial indexes tests

* rename cluster_timeout

* fix py test

* refactor tests

* mll change to test

* Some cargo clippy fixes (#6122)

Clippy fixes

* Use the correct Rust profile for every task (#6128)

* Compute coverage for Rust tests in CI (#6127)

* restore macos-latest-xlarge (#6132)

* CI - update release flow - [MOD-9681] (#6131)

* change release flow to be triggered by a tag push

* TEMP call this branch 3.2.1

* dummy content

* fix branch validation

* improve error message

* revert temporary changes

* improve message

* [MOD-9694] Trie prefixes iterator (#6119)

* Prefixes iterator

* Align with C: do not track keys in the prefixes iterator

* Compile arr.h as a standalone static library

* Fix memory issue in bench

* Add license header

* Re-add tests for values iterator

* Add another prefix benchmark using the full Gutenberg dataset.

* [MOD-9693] IntoValues trie iterator. (#6129)

IntoValues iterator

* rockylinux:8 install python3.12 packages (#6147)

rockylinux:8 install python3.12-devel

* Fix linting failure on Rust LowMemoryThinVec drop implementation (#6146)

replace `mem::replace` with `mem::take`

* [master] [8.0] Remove -Werror linker flags [MOD-9624] (#6105)

[8.0] Remove -Werror linker flags [MOD-9624] (#6104)

* remove linker flag -werror

* remove linker flag -werror - one more

(cherry picked from commit 5688fcc)

Co-authored-by: alonre24 <[email protected]>

* Coverage Report Without `readies` - Phase 2 - [MOD-6711] (#5909)

* WIP new coverage flow

* move coverage logic to task-test.yml and build.sh

* type change and rust test fixes

* comment-out run_miri input

* minor fixes and improvements

* update file paths

* fix build for coverage

* remove redundancy

* fix redis build

* fix command

* another attempt

* fix

* run against latest redis

* fix coordinator env vars

* add missing rust tests

* don't run rust tests on sanitizer (for now)

* attempt to fix coverage capture

* attempt to fix rust sanitizer

* attempt to fix the rust test instead

* replace single quotes with double quotes

* improve prints

* remove quotes

* attempt to fix unit-test paths

* debug build on coverage

* cleanups

* Revert "MOD-8391: Report active threads-indexes upon crash (#5403)"

This reverts commit 384a2f6.

* Reapply "MOD-8391: Report active threads-indexes upon crash (#5403)"

This reverts commit ed38b62.

* better capture

* add a tests extraction step for a unified flow in older versions

* fix step to allow no-op

* CI Improvements (#6153)

* improvements

* use job-based concurrency

* notify benchmark failure only on push

* improve job name

* test notification

* skip actual benchamrks

* Revert "skip actual benchamrks"

This reverts commit 59e8fb2.

* Revert "test notification"

This reverts commit 1661e0a.

* re-implement notification condition with input

* test

* Revert "test"

This reverts commit 80206ae.

* cleanup

* test recent changes

* change and test

* fix

* cleanup

* fixes

* Add .cache to git ignore list (#6157)

Seems clang creates a cache under this directory, so let's put it to
git ignore list.

* Remove base64 code from codebase (#6161)

remove base64 code

* [MOD-9372 , MOD-9733] Stop indexing OOM - Add wait before OOM (#6114)

* Add config

* insert oom_scan field to scanner, remove check for oom in debug pause on oom

* fix comment

* create basis for function

* handle last scanned key

* change sleep function name

* Add pause before and after reset

* Add pause before and after statuses code

* Add pause before and after reset to bg scan

* wait if config >0

* adjust default sleep time

* fix last scanned key

* temp tests

* bsaic test strcuture

* styling

* Add scanner status strings

* Add first test

* Add update option for dbug scanner and more tests

* remove scanner cancleation

* Alter and Drop tests

* spellcheck

* skip cluster in tests

* Add config test for new config

* shortern tests

* styling

* remove pause on OOM from drop test

* Add tests, style

* style

* style

* Remove unused pause after OOM reset

* debug commands tests and cluster tests

* Naming, styling, formatting

* rename, change structure for simplicity

* improve test robustness

* remove unused and move to better location

* Ben comments round1

* Add 0 thresh test

* comment

* ADd skip cluster

* change config and other Alon's comments

* small tess

* remove pause after
Remove duplicates in tests

* more test compression

* fix assert

* Temp disable coverage as required for CI to pass in PRs (#6166)

temp to allow PRs

* [MOD-9735] Track the number of unique keys in the trie. (#6156)

Track the number of unique keys in the trie.

* [MOD-9692] Wildcard trie iterator. (#6130)

Wildcard trie iterator

* Fix micro-benchmarking job on master branch (#6162)

Fix workspace benching

* Use the correct profile name for Rust debug builds (#6170)

* Add More Diagnostics When Active Queries Are Not Empty (#6167)

* initial commit

* Apply suggestions from code review

Co-authored-by: GuyAv46 <[email protected]>

* code review comments

* output the explain string for queries if hide user data from logs is off when dumping queries

* fix typo

* free allocated query string

---------

Co-authored-by: GuyAv46 <[email protected]>

* [MOD-9372 , MOD-9733] Stop indexing OOM - Add wait before OOM (#6114)

* Add config

* insert oom_scan field to scanner, remove check for oom in debug pause on oom

* fix comment

* create basis for function

* handle last scanned key

* change sleep function name

* Add pause before and after reset

* Add pause before and after statuses code

* Add pause before and after reset to bg scan

* wait if config >0

* adjust default sleep time

* fix last scanned key

* temp tests

* bsaic test strcuture

* styling

* Add scanner status strings

* Add first test

* Add update option for dbug scanner and more tests

* remove scanner cancleation

* Alter and Drop tests

* spellcheck

* skip cluster in tests

* Add config test for new config

* shortern tests

* styling

* remove pause on OOM from drop test

* Add tests, style

* style

* style

* Remove unused pause after OOM reset

* debug commands tests and cluster tests

* Naming, styling, formatting

* rename, change structure for simplicity

* improve test robustness

* remove unused and move to better location

* Ben comments round1

* Add 0 thresh test

* comment

* ADd skip cluster

* change config and other Alon's comments

* small tess

* remove pause after
Remove duplicates in tests

* more test compression

* fix assert

* mention BM25 as default scorer in README (#6165)

* Yield to redis while indexing - [MOD-9220] (#6103)

* add op-counter + config

* Run all Rust benchmarks in the workspace (#6030)

* Run all Rust benchmarks in the workspace

* Allow forcing the run for micro-benchmarks

* Fix bench run

* Update .github/workflows/flow-micro-benchmarks.yml

Co-authored-by: Tim Janus <[email protected]>

---------

Co-authored-by: Tim Janus <[email protected]>

* Fix Rust build after switching off readies (#6086)

Fix Rust linking

* [MOD-9560] Change default config value for _BG_INDEX_MEM_PCT_THR (#6053)

* change default value

* change default value in config pytest

* change index oom tests to lower value then new default

* change debug commands tests to lower value then default value

* change default value of set tight memory functions

* 100testv1

* test default value

* Build unit tests without readies [MOD-9099] (#6082)

* purge readeis from cmake. Use new build script instead of makefile

* call script in CI

* libuv

* restore build folder

* libuv

* unit tests

* pretty build.sh

* pytest

* fixed extension build error

* Add profile flags + try pass linker flag properly (wip)

* fix build and linkage + pytest + adjust json

* small cleanup

* Add pytz dep

* bump googletest version, use same variant name as before

* remove ld libs

* Fix executable linker flags - build unit test properly.
run unit tests with old "make unit-tests" way (until this is fixed via ./build.sh)

* fix rust build via build.sh

* fix san build

* Set coverage flags and use the same bin dir for sanitizer (build it for debug)

* update benchmark image for regression test

* remove leftover

* Build hiredis static

* fix json env flag in CI

* CR changes

* Restore deleted files until unit tests are done as well, use boolean params

* restore vecsim version

* fix profile and fix build error

* fix for profile

* try fix mac build

* fix unit tests for arm

* Define boost dir

* remove the policies

* try fix the binroot for unit-tests

* change dir name for arm

* remove policy from hash as well

* use clang in mac

* try to set hardcoded clang

* try to mimic readies in macos includes

* try set CMAKE_OSX_DEPLOYMENT_TARGET

* whitespace formatting

* Set CC to clang in apple

* try to export llvm

* try set compiler in build script

* use bin in path

* try EXPORT properly with homebrew

* remove setting bad path to clang

* set clang path

* update to llvm@18

* update c++ path

* set compiler with LLVM env var

* fix?

* set proper link flags for mac

* remove bsymbolic from hiredis

* revert hiredis in non macos to how it was

* clean stuff

* cleanups

* Fix the extract debug symbols command so it will work as before (required for packing properly)

* update deps

* fix for alpine

* Refactor unit-tests script to not using readies WIP

* revert vecsim accidental change

* Fix script so it will work for sanitizer as well, address CR

---------

Co-authored-by: DvirDukhan <[email protected]>
Co-authored-by: GuyAv46 <[email protected]>

* Wrongly included the text of GPLv3 instead of AGPLv3 (#6089)

* README.md - added no standalone released note (#6074)

* Update README.md

* Update README.md

* Update README.md

* small change to opcounter

* debug command for yield counter + test

* add config test

* Update SECURITY.md (#6070)

* Update SECURITY.md

* Update SECURITY.md

* Enforce license headers for Rust files (#6090)

* A small binary to enforce license headers in all Rust files

* Enforce license headers in CI

* MOD-6151: Build without readies - simplify packing (#5908)

* fix

* change ramp version

* fix macos

* GHA3

* fix macos

* fix maxos

* fix

* fix

* fix

* new

* purge readies

* double

* typo

* remove debug

* shapshot

* gp

* go conflict

* darwin to macos

* Account for get-platform change, remove redundant mkdir, remove not used/used once variables

* remove unused DEP_NAMES

* remove unused function

* remove un condition

* improve pack_ramp

* remove unused DEPS

* macos

* remove xtx

* replace eprint

* remove runn

* replace realapth

* Revert "remove xtx"

This reverts commit 6efb895.

* try remove xtx

* remove NOP

* remove NUMVER

* without tmp

* delete

* remove eval

* remove release

* Assume SNAPSHOT=1

* remove SEMVER

* fin

* remove sbin/getver

* Don't add unnecessary \n after the license header (#6095)

Otherwise, `cargo fmt` fails on them.

* [MOD-9547] Core trie iterators (#6016)

* Basic iterators

* Typo

* Clean up API

* Add tests for iterators

* More comments

* Fix warning for miri

* Verify that all prefixed iterators agree with each other and match expectations

* Test both lending and non-lending traversals

* Clarify comment

* Test the empty key case

* Fix where bounds on traversal_filter

* Add missing license headers

* SSPLv1.txt - removed irrelevant text (#6097)

* MOD-9612: Fix flaky test and change early timeout error message (#6100)

* Fix flaky test AND change early timeout error message

* Fix TimeLimit initialization

* Add comment

* remove deps changes

* remove deps changes

* fix a possibly new flaky test

* temp fix for config

* changes to use the config_cmd func

* try to fix issue test

* use cluster conn in test

* handle num of yield with cluster

* dont test with cluster

* pr changes

* move whitespace

* add tag and geoshape idx

* pr changes

* changes to yield only while server is loading

* and check if module function exist

* remove white space

* add wait_for_index in test

* pr change rename config

* change help text

* add isLoading argument

* check globally if loading with g_isLoading

* pr change

---------

Co-authored-by: Luca Palmieri <[email protected]>
Co-authored-by: Tim Janus <[email protected]>
Co-authored-by: lerman25 <[email protected]>
Co-authored-by: alonre24 <[email protected]>
Co-authored-by: DvirDukhan <[email protected]>
Co-authored-by: GuyAv46 <[email protected]>
Co-authored-by: Lior Kogan <[email protected]>
Co-authored-by: Zeeshan Ali Khan <[email protected]>
Co-authored-by: Raz Monsonego <[email protected]>

* change config setters and getters

* rename cluster_timeout

* [MOD-9372 , MOD-9733] Stop indexing OOM - Add wait before OOM (#6114)

* Add config

* insert oom_scan field to scanner, remove check for oom in debug pause on oom

* fix comment

* create basis for function

* handle last scanned key

* change sleep function name

* Add pause before and after reset

* Add pause before and after statuses code

* Add pause before and after reset to bg scan

* wait if config >0

* adjust default sleep time

* fix last scanned key

* temp tests

* bsaic test strcuture

* styling

* Add scanner status strings

* Add first test

* Add update option for dbug scanner and more tests

* remove scanner cancleation

* Alter and Drop tests

* spellcheck

* skip cluster in tests

* Add config test for new config

* shortern tests

* styling

* remove pause on OOM from drop test

* Add tests, style

* style

* style

* Remove unused pause after OOM reset

* debug commands tests and cluster tests

* Naming, styling, formatting

* rename, change structure for simplicity

* improve test robustness

* remove unused and move to better location

* Ben comments round1

* Add 0 thresh test

* comment

* ADd skip cluster

* change config and other Alon's comments

* small tess

* remove pause after
Remove duplicates in tests

* more test compression

* fix assert

* pr changes

* remove unwanted changes

* dont change 'search-conn-per-shard' to max value, because it causes the test to be flaky

* change 'search-conn-per-shard' maxvalue in test

* change 'search-conn-per-shard' maxvalue in test

---------

Co-authored-by: Zeeshan Ali Khan <[email protected]>
Co-authored-by: Luca Palmieri <[email protected]>
Co-authored-by: meiravgri <[email protected]>
Co-authored-by: GuyAv46 <[email protected]>
Co-authored-by: nafraf <[email protected]>
Co-authored-by: redisearch-backport-pull-request[bot] <182669528+redisearch-backport-pull-request[bot]@users.noreply.github.com>
Co-authored-by: alonre24 <[email protected]>
Co-authored-by: lerman25 <[email protected]>
Co-authored-by: kei-nan <[email protected]>
Co-authored-by: Joan Fontanals <[email protected]>
Co-authored-by: Tim Janus <[email protected]>
Co-authored-by: DvirDukhan <[email protected]>
Co-authored-by: GuyAv46 <[email protected]>
Co-authored-by: Lior Kogan <[email protected]>
Co-authored-by: Raz Monsonego <[email protected]>
BenGoldberger added a commit that referenced this pull request May 28, 2025
* test config set doesnt effect other configs

* change config setters and getters

* handle isInverted

* fix partial indexes tests

* rename cluster_timeout

* fix py test

* refactor tests

* mll change to test

* Some cargo clippy fixes (#6122)

Clippy fixes

* Use the correct Rust profile for every task (#6128)

* Compute coverage for Rust tests in CI (#6127)

* restore macos-latest-xlarge (#6132)

* CI - update release flow - [MOD-9681] (#6131)

* change release flow to be triggered by a tag push

* TEMP call this branch 3.2.1

* dummy content

* fix branch validation

* improve error message

* revert temporary changes

* improve message

* [MOD-9694] Trie prefixes iterator (#6119)

* Prefixes iterator

* Align with C: do not track keys in the prefixes iterator

* Compile arr.h as a standalone static library

* Fix memory issue in bench

* Add license header

* Re-add tests for values iterator

* Add another prefix benchmark using the full Gutenberg dataset.

* [MOD-9693] IntoValues trie iterator. (#6129)

IntoValues iterator

* rockylinux:8 install python3.12 packages (#6147)

rockylinux:8 install python3.12-devel

* Fix linting failure on Rust LowMemoryThinVec drop implementation (#6146)

replace `mem::replace` with `mem::take`

* [master] [8.0] Remove -Werror linker flags [MOD-9624] (#6105)

[8.0] Remove -Werror linker flags [MOD-9624] (#6104)

* remove linker flag -werror

* remove linker flag -werror - one more

(cherry picked from commit 5688fcc)

Co-authored-by: alonre24 <[email protected]>

* Coverage Report Without `readies` - Phase 2 - [MOD-6711] (#5909)

* WIP new coverage flow

* move coverage logic to task-test.yml and build.sh

* type change and rust test fixes

* comment-out run_miri input

* minor fixes and improvements

* update file paths

* fix build for coverage

* remove redundancy

* fix redis build

* fix command

* another attempt

* fix

* run against latest redis

* fix coordinator env vars

* add missing rust tests

* don't run rust tests on sanitizer (for now)

* attempt to fix coverage capture

* attempt to fix rust sanitizer

* attempt to fix the rust test instead

* replace single quotes with double quotes

* improve prints

* remove quotes

* attempt to fix unit-test paths

* debug build on coverage

* cleanups

* Revert "MOD-8391: Report active threads-indexes upon crash (#5403)"

This reverts commit 384a2f6.

* Reapply "MOD-8391: Report active threads-indexes upon crash (#5403)"

This reverts commit ed38b62.

* better capture

* add a tests extraction step for a unified flow in older versions

* fix step to allow no-op

* CI Improvements (#6153)

* improvements

* use job-based concurrency

* notify benchmark failure only on push

* improve job name

* test notification

* skip actual benchamrks

* Revert "skip actual benchamrks"

This reverts commit 59e8fb2.

* Revert "test notification"

This reverts commit 1661e0a.

* re-implement notification condition with input

* test

* Revert "test"

This reverts commit 80206ae.

* cleanup

* test recent changes

* change and test

* fix

* cleanup

* fixes

* Add .cache to git ignore list (#6157)

Seems clang creates a cache under this directory, so let's put it to
git ignore list.

* Remove base64 code from codebase (#6161)

remove base64 code

* [MOD-9372 , MOD-9733] Stop indexing OOM - Add wait before OOM (#6114)

* Add config

* insert oom_scan field to scanner, remove check for oom in debug pause on oom

* fix comment

* create basis for function

* handle last scanned key

* change sleep function name

* Add pause before and after reset

* Add pause before and after statuses code

* Add pause before and after reset to bg scan

* wait if config >0

* adjust default sleep time

* fix last scanned key

* temp tests

* bsaic test strcuture

* styling

* Add scanner status strings

* Add first test

* Add update option for dbug scanner and more tests

* remove scanner cancleation

* Alter and Drop tests

* spellcheck

* skip cluster in tests

* Add config test for new config

* shortern tests

* styling

* remove pause on OOM from drop test

* Add tests, style

* style

* style

* Remove unused pause after OOM reset

* debug commands tests and cluster tests

* Naming, styling, formatting

* rename, change structure for simplicity

* improve test robustness

* remove unused and move to better location

* Ben comments round1

* Add 0 thresh test

* comment

* ADd skip cluster

* change config and other Alon's comments

* small tess

* remove pause after
Remove duplicates in tests

* more test compression

* fix assert

* Temp disable coverage as required for CI to pass in PRs (#6166)

temp to allow PRs

* [MOD-9735] Track the number of unique keys in the trie. (#6156)

Track the number of unique keys in the trie.

* [MOD-9692] Wildcard trie iterator. (#6130)

Wildcard trie iterator

* Fix micro-benchmarking job on master branch (#6162)

Fix workspace benching

* Use the correct profile name for Rust debug builds (#6170)

* Add More Diagnostics When Active Queries Are Not Empty (#6167)

* initial commit

* Apply suggestions from code review

Co-authored-by: GuyAv46 <[email protected]>

* code review comments

* output the explain string for queries if hide user data from logs is off when dumping queries

* fix typo

* free allocated query string

---------

Co-authored-by: GuyAv46 <[email protected]>

* [MOD-9372 , MOD-9733] Stop indexing OOM - Add wait before OOM (#6114)

* Add config

* insert oom_scan field to scanner, remove check for oom in debug pause on oom

* fix comment

* create basis for function

* handle last scanned key

* change sleep function name

* Add pause before and after reset

* Add pause before and after statuses code

* Add pause before and after reset to bg scan

* wait if config >0

* adjust default sleep time

* fix last scanned key

* temp tests

* bsaic test strcuture

* styling

* Add scanner status strings

* Add first test

* Add update option for dbug scanner and more tests

* remove scanner cancleation

* Alter and Drop tests

* spellcheck

* skip cluster in tests

* Add config test for new config

* shortern tests

* styling

* remove pause on OOM from drop test

* Add tests, style

* style

* style

* Remove unused pause after OOM reset

* debug commands tests and cluster tests

* Naming, styling, formatting

* rename, change structure for simplicity

* improve test robustness

* remove unused and move to better location

* Ben comments round1

* Add 0 thresh test

* comment

* ADd skip cluster

* change config and other Alon's comments

* small tess

* remove pause after
Remove duplicates in tests

* more test compression

* fix assert

* mention BM25 as default scorer in README (#6165)

* Yield to redis while indexing - [MOD-9220] (#6103)

* add op-counter + config

* Run all Rust benchmarks in the workspace (#6030)

* Run all Rust benchmarks in the workspace

* Allow forcing the run for micro-benchmarks

* Fix bench run

* Update .github/workflows/flow-micro-benchmarks.yml

Co-authored-by: Tim Janus <[email protected]>

---------

Co-authored-by: Tim Janus <[email protected]>

* Fix Rust build after switching off readies (#6086)

Fix Rust linking

* [MOD-9560] Change default config value for _BG_INDEX_MEM_PCT_THR (#6053)

* change default value

* change default value in config pytest

* change index oom tests to lower value then new default

* change debug commands tests to lower value then default value

* change default value of set tight memory functions

* 100testv1

* test default value

* Build unit tests without readies [MOD-9099] (#6082)

* purge readeis from cmake. Use new build script instead of makefile

* call script in CI

* libuv

* restore build folder

* libuv

* unit tests

* pretty build.sh

* pytest

* fixed extension build error

* Add profile flags + try pass linker flag properly (wip)

* fix build and linkage + pytest + adjust json

* small cleanup

* Add pytz dep

* bump googletest version, use same variant name as before

* remove ld libs

* Fix executable linker flags - build unit test properly.
run unit tests with old "make unit-tests" way (until this is fixed via ./build.sh)

* fix rust build via build.sh

* fix san build

* Set coverage flags and use the same bin dir for sanitizer (build it for debug)

* update benchmark image for regression test

* remove leftover

* Build hiredis static

* fix json env flag in CI

* CR changes

* Restore deleted files until unit tests are done as well, use boolean params

* restore vecsim version

* fix profile and fix build error

* fix for profile

* try fix mac build

* fix unit tests for arm

* Define boost dir

* remove the policies

* try fix the binroot for unit-tests

* change dir name for arm

* remove policy from hash as well

* use clang in mac

* try to set hardcoded clang

* try to mimic readies in macos includes

* try set CMAKE_OSX_DEPLOYMENT_TARGET

* whitespace formatting

* Set CC to clang in apple

* try to export llvm

* try set compiler in build script

* use bin in path

* try EXPORT properly with homebrew

* remove setting bad path to clang

* set clang path

* update to llvm@18

* update c++ path

* set compiler with LLVM env var

* fix?

* set proper link flags for mac

* remove bsymbolic from hiredis

* revert hiredis in non macos to how it was

* clean stuff

* cleanups

* Fix the extract debug symbols command so it will work as before (required for packing properly)

* update deps

* fix for alpine

* Refactor unit-tests script to not using readies WIP

* revert vecsim accidental change

* Fix script so it will work for sanitizer as well, address CR

---------

Co-authored-by: DvirDukhan <[email protected]>
Co-authored-by: GuyAv46 <[email protected]>

* Wrongly included the text of GPLv3 instead of AGPLv3 (#6089)

* README.md - added no standalone released note (#6074)

* Update README.md

* Update README.md

* Update README.md

* small change to opcounter

* debug command for yield counter + test

* add config test

* Update SECURITY.md (#6070)

* Update SECURITY.md

* Update SECURITY.md

* Enforce license headers for Rust files (#6090)

* A small binary to enforce license headers in all Rust files

* Enforce license headers in CI

* MOD-6151: Build without readies - simplify packing (#5908)

* fix

* change ramp version

* fix macos

* GHA3

* fix macos

* fix maxos

* fix

* fix

* fix

* new

* purge readies

* double

* typo

* remove debug

* shapshot

* gp

* go conflict

* darwin to macos

* Account for get-platform change, remove redundant mkdir, remove not used/used once variables

* remove unused DEP_NAMES

* remove unused function

* remove un condition

* improve pack_ramp

* remove unused DEPS

* macos

* remove xtx

* replace eprint

* remove runn

* replace realapth

* Revert "remove xtx"

This reverts commit 6efb895.

* try remove xtx

* remove NOP

* remove NUMVER

* without tmp

* delete

* remove eval

* remove release

* Assume SNAPSHOT=1

* remove SEMVER

* fin

* remove sbin/getver

* Don't add unnecessary \n after the license header (#6095)

Otherwise, `cargo fmt` fails on them.

* [MOD-9547] Core trie iterators (#6016)

* Basic iterators

* Typo

* Clean up API

* Add tests for iterators

* More comments

* Fix warning for miri

* Verify that all prefixed iterators agree with each other and match expectations

* Test both lending and non-lending traversals

* Clarify comment

* Test the empty key case

* Fix where bounds on traversal_filter

* Add missing license headers

* SSPLv1.txt - removed irrelevant text (#6097)

* MOD-9612: Fix flaky test and change early timeout error message (#6100)

* Fix flaky test AND change early timeout error message

* Fix TimeLimit initialization

* Add comment

* remove deps changes

* remove deps changes

* fix a possibly new flaky test

* temp fix for config

* changes to use the config_cmd func

* try to fix issue test

* use cluster conn in test

* handle num of yield with cluster

* dont test with cluster

* pr changes

* move whitespace

* add tag and geoshape idx

* pr changes

* changes to yield only while server is loading

* and check if module function exist

* remove white space

* add wait_for_index in test

* pr change rename config

* change help text

* add isLoading argument

* check globally if loading with g_isLoading

* pr change

---------

Co-authored-by: Luca Palmieri <[email protected]>
Co-authored-by: Tim Janus <[email protected]>
Co-authored-by: lerman25 <[email protected]>
Co-authored-by: alonre24 <[email protected]>
Co-authored-by: DvirDukhan <[email protected]>
Co-authored-by: GuyAv46 <[email protected]>
Co-authored-by: Lior Kogan <[email protected]>
Co-authored-by: Zeeshan Ali Khan <[email protected]>
Co-authored-by: Raz Monsonego <[email protected]>

* change config setters and getters

* rename cluster_timeout

* [MOD-9372 , MOD-9733] Stop indexing OOM - Add wait before OOM (#6114)

* Add config

* insert oom_scan field to scanner, remove check for oom in debug pause on oom

* fix comment

* create basis for function

* handle last scanned key

* change sleep function name

* Add pause before and after reset

* Add pause before and after statuses code

* Add pause before and after reset to bg scan

* wait if config >0

* adjust default sleep time

* fix last scanned key

* temp tests

* bsaic test strcuture

* styling

* Add scanner status strings

* Add first test

* Add update option for dbug scanner and more tests

* remove scanner cancleation

* Alter and Drop tests

* spellcheck

* skip cluster in tests

* Add config test for new config

* shortern tests

* styling

* remove pause on OOM from drop test

* Add tests, style

* style

* style

* Remove unused pause after OOM reset

* debug commands tests and cluster tests

* Naming, styling, formatting

* rename, change structure for simplicity

* improve test robustness

* remove unused and move to better location

* Ben comments round1

* Add 0 thresh test

* comment

* ADd skip cluster

* change config and other Alon's comments

* small tess

* remove pause after
Remove duplicates in tests

* more test compression

* fix assert

* pr changes

* remove unwanted changes

* dont change 'search-conn-per-shard' to max value, because it causes the test to be flaky

* change 'search-conn-per-shard' maxvalue in test

* change 'search-conn-per-shard' maxvalue in test

---------

Co-authored-by: Zeeshan Ali Khan <[email protected]>
Co-authored-by: Luca Palmieri <[email protected]>
Co-authored-by: meiravgri <[email protected]>
Co-authored-by: GuyAv46 <[email protected]>
Co-authored-by: nafraf <[email protected]>
Co-authored-by: redisearch-backport-pull-request[bot] <182669528+redisearch-backport-pull-request[bot]@users.noreply.github.com>
Co-authored-by: alonre24 <[email protected]>
Co-authored-by: lerman25 <[email protected]>
Co-authored-by: kei-nan <[email protected]>
Co-authored-by: Joan Fontanals <[email protected]>
Co-authored-by: Tim Janus <[email protected]>
Co-authored-by: DvirDukhan <[email protected]>
Co-authored-by: GuyAv46 <[email protected]>
Co-authored-by: Lior Kogan <[email protected]>
Co-authored-by: Raz Monsonego <[email protected]>
(cherry picked from commit fc03b3e)
github-merge-queue bot pushed a commit that referenced this pull request May 29, 2025
change config setters and getters - [MOD-9673] (#6151)

* test config set doesnt effect other configs

* change config setters and getters

* handle isInverted

* fix partial indexes tests

* rename cluster_timeout

* fix py test

* refactor tests

* mll change to test

* Some cargo clippy fixes (#6122)

Clippy fixes

* Use the correct Rust profile for every task (#6128)

* Compute coverage for Rust tests in CI (#6127)

* restore macos-latest-xlarge (#6132)

* CI - update release flow - [MOD-9681] (#6131)

* change release flow to be triggered by a tag push

* TEMP call this branch 3.2.1

* dummy content

* fix branch validation

* improve error message

* revert temporary changes

* improve message

* [MOD-9694] Trie prefixes iterator (#6119)

* Prefixes iterator

* Align with C: do not track keys in the prefixes iterator

* Compile arr.h as a standalone static library

* Fix memory issue in bench

* Add license header

* Re-add tests for values iterator

* Add another prefix benchmark using the full Gutenberg dataset.

* [MOD-9693] IntoValues trie iterator. (#6129)

IntoValues iterator

* rockylinux:8 install python3.12 packages (#6147)

rockylinux:8 install python3.12-devel

* Fix linting failure on Rust LowMemoryThinVec drop implementation (#6146)

replace `mem::replace` with `mem::take`

* [master] [8.0] Remove -Werror linker flags [MOD-9624] (#6105)

[8.0] Remove -Werror linker flags [MOD-9624] (#6104)

* remove linker flag -werror

* remove linker flag -werror - one more

(cherry picked from commit 5688fcc)



* Coverage Report Without `readies` - Phase 2 - [MOD-6711] (#5909)

* WIP new coverage flow

* move coverage logic to task-test.yml and build.sh

* type change and rust test fixes

* comment-out run_miri input

* minor fixes and improvements

* update file paths

* fix build for coverage

* remove redundancy

* fix redis build

* fix command

* another attempt

* fix

* run against latest redis

* fix coordinator env vars

* add missing rust tests

* don't run rust tests on sanitizer (for now)

* attempt to fix coverage capture

* attempt to fix rust sanitizer

* attempt to fix the rust test instead

* replace single quotes with double quotes

* improve prints

* remove quotes

* attempt to fix unit-test paths

* debug build on coverage

* cleanups

* Revert "MOD-8391: Report active threads-indexes upon crash (#5403)"

This reverts commit 384a2f6.

* Reapply "MOD-8391: Report active threads-indexes upon crash (#5403)"

This reverts commit ed38b62.

* better capture

* add a tests extraction step for a unified flow in older versions

* fix step to allow no-op

* CI Improvements (#6153)

* improvements

* use job-based concurrency

* notify benchmark failure only on push

* improve job name

* test notification

* skip actual benchamrks

* Revert "skip actual benchamrks"

This reverts commit 59e8fb2.

* Revert "test notification"

This reverts commit 1661e0a.

* re-implement notification condition with input

* test

* Revert "test"

This reverts commit 80206ae.

* cleanup

* test recent changes

* change and test

* fix

* cleanup

* fixes

* Add .cache to git ignore list (#6157)

Seems clang creates a cache under this directory, so let's put it to
git ignore list.

* Remove base64 code from codebase (#6161)

remove base64 code

* [MOD-9372 , MOD-9733] Stop indexing OOM - Add wait before OOM (#6114)

* Add config

* insert oom_scan field to scanner, remove check for oom in debug pause on oom

* fix comment

* create basis for function

* handle last scanned key

* change sleep function name

* Add pause before and after reset

* Add pause before and after statuses code

* Add pause before and after reset to bg scan

* wait if config >0

* adjust default sleep time

* fix last scanned key

* temp tests

* bsaic test strcuture

* styling

* Add scanner status strings

* Add first test

* Add update option for dbug scanner and more tests

* remove scanner cancleation

* Alter and Drop tests

* spellcheck

* skip cluster in tests

* Add config test for new config

* shortern tests

* styling

* remove pause on OOM from drop test

* Add tests, style

* style

* style

* Remove unused pause after OOM reset

* debug commands tests and cluster tests

* Naming, styling, formatting

* rename, change structure for simplicity

* improve test robustness

* remove unused and move to better location

* Ben comments round1

* Add 0 thresh test

* comment

* ADd skip cluster

* change config and other Alon's comments

* small tess

* remove pause after
Remove duplicates in tests

* more test compression

* fix assert

* Temp disable coverage as required for CI to pass in PRs (#6166)

temp to allow PRs

* [MOD-9735] Track the number of unique keys in the trie. (#6156)

Track the number of unique keys in the trie.

* [MOD-9692] Wildcard trie iterator. (#6130)

Wildcard trie iterator

* Fix micro-benchmarking job on master branch (#6162)

Fix workspace benching

* Use the correct profile name for Rust debug builds (#6170)

* Add More Diagnostics When Active Queries Are Not Empty (#6167)

* initial commit

* Apply suggestions from code review



* code review comments

* output the explain string for queries if hide user data from logs is off when dumping queries

* fix typo

* free allocated query string

---------



* [MOD-9372 , MOD-9733] Stop indexing OOM - Add wait before OOM (#6114)

* Add config

* insert oom_scan field to scanner, remove check for oom in debug pause on oom

* fix comment

* create basis for function

* handle last scanned key

* change sleep function name

* Add pause before and after reset

* Add pause before and after statuses code

* Add pause before and after reset to bg scan

* wait if config >0

* adjust default sleep time

* fix last scanned key

* temp tests

* bsaic test strcuture

* styling

* Add scanner status strings

* Add first test

* Add update option for dbug scanner and more tests

* remove scanner cancleation

* Alter and Drop tests

* spellcheck

* skip cluster in tests

* Add config test for new config

* shortern tests

* styling

* remove pause on OOM from drop test

* Add tests, style

* style

* style

* Remove unused pause after OOM reset

* debug commands tests and cluster tests

* Naming, styling, formatting

* rename, change structure for simplicity

* improve test robustness

* remove unused and move to better location

* Ben comments round1

* Add 0 thresh test

* comment

* ADd skip cluster

* change config and other Alon's comments

* small tess

* remove pause after
Remove duplicates in tests

* more test compression

* fix assert

* mention BM25 as default scorer in README (#6165)

* Yield to redis while indexing - [MOD-9220] (#6103)

* add op-counter + config

* Run all Rust benchmarks in the workspace (#6030)

* Run all Rust benchmarks in the workspace

* Allow forcing the run for micro-benchmarks

* Fix bench run

* Update .github/workflows/flow-micro-benchmarks.yml



---------



* Fix Rust build after switching off readies (#6086)

Fix Rust linking

* [MOD-9560] Change default config value for _BG_INDEX_MEM_PCT_THR (#6053)

* change default value

* change default value in config pytest

* change index oom tests to lower value then new default

* change debug commands tests to lower value then default value

* change default value of set tight memory functions

* 100testv1

* test default value

* Build unit tests without readies [MOD-9099] (#6082)

* purge readeis from cmake. Use new build script instead of makefile

* call script in CI

* libuv

* restore build folder

* libuv

* unit tests

* pretty build.sh

* pytest

* fixed extension build error

* Add profile flags + try pass linker flag properly (wip)

* fix build and linkage + pytest + adjust json

* small cleanup

* Add pytz dep

* bump googletest version, use same variant name as before

* remove ld libs

* Fix executable linker flags - build unit test properly.
run unit tests with old "make unit-tests" way (until this is fixed via ./build.sh)

* fix rust build via build.sh

* fix san build

* Set coverage flags and use the same bin dir for sanitizer (build it for debug)

* update benchmark image for regression test

* remove leftover

* Build hiredis static

* fix json env flag in CI

* CR changes

* Restore deleted files until unit tests are done as well, use boolean params

* restore vecsim version

* fix profile and fix build error

* fix for profile

* try fix mac build

* fix unit tests for arm

* Define boost dir

* remove the policies

* try fix the binroot for unit-tests

* change dir name for arm

* remove policy from hash as well

* use clang in mac

* try to set hardcoded clang

* try to mimic readies in macos includes

* try set CMAKE_OSX_DEPLOYMENT_TARGET

* whitespace formatting

* Set CC to clang in apple

* try to export llvm

* try set compiler in build script

* use bin in path

* try EXPORT properly with homebrew

* remove setting bad path to clang

* set clang path

* update to llvm@18

* update c++ path

* set compiler with LLVM env var

* fix?

* set proper link flags for mac

* remove bsymbolic from hiredis

* revert hiredis in non macos to how it was

* clean stuff

* cleanups

* Fix the extract debug symbols command so it will work as before (required for packing properly)

* update deps

* fix for alpine

* Refactor unit-tests script to not using readies WIP

* revert vecsim accidental change

* Fix script so it will work for sanitizer as well, address CR

---------




* Wrongly included the text of GPLv3 instead of AGPLv3 (#6089)

* README.md - added no standalone released note (#6074)

* Update README.md

* Update README.md

* Update README.md

* small change to opcounter

* debug command for yield counter + test

* add config test

* Update SECURITY.md (#6070)

* Update SECURITY.md

* Update SECURITY.md

* Enforce license headers for Rust files (#6090)

* A small binary to enforce license headers in all Rust files

* Enforce license headers in CI

* MOD-6151: Build without readies - simplify packing (#5908)

* fix

* change ramp version

* fix macos

* GHA3

* fix macos

* fix maxos

* fix

* fix

* fix

* new

* purge readies

* double

* typo

* remove debug

* shapshot

* gp

* go conflict

* darwin to macos

* Account for get-platform change, remove redundant mkdir, remove not used/used once variables

* remove unused DEP_NAMES

* remove unused function

* remove un condition

* improve pack_ramp

* remove unused DEPS

* macos

* remove xtx

* replace eprint

* remove runn

* replace realapth

* Revert "remove xtx"

This reverts commit 6efb895.

* try remove xtx

* remove NOP

* remove NUMVER

* without tmp

* delete

* remove eval

* remove release

* Assume SNAPSHOT=1

* remove SEMVER

* fin

* remove sbin/getver

* Don't add unnecessary \n after the license header (#6095)

Otherwise, `cargo fmt` fails on them.

* [MOD-9547] Core trie iterators (#6016)

* Basic iterators

* Typo

* Clean up API

* Add tests for iterators

* More comments

* Fix warning for miri

* Verify that all prefixed iterators agree with each other and match expectations

* Test both lending and non-lending traversals

* Clarify comment

* Test the empty key case

* Fix where bounds on traversal_filter

* Add missing license headers

* SSPLv1.txt - removed irrelevant text (#6097)

* MOD-9612: Fix flaky test and change early timeout error message (#6100)

* Fix flaky test AND change early timeout error message

* Fix TimeLimit initialization

* Add comment

* remove deps changes

* remove deps changes

* fix a possibly new flaky test

* temp fix for config

* changes to use the config_cmd func

* try to fix issue test

* use cluster conn in test

* handle num of yield with cluster

* dont test with cluster

* pr changes

* move whitespace

* add tag and geoshape idx

* pr changes

* changes to yield only while server is loading

* and check if module function exist

* remove white space

* add wait_for_index in test

* pr change rename config

* change help text

* add isLoading argument

* check globally if loading with g_isLoading

* pr change

---------











* change config setters and getters

* rename cluster_timeout

* [MOD-9372 , MOD-9733] Stop indexing OOM - Add wait before OOM (#6114)

* Add config

* insert oom_scan field to scanner, remove check for oom in debug pause on oom

* fix comment

* create basis for function

* handle last scanned key

* change sleep function name

* Add pause before and after reset

* Add pause before and after statuses code

* Add pause before and after reset to bg scan

* wait if config >0

* adjust default sleep time

* fix last scanned key

* temp tests

* bsaic test strcuture

* styling

* Add scanner status strings

* Add first test

* Add update option for dbug scanner and more tests

* remove scanner cancleation

* Alter and Drop tests

* spellcheck

* skip cluster in tests

* Add config test for new config

* shortern tests

* styling

* remove pause on OOM from drop test

* Add tests, style

* style

* style

* Remove unused pause after OOM reset

* debug commands tests and cluster tests

* Naming, styling, formatting

* rename, change structure for simplicity

* improve test robustness

* remove unused and move to better location

* Ben comments round1

* Add 0 thresh test

* comment

* ADd skip cluster

* change config and other Alon's comments

* small tess

* remove pause after
Remove duplicates in tests

* more test compression

* fix assert

* pr changes

* remove unwanted changes

* dont change 'search-conn-per-shard' to max value, because it causes the test to be flaky

* change 'search-conn-per-shard' maxvalue in test

* change 'search-conn-per-shard' maxvalue in test

---------
















(cherry picked from commit fc03b3e)

Co-authored-by: Zeeshan Ali Khan <[email protected]>
Co-authored-by: Luca Palmieri <[email protected]>
Co-authored-by: meiravgri <[email protected]>
Co-authored-by: GuyAv46 <[email protected]>
Co-authored-by: nafraf <[email protected]>
Co-authored-by: redisearch-backport-pull-request[bot] <182669528+redisearch-backport-pull-request[bot]@users.noreply.github.com>
Co-authored-by: alonre24 <[email protected]>
Co-authored-by: lerman25 <[email protected]>
Co-authored-by: kei-nan <[email protected]>
Co-authored-by: Joan Fontanals <[email protected]>
Co-authored-by: Tim Janus <[email protected]>
Co-authored-by: DvirDukhan <[email protected]>
Co-authored-by: GuyAv46 <[email protected]>
Co-authored-by: Lior Kogan <[email protected]>
Co-authored-by: Raz Monsonego <[email protected]>
github-merge-queue bot pushed a commit that referenced this pull request Jun 8, 2025
* [MOD-9372 , MOD-9733] Stop indexing OOM - Add wait before OOM (#6114)

* Add config

* insert oom_scan field to scanner, remove check for oom in debug pause on oom

* fix comment

* create basis for function

* handle last scanned key

* change sleep function name

* Add pause before and after reset

* Add pause before and after statuses code

* Add pause before and after reset to bg scan

* wait if config >0

* adjust default sleep time

* fix last scanned key

* temp tests

* bsaic test strcuture

* styling

* Add scanner status strings

* Add first test

* Add update option for dbug scanner and more tests

* remove scanner cancleation

* Alter and Drop tests

* spellcheck

* skip cluster in tests

* Add config test for new config

* shortern tests

* styling

* remove pause on OOM from drop test

* Add tests, style

* style

* style

* Remove unused pause after OOM reset

* debug commands tests and cluster tests

* Naming, styling, formatting

* rename, change structure for simplicity

* improve test robustness

* remove unused and move to better location

* Ben comments round1

* Add 0 thresh test

* comment

* ADd skip cluster

* change config and other Alon's comments

* small tess

* remove pause after
Remove duplicates in tests

* more test compression

* fix assert

* [MOD-9560]  Revert marking index as faulty after reaching OOM (#6140)

* Add config

* insert oom_scan field to scanner, remove check for oom in debug pause on oom

* fix comment

* create basis for function

* handle last scanned key

* change sleep function name

* Add pause before and after reset

* Add pause before and after statuses code

* Add pause before and after reset to bg scan

* wait if config >0

* adjust default sleep time

* fix last scanned key

* temp tests

* bsaic test strcuture

* styling

* Add scanner status strings

* Add first test

* Add update option for dbug scanner and more tests

* remove scanner cancleation

* Alter and Drop tests

* spellcheck

* skip cluster in tests

* Add config test for new config

* shortern tests

* styling

* remove pause on OOM from drop test

* Add tests, style

* style

* style

* Remove unused pause after OOM reset

* debug commands tests and cluster tests

* Naming, styling, formatting

* rename, change structure for simplicity

* improve test robustness

* remove unused and move to better location

* remove OOM from ACL

* remove oom from aggregate_exec

* Change config help message

* remove from ft.search

* remove query error test

* Add warning to query_error

* Add warning to resp3

* Add resp3 test

* format

* revert pytest

* resp3 tests

* Ben comments round1

* Add 0 thresh test

* comment

* ADd skip cluster

* change config and other Alon's comments

* small tess

* remove pause after
Remove duplicates in tests

* more test compression

* fix assert

* FT.PROFILE and OOM string

* Add resp2 warning

* Revert "Add resp2 warning"

This reverts commit 6b73b3c.

* Add resp2 warning

* format

* [MOD-9372] - Add GIL release in OOM wait (#6203)

release gil

* fix merge
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enforce:coverage Run coverage flow even on draft pull request size:L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants