Skip to content

[2.6] Yield to redis while indexing - [MOD-9220] (#6103)#6195

Merged
BenGoldberger merged 9 commits into2.6from
backport-6103-to-2.6
Jun 3, 2025
Merged

[2.6] Yield to redis while indexing - [MOD-9220] (#6103)#6195
BenGoldberger merged 9 commits into2.6from
backport-6103-to-2.6

Conversation

@BenGoldberger
Copy link
Collaborator

This PR should fix the issue that a shard may not respond to "PING" while indexing on load.
this is done by yielding to redis after certain amount of indexing operations. the amount of operations between "yields" is configurable with FT.CONFIG SET INDEXER_YIELD_EVERY_OPS .
It is possible to check the number of "yields" was called with FT.DEBUG YIELDS_ON_LOAD_COUNTER

BenGoldberger and others added 8 commits May 22, 2025 13:48
* add op-counter + config

* Run all Rust benchmarks in the workspace (#6030)

* Run all Rust benchmarks in the workspace

* Allow forcing the run for micro-benchmarks

* Fix bench run

* Update .github/workflows/flow-micro-benchmarks.yml

Co-authored-by: Tim Janus <[email protected]>

---------

Co-authored-by: Tim Janus <[email protected]>

* Fix Rust build after switching off readies (#6086)

Fix Rust linking

* [MOD-9560] Change default config value for _BG_INDEX_MEM_PCT_THR (#6053)

* change default value

* change default value in config pytest

* change index oom tests to lower value then new default

* change debug commands tests to lower value then default value

* change default value of set tight memory functions

* 100testv1

* test default value

* Build unit tests without readies [MOD-9099] (#6082)

* purge readeis from cmake. Use new build script instead of makefile

* call script in CI

* libuv

* restore build folder

* libuv

* unit tests

* pretty build.sh

* pytest

* fixed extension build error

* Add profile flags + try pass linker flag properly (wip)

* fix build and linkage + pytest + adjust json

* small cleanup

* Add pytz dep

* bump googletest version, use same variant name as before

* remove ld libs

* Fix executable linker flags - build unit test properly.
run unit tests with old "make unit-tests" way (until this is fixed via ./build.sh)

* fix rust build via build.sh

* fix san build

* Set coverage flags and use the same bin dir for sanitizer (build it for debug)

* update benchmark image for regression test

* remove leftover

* Build hiredis static

* fix json env flag in CI

* CR changes

* Restore deleted files until unit tests are done as well, use boolean params

* restore vecsim version

* fix profile and fix build error

* fix for profile

* try fix mac build

* fix unit tests for arm

* Define boost dir

* remove the policies

* try fix the binroot for unit-tests

* change dir name for arm

* remove policy from hash as well

* use clang in mac

* try to set hardcoded clang

* try to mimic readies in macos includes

* try set CMAKE_OSX_DEPLOYMENT_TARGET

* whitespace formatting

* Set CC to clang in apple

* try to export llvm

* try set compiler in build script

* use bin in path

* try EXPORT properly with homebrew

* remove setting bad path to clang

* set clang path

* update to llvm@18

* update c++ path

* set compiler with LLVM env var

* fix?

* set proper link flags for mac

* remove bsymbolic from hiredis

* revert hiredis in non macos to how it was

* clean stuff

* cleanups

* Fix the extract debug symbols command so it will work as before (required for packing properly)

* update deps

* fix for alpine

* Refactor unit-tests script to not using readies WIP

* revert vecsim accidental change

* Fix script so it will work for sanitizer as well, address CR

---------

Co-authored-by: DvirDukhan <[email protected]>
Co-authored-by: GuyAv46 <[email protected]>

* Wrongly included the text of GPLv3 instead of AGPLv3 (#6089)

* README.md - added no standalone released note (#6074)

* Update README.md

* Update README.md

* Update README.md

* small change to opcounter

* debug command for yield counter + test

* add config test

* Update SECURITY.md (#6070)

* Update SECURITY.md

* Update SECURITY.md

* Enforce license headers for Rust files (#6090)

* A small binary to enforce license headers in all Rust files

* Enforce license headers in CI

* MOD-6151: Build without readies - simplify packing (#5908)

* fix

* change ramp version

* fix macos

* GHA3

* fix macos

* fix maxos

* fix

* fix

* fix

* new

* purge readies

* double

* typo

* remove debug

* shapshot

* gp

* go conflict

* darwin to macos

* Account for get-platform change, remove redundant mkdir, remove not used/used once variables

* remove unused DEP_NAMES

* remove unused function

* remove un condition

* improve pack_ramp

* remove unused DEPS

* macos

* remove xtx

* replace eprint

* remove runn

* replace realapth

* Revert "remove xtx"

This reverts commit 6efb895.

* try remove xtx

* remove NOP

* remove NUMVER

* without tmp

* delete

* remove eval

* remove release

* Assume SNAPSHOT=1

* remove SEMVER

* fin

* remove sbin/getver

* Don't add unnecessary \n after the license header (#6095)

Otherwise, `cargo fmt` fails on them.

* [MOD-9547] Core trie iterators (#6016)

* Basic iterators

* Typo

* Clean up API

* Add tests for iterators

* More comments

* Fix warning for miri

* Verify that all prefixed iterators agree with each other and match expectations

* Test both lending and non-lending traversals

* Clarify comment

* Test the empty key case

* Fix where bounds on traversal_filter

* Add missing license headers

* SSPLv1.txt - removed irrelevant text (#6097)

* MOD-9612: Fix flaky test and change early timeout error message (#6100)

* Fix flaky test AND change early timeout error message

* Fix TimeLimit initialization

* Add comment

* remove deps changes

* remove deps changes

* fix a possibly new flaky test

* temp fix for config

* changes to use the config_cmd func

* try to fix issue test

* use cluster conn in test

* handle num of yield with cluster

* dont test with cluster

* pr changes

* move whitespace

* add tag and geoshape idx

* pr changes

* changes to yield only while server is loading

* and check if module function exist

* remove white space

* add wait_for_index in test

* pr change rename config

* change help text

* add isLoading argument

* check globally if loading with g_isLoading

* pr change

---------

Co-authored-by: Luca Palmieri <[email protected]>
Co-authored-by: Tim Janus <[email protected]>
Co-authored-by: lerman25 <[email protected]>
Co-authored-by: alonre24 <[email protected]>
Co-authored-by: DvirDukhan <[email protected]>
Co-authored-by: GuyAv46 <[email protected]>
Co-authored-by: Lior Kogan <[email protected]>
Co-authored-by: Zeeshan Ali Khan <[email protected]>
Co-authored-by: Raz Monsonego <[email protected]>
(cherry picked from commit 035a8b4)
@codecov
Copy link

codecov bot commented May 22, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 85.68%. Comparing base (8dcc897) to head (dcc3c55).
Report is 4 commits behind head on 2.6.

Additional details and impacted files
@@            Coverage Diff             @@
##              2.6    #6195      +/-   ##
==========================================
- Coverage   85.70%   85.68%   -0.03%     
==========================================
  Files         189      189              
  Lines       31653    31687      +34     
==========================================
+ Hits        27129    27150      +21     
- Misses       4524     4537      +13     
Flag Coverage Δ
flow 79.45% <100.00%> (-0.27%) ⬇️
unit 44.44% <17.64%> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@BenGoldberger BenGoldberger requested review from GuyAv46 and alonre24 May 22, 2025 14:01
@BenGoldberger BenGoldberger added this pull request to the merge queue Jun 3, 2025
Merged via the queue into 2.6 with commit 4a4cc0e Jun 3, 2025
10 checks passed
@BenGoldberger BenGoldberger deleted the backport-6103-to-2.6 branch June 3, 2025 11:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants