Skip to content

MOD-8862: Add DebugCTX mechanism and DebugIndexesScanner#5672

Merged
lerman25 merged 19 commits intomasterfrom
OmerL_AddDebugContext
Feb 26, 2025
Merged

MOD-8862: Add DebugCTX mechanism and DebugIndexesScanner#5672
lerman25 merged 19 commits intomasterfrom
OmerL_AddDebugContext

Conversation

@lerman25
Copy link
Collaborator

@lerman25 lerman25 commented Feb 23, 2025

This pull request adds several new ft.debug commands, DebugCTX global and a DebugIndexesScanner "derived" from IndexesScanner.

The new debug commands are:

  1. SET_MAX_SCANNED_DOCS - limit the amount of docs that can be scanned in a background-scan.
  2. SET_PAUSE_ON_SCANNED_DOCS - pausing the scan after an amount of docs scanned.
  3. SET_PAUSE_BEFORE_SCAN - pausing the scanning process before reading the first docm
  4. SET_RESUME - resuming any paused state (such as 2,3).
  5. GET_DEBUG_SCANNER_STATUS - getting the status of the (debug) scanner.

Changing 1,2,3 will put DebugCTX in debug mode, which is revertible by setting negative/zero values.

DebugCTX can be used for other debugging mechanism other then indexes scan.

Also remove whitespaces from a previous pytest

@lerman25 lerman25 requested review from GuyAv46 and alonre24 February 23, 2025 15:03
GuyAv46
GuyAv46 reviewed Feb 23, 2025
View reviewed changes
Copy link
Collaborator

@GuyAv46 GuyAv46 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice implementation
Some styling comments and a few suggestions
Keep up the good work!

@codecov
Copy link

codecov bot commented Feb 24, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 92.96875% with 9 lines in your changes missing coverage. Please review.

Project coverage is 88.05%. Comparing base (c2bab6e) to head (b42bf33).
Report is 8 commits behind head on master.

Files with missing lines Patch % Lines
src/debug_commands.c 89.02% 9 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #5672      +/-   ##
==========================================
+ Coverage   88.04%   88.05%   +0.01%     
==========================================
  Files         197      197              
  Lines       35577    35703     +126     
==========================================
+ Hits        31322    31437     +115     
- Misses       4255     4266      +11

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@lerman25 lerman25 requested review from meiravgri and removed request for alonre24 February 24, 2025 14:58
@lerman25 lerman25 force-pushed the OmerL_AddDebugContext branch from 940a102 to 1e64b19 Compare February 25, 2025 10:05
alonre24
alonre24 reviewed Feb 25, 2025
View reviewed changes
alonre24
alonre24 reviewed Feb 25, 2025
View reviewed changes
alonre24
alonre24 reviewed Feb 25, 2025
View reviewed changes
alonre24
alonre24 reviewed Feb 25, 2025
View reviewed changes
alonre24
alonre24 reviewed Feb 25, 2025
View reviewed changes
alonre24
alonre24 reviewed Feb 25, 2025
View reviewed changes
alonre24
alonre24 previously approved these changes Feb 26, 2025
View reviewed changes
GuyAv46
GuyAv46 reviewed Feb 26, 2025
View reviewed changes
GuyAv46
GuyAv46 approved these changes Feb 26, 2025
View reviewed changes
Copy link
Collaborator

@GuyAv46 GuyAv46 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice

@lerman25 lerman25 enabled auto-merge February 26, 2025 15:06
@lerman25 lerman25 added this pull request to the merge queue Feb 26, 2025
Merged via the queue into master with commit a0536fe Feb 26, 2025
10 checks passed
@lerman25 lerman25 deleted the OmerL_AddDebugContext branch February 26, 2025 18:02
@lerman25 lerman25 restored the OmerL_AddDebugContext branch March 2, 2025 08:22
@lerman25
Copy link
Collaborator Author

lerman25 commented Mar 2, 2025

\backport

@lerman25
Copy link
Collaborator Author

lerman25 commented Mar 2, 2025

/backport

@redisearch-backport-pull-request
Copy link
Contributor

redisearch-backport-pull-request bot commented Mar 2, 2025

Successfully created backport PR for 8.0:

redisearch-backport-pull-request bot pushed a commit that referenced this pull request Mar 2, 2025
@lerman25 @github-actions
MOD-8862: Add DebugCTX mechanism and DebugIndexesScanner (#5672)
92408f5 
* Create dbg ctx and dbgIndScan

* Fix

* fixing mechanism and pytest

* Add pause before scan mechanism and pytest

* Revert "Add pause before scan mechanism and pytest"

This reverts commit b842ce9.

* Re-adding 'Add pause before scan mechanism and pytest'

* fix testDebugHelp

* fixing Guy's comment + skipping on cluster in pytest

* Move volatile pause to BG scanner,  move debug scanner flag to base IndexesScanner

* fix pytest

* thread safety

* Cover more code

* pause before scan coverage

* fix - can't create a flow to reach    'Debug mode enabled but scanner is not a debug scanner' error

* fixing Alon's comment

* wrong arity check and comment on not used function

* format

(cherry picked from commit a0536fe)
@redisearch-backport-pull-request redisearch-backport-pull-request bot mentioned this pull request Mar 2, 2025
Merged
github-merge-queue bot pushed a commit that referenced this pull request Mar 2, 2025
@redisearch-backport-pull-request @lerman25
[8.0] MOD-8862: Add DebugCTX mechanism and DebugIndexesScanner (#5711)
17a01a5 
MOD-8862: Add DebugCTX mechanism and DebugIndexesScanner (#5672)

* Create dbg ctx and dbgIndScan

* Fix

* fixing mechanism and pytest

* Add pause before scan mechanism and pytest

* Revert "Add pause before scan mechanism and pytest"

This reverts commit b842ce9.

* Re-adding 'Add pause before scan mechanism and pytest'

* fix testDebugHelp

* fixing Guy's comment + skipping on cluster in pytest

* Move volatile pause to BG scanner,  move debug scanner flag to base IndexesScanner

* fix pytest

* thread safety

* Cover more code

* pause before scan coverage

* fix - can't create a flow to reach    'Debug mode enabled but scanner is not a debug scanner' error

* fixing Alon's comment

* wrong arity check and comment on not used function

* format

(cherry picked from commit a0536fe)

Co-authored-by: lerman25 <[email protected]>
@lerman25 lerman25 mentioned this pull request Mar 17, 2025
Merged
@lerman25 lerman25 mentioned this pull request May 13, 2025
Merged
lerman25 added a commit that referenced this pull request Jun 5, 2025
@lerman25
MOD-8862: Add DebugCTX mechanism and DebugIndexesScanner (#5672)
1f94be1 
* Create dbg ctx and dbgIndScan

* Fix

* fixing mechanism and pytest

* Add pause before scan mechanism and pytest

* Revert "Add pause before scan mechanism and pytest"

This reverts commit b842ce9.

* Re-adding 'Add pause before scan mechanism and pytest'

* fix testDebugHelp

* fixing Guy's comment + skipping on cluster in pytest

* Move volatile pause to BG scanner,  move debug scanner flag to base IndexesScanner

* fix pytest

* thread safety

* Cover more code

* pause before scan coverage

* fix - can't create a flow to reach    'Debug mode enabled but scanner is not a debug scanner' error

* fixing Alon's comment

* wrong arity check and comment on not used function

* format
@lerman25 lerman25 mentioned this pull request Jun 8, 2025
Merged
github-merge-queue bot pushed a commit that referenced this pull request Jun 10, 2025
@lerman25
[2.10] Backport OOM indexing to all branches [MOD-9715] (#6262)
88f146d 
* MOD-8862: Add DebugCTX mechanism and DebugIndexesScanner (#5672)

* Create dbg ctx and dbgIndScan

* Fix

* fixing mechanism and pytest

* Add pause before scan mechanism and pytest

* Revert "Add pause before scan mechanism and pytest"

This reverts commit b842ce9.

* Re-adding 'Add pause before scan mechanism and pytest'

* fix testDebugHelp

* fixing Guy's comment + skipping on cluster in pytest

* Move volatile pause to BG scanner,  move debug scanner flag to base IndexesScanner

* fix pytest

* thread safety

* Cover more code

* pause before scan coverage

* fix - can't create a flow to reach    'Debug mode enabled but scanner is not a debug scanner' error

* fixing Alon's comment

* wrong arity check and comment on not used function

* format

* MOD-7795: Background indexing memory limitation & configuration

* fix test

* fix-test2

* fix flaky OOM tests MOD-9406 (#5941)

* fix flaky

* comment

* change to ceil

* [MOD-9560] Change default config value for _BG_INDEX_MEM_PCT_THR (#6053)

* change default value

* change default value in config pytest

* change index oom tests to lower value then new default

* change debug commands tests to lower value then default value

* change default value of set tight memory functions

* 100testv1

* test default value

* [MOD-9560] Change default config value for _BG_INDEX_MEM_PCT_THR

* fix json

* [MOD-9372 , MOD-9733] Stop indexing OOM - Add wait before OOM

* skip query error test

* fix typo

* support different default values

* change test config

* fix config

* skip non cluster

* [MOD-9560] Revert marking index as faulty after reaching OOM

* [MOD-9372] - Add GIL release in OOM wait

* remove cargo

* fix pytest
@lerman25 lerman25 mentioned this pull request Jun 15, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alonre24 alonre24 alonre24 approved these changes

@GuyAv46 GuyAv46 GuyAv46 approved these changes

@meiravgri meiravgri Awaiting requested review from meiravgri

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@lerman25 @alonre24 @GuyAv46