[MOD-12694] [MOD-12069] Add active_worker_threads metric by meiravgri · Pull Request #7538 · RediSearch/RediSearch

meiravgri · 2025-11-26T15:23:11Z

Extends the multi_threading section in INFO MODULES by adding search_active_worker_threads metric to track currently executing worker thread queries.

Continues #7530 which added search_multi_threading section and search_active_io_threads field.

INFO MODULES Output

# search_multi_threading
...
search_active_worker_threads:0

Note

Expose current worker thread activity via search_active_worker_threads in INFO MODULES alongside existing I/O stats.

Metrics/INFO MODULES:
- Extend multi_threading with active_worker_threads in src/info/info_redis/info_redis.c using GlobalStats_GetMultiThreadingStats().
Global stats:
- Add active_worker_threads to MultiThreadingStats (src/info/global_stats.h).
- Populate via workersThreadPool_WorkingThreadCount() with safety assert workersThreadPool_isCreated() (src/info/global_stats.c).
Workers utility:
- Add workersThreadPool_isCreated() API (src/util/workers.{h,c}).
Tests:
- Add SA and cluster tests validating search_multi_threading.search_active_worker_threads behavior with paused queries (tests/pytests/test_info_modules.py).

^{Written by Cursor Bugbot for commit 399bb53. This will update automatically on new commits. Configure here.}

src/info/global_stats.c

tests/pytests/test_info_modules.py

JoanFM · 2025-11-26T18:23:59Z

src/info/global_stats.h


 typedef struct {
  size_t active_io_threads; // number of I/O thread callbacks currently executing
+  size_t active_coord_threads; // number of coordinator thread callbacks currently executing (reserved for future use)


let's remove the coord_threads and add them in another PR?

JoanFM · 2025-11-26T18:26:32Z

tests/pytests/test_info_modules.py

+                args=(runDebugQueryCommandPauseBeforeRPAfterN,
+                      (env, [query_type, 'idx', '*'], 'Index', 0, ['INTERNAL_ONLY']),
+                      result_list),
+                daemon=True


just comment, if we are joining, do not need to add daemon true

codecov · 2025-11-26T19:06:12Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.69%. Comparing base (9ccdf3e) to head (399bb53).
⚠️ Report is 3 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7538      +/-   ##
==========================================
- Coverage   84.69%   84.69%   -0.01%     
==========================================
  Files         349      349              
  Lines       53923    53928       +5     
  Branches    14485    14485              
==========================================
+ Hits        45671    45674       +3     
- Misses       8063     8065       +2     
  Partials      189      189

Flag	Coverage Δ
flow	`85.03% <100.00%> (-0.13%)`	⬇️
unit	`52.04% <80.00%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

cursor · 2025-11-27T05:07:45Z

src/info/global_stats.c

 MultiThreadingStats GlobalStats_GetMultiThreadingStats() {
  MultiThreadingStats stats;
  stats.active_io_threads = READ(RSGlobalStats.totalStats.multi_threading.active_io_threads);
+  stats.active_worker_threads = workersThreadPool_WorkingThreadCount();


Bug: NULL pointer dereference when worker pool uninitialized

The new call to workersThreadPool_WorkingThreadCount() in GlobalStats_GetMultiThreadingStats() can crash when the worker thread pool hasn't been initialized. The function workersThreadPool_WorkingThreadCount() only has a debug assertion (RS_ASSERT) for the null check, which is stripped in release builds. When _workers_thpool is NULL, calling redisearch_thpool_num_jobs_in_progress(_workers_thpool) will dereference a null pointer. This breaks existing tests like ActiveIoThreadsMetric in test_cpp_io_runtime_ctx.cpp that call GlobalStats_GetMultiThreadingStats() without initializing the worker pool. Other similar functions like workersThreadPool_getStats() properly return a default value when the pool is NULL.

thank you !

i decided not to call workersThreadPool_getStats because intenally it locks the thpool, which i like to avoid

The base branch was changed.

assert is initizlied in GlobalStats_GetMultiThreadingStats

JoanFM · 2025-11-27T12:18:43Z

src/info/global_stats.c

 MultiThreadingStats GlobalStats_GetMultiThreadingStats() {
  MultiThreadingStats stats;
  stats.active_io_threads = READ(RSGlobalStats.totalStats.multi_threading.active_io_threads);
+  RS_ASSERT(workersThreadPool_isInitialized()); // In production workers threadpool is initialized at startup.


This is not really true I believe. The ThreadPool is initialized lazily when some queries have been scheduled.

Ah I see, it differs from the state. We should call it differently

JoanFM · 2025-11-27T12:20:01Z

src/util/workers.c

 }
+
+bool workersThreadPool_isInitialized() {
+  return _workers_thpool != NULL;


Let's choice another naming to avoid the confusion with the thpool->state (UNITIALIZED, etc ...)

Removed skip decorator from test_active_io_threads_stats.

JoanFM

I would perhaps opt to do the assert inside the workers API

redisearch-backport-pull-request · 2025-11-27T22:21:34Z

Backport failed for 2.8, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin 2.8
git worktree add -d .worktree/backport-7538-to-2.8 origin/2.8
cd .worktree/backport-7538-to-2.8
git switch --create backport-7538-to-2.8
git cherry-pick -x d9574d8b0de0a9ead43c0de696686c187e410173

redisearch-backport-pull-request · 2025-11-27T22:21:36Z

Backport failed for 2.10, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin 2.10
git worktree add -d .worktree/backport-7538-to-2.10 origin/2.10
cd .worktree/backport-7538-to-2.10
git switch --create backport-7538-to-2.10
git cherry-pick -x d9574d8b0de0a9ead43c0de696686c187e410173

* add "active_worker_threads" metric * fix declartion * remove coord threads * make the tests run... * introduce workersThreadPool_isInitialized assert is initizlied in GlobalStats_GetMultiThreadingStats * cleanup * rename workersThreadPool_isCreated * Unskip test_active_io_threads_stats Removed skip decorator from test_active_io_threads_stats. (cherry picked from commit d9574d8)

redisearch-backport-pull-request · 2025-11-27T22:21:42Z

Successfully created backport PR for 8.2:

[8.2] [MOD-12694] [MOD-12069] Add active_worker_threads metric #7564

redisearch-backport-pull-request · 2025-11-27T22:21:44Z

Backport failed for 8.4, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin 8.4
git worktree add -d .worktree/backport-7538-to-8.4 origin/8.4
cd .worktree/backport-7538-to-8.4
git switch --create backport-7538-to-8.4
git cherry-pick -x d9574d8b0de0a9ead43c0de696686c187e410173

* [MOD-12694] [MOD-12069] Add active_worker_threads metric (#7538) * add "active_worker_threads" metric * fix declartion * remove coord threads * make the tests run... * introduce workersThreadPool_isInitialized assert is initizlied in GlobalStats_GetMultiThreadingStats * cleanup * rename workersThreadPool_isCreated * Unskip test_active_io_threads_stats Removed skip decorator from test_active_io_threads_stats. (cherry picked from commit d9574d8) * import threading * take common from master * fix merge * revert change * add threading --------- Co-authored-by: meiravgri <[email protected]> Co-authored-by: meiravgri <[email protected]>

* add "active_worker_threads" metric * fix declartion * remove coord threads * make the tests run... * introduce workersThreadPool_isInitialized assert is initizlied in GlobalStats_GetMultiThreadingStats * cleanup * rename workersThreadPool_isCreated * Unskip test_active_io_threads_stats Removed skip decorator from test_active_io_threads_stats. (cherry picked from commit d9574d8)

meiravgri · 2025-12-02T15:35:48Z

/backport

redisearch-backport-pull-request · 2025-12-02T15:36:09Z

Backport failed for 2.8, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin 2.8
git worktree add -d .worktree/backport-7538-to-2.8 origin/2.8
cd .worktree/backport-7538-to-2.8
git switch --create backport-7538-to-2.8
git cherry-pick -x d9574d8b0de0a9ead43c0de696686c187e410173

redisearch-backport-pull-request · 2025-12-02T15:36:11Z

Backport failed for 2.10, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin 2.10
git worktree add -d .worktree/backport-7538-to-2.10 origin/2.10
cd .worktree/backport-7538-to-2.10
git switch --create backport-7538-to-2.10
git cherry-pick -x d9574d8b0de0a9ead43c0de696686c187e410173

redisearch-backport-pull-request · 2025-12-02T15:36:13Z

Backport failed for 8.2, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin 8.2
git worktree add -d .worktree/backport-7538-to-8.2 origin/8.2
cd .worktree/backport-7538-to-8.2
git switch --create backport-7538-to-8.2
git cherry-pick -x d9574d8b0de0a9ead43c0de696686c187e410173

* [MOD-12694] [MOD-12069] Add active_worker_threads metric (#7538) * add "active_worker_threads" metric * fix declartion * remove coord threads * make the tests run... * introduce workersThreadPool_isInitialized assert is initizlied in GlobalStats_GetMultiThreadingStats * cleanup * rename workersThreadPool_isCreated * Unskip test_active_io_threads_stats Removed skip decorator from test_active_io_threads_stats. (cherry picked from commit d9574d8) * Merge branch '8.4' into backport-7538-to-8.4 * Revert "Merge branch '8.4' into backport-7538-to-8.4" This reverts commit 493db61.

WIP: needs #7579 backport

* [MOD-12694] [MOD-12069] Add active_worker_threads metric (#7538) WIP: needs #7579 backport * wrapp reading from workers only if mt_build fix test * fix test * review

* add "active_worker_threads" metric * fix declartion * remove coord threads * make the tests run... * introduce workersThreadPool_isInitialized assert is initizlied in GlobalStats_GetMultiThreadingStats * cleanup * rename workersThreadPool_isCreated * Unskip test_active_io_threads_stats Removed skip decorator from test_active_io_threads_stats. (cherry picked from commit d9574d8)

* [MOD-12694] [MOD-12069] Add active_worker_threads metric (#7538) * add "active_worker_threads" metric * fix declartion * remove coord threads * make the tests run... * introduce workersThreadPool_isInitialized assert is initizlied in GlobalStats_GetMultiThreadingStats * cleanup * rename workersThreadPool_isCreated * Unskip test_active_io_threads_stats Removed skip decorator from test_active_io_threads_stats. (cherry picked from commit d9574d8) * add deps in common.py * remove call_and_store * fix io test

github-actions bot added the size:M label Nov 26, 2025

cursor bot reviewed Nov 26, 2025

View reviewed changes

src/info/global_stats.c Show resolved Hide resolved

tests/pytests/test_info_modules.py Show resolved Hide resolved

github-actions bot added the size:S label Nov 26, 2025

meiravgri requested a review from JoanFM November 26, 2025 18:15

meiravgri changed the title ~~Meiravg_active_threads~~ [MOD-12694] [MOD-12069] Add active_worker_threads metric Nov 26, 2025

meiravgri added backport 2.8 backport 2.10 backport 8.2 backport 8.4 labels Nov 26, 2025

JoanFM reviewed Nov 26, 2025

View reviewed changes

JoanFM previously approved these changes Nov 26, 2025

View reviewed changes

cursor bot reviewed Nov 27, 2025

View reviewed changes

Base automatically changed from meiravg_uv_active_thread_counter to master November 27, 2025 10:10

meiravgri added 6 commits November 27, 2025 11:59

add "active_worker_threads" metric

41f2e1a

fix declartion

dc7e69d

remove coord threads

7d4733d

make the tests run...

acf5682

introduce workersThreadPool_isInitialized

53b2be3

assert is initizlied in GlobalStats_GetMultiThreadingStats

Merge remote-tracking branch 'origin/master' into meiravg_active_threads

be74191

meiravgri force-pushed the meiravg_active_threads branch from 161317e to be74191 Compare November 27, 2025 12:16

cleanup

9365a2a

meiravgri requested a review from JoanFM November 27, 2025 12:17

meiravgri enabled auto-merge November 27, 2025 12:18

JoanFM requested changes Nov 27, 2025

View reviewed changes

rename workersThreadPool_isCreated

e60d90a

JoanFM previously approved these changes Nov 27, 2025

View reviewed changes

meiravgri added this pull request to the merge queue Nov 27, 2025

Merge branch 'master' into meiravg_active_threads

d21bbd2

meiravgri dismissed JoanFM’s stale review via d21bbd2 November 27, 2025 18:06

Unskip test_active_io_threads_stats

399bb53

Removed skip decorator from test_active_io_threads_stats.

meiravgri requested a review from JoanFM November 27, 2025 18:08

JoanFM approved these changes Nov 27, 2025

View reviewed changes

meiravgri added this pull request to the merge queue Nov 27, 2025

Merged via the queue into master with commit d9574d8 Nov 27, 2025
26 checks passed

meiravgri deleted the meiravg_active_threads branch November 27, 2025 22:21

redisearch-backport-pull-request bot mentioned this pull request Nov 27, 2025

[8.2] [MOD-12694] [MOD-12069] Add active_worker_threads metric #7564

Merged

meiravgri mentioned this pull request Nov 30, 2025

[MOD-12069] Add *_pending_jobs metrics #7556

Merged

meiravgri mentioned this pull request Dec 2, 2025

[8.4] [MOD-12694] [MOD-12069] Add active_worker_threads metric #7616

Merged

meiravgri added a commit that referenced this pull request Dec 4, 2025

[MOD-12694] [MOD-12069] Add active_worker_threads metric (#7538)

7ee51bc

WIP: needs #7579 backport

meiravgri mentioned this pull request Dec 7, 2025

[2.8] [MOD-12694] [MOD-12069] Add active_worker_threads metric #7645

Merged

meiravgri mentioned this pull request Dec 9, 2025

[2.10] [MOD-12694] [MOD-12069] Add active_worker_threads metric #7688

Merged

Conversation

meiravgri commented Nov 26, 2025 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

INFO MODULES Output

Uh oh!

Uh oh!

Uh oh!

JoanFM Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

JoanFM Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

cursor bot Nov 27, 2025

Choose a reason for hiding this comment

Bug: NULL pointer dereference when worker pool uninitialized

Uh oh!

meiravgri Nov 27, 2025

Choose a reason for hiding this comment

Uh oh!

meiravgri Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JoanFM Nov 27, 2025

Choose a reason for hiding this comment

Uh oh!

JoanFM Nov 27, 2025

Choose a reason for hiding this comment

Uh oh!

JoanFM Nov 27, 2025

Choose a reason for hiding this comment

Uh oh!

JoanFM left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

redisearch-backport-pull-request bot commented Nov 27, 2025

Uh oh!

redisearch-backport-pull-request bot commented Nov 27, 2025

Uh oh!

redisearch-backport-pull-request bot commented Nov 27, 2025

Uh oh!

redisearch-backport-pull-request bot commented Nov 27, 2025

Uh oh!

meiravgri commented Dec 2, 2025

Uh oh!

redisearch-backport-pull-request bot commented Dec 2, 2025

Uh oh!

redisearch-backport-pull-request bot commented Dec 2, 2025

Uh oh!

redisearch-backport-pull-request bot commented Dec 2, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

meiravgri commented Nov 26, 2025 •

edited by cursor bot

Loading

codecov bot commented Nov 26, 2025 •

edited

Loading

meiravgri Nov 27, 2025 •

edited

Loading