[8.4] MOD-11751: Support WITHCOUNT keyword in FT.AGGREGATE by nafraf · Pull Request #7707 · RediSearch/RediSearch

nafraf · 2025-12-09T20:48:24Z

Manual backport #7202 to 8.4

Note

Add FT.AGGREGATE WITHCOUNT with accurate total_results (incl. cluster) using a shard response barrier and (safe) depleters, with parser, pipeline, coordinator, and test updates.

Implements FT.AGGREGATE WITHCOUNT end-to-end with accurate totals in clustered and standalone modes, plus supporting infra and tests.

Add request flags and parser handling for WITHCOUNT, SORTBY 0 [MAX n], GROUPBY; disallow WITHCOUNT with WITHCURSOR and guard behind ENABLE_UNSTABLE_FEATURES.
Pipeline: introduce depleter usage for accurate counting; add RPSafeDepleter (threaded) and a synchronous RPDepleter; arrange step helper; pager/sorter integration; optimizer tweaks.
Coordinator: add ShardResponseBarrier to wait for first replies from all shards and accumulate total_results; propagate errors/timeouts; RESP2/3 support; MR iterator/channel gain timed waits, private-data init/destructor, and shard index tracking; KNN/hybrid paths use safe depleter.
Misc fixes: use memmove in array_del, monotonic timed waits, iterator return codes, profiling strings/types.
Tests: extensive py/cpp coverage for WITHCOUNT, barrier behavior, profiles, pagers, hybrid pipelines, and depleters.

^{Written by Cursor Bugbot for commit d3973dd. This will update automatically on new commits. Configure here.}

src/coord/rpnet.c

codecov · 2025-12-09T22:45:50Z

Codecov Report

❌ Patch coverage is 93.71728% with 24 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.99%. Comparing base (7ba0c46) to head (d3973dd).
⚠️ Report is 1 commits behind head on 8.4.

Files with missing lines	Patch %	Lines
src/coord/rmr/chan.c	86.48%	5 Missing ⚠️
src/coord/dist_aggregate.c	76.47%	4 Missing ⚠️
src/coord/rpnet.c	96.55%	4 Missing ⚠️
src/coord/dist_utils.c	88.00%	3 Missing ⚠️
src/pipeline/pipeline_construction.c	90.62%	3 Missing ⚠️
src/result_processor.c	96.42%	3 Missing ⚠️
src/util/timeout.h	60.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##              8.4    #7707      +/-   ##
==========================================
- Coverage   86.00%   85.99%   -0.01%     
==========================================
  Files         331      331              
  Lines       53065    53362     +297     
  Branches    11998    11998              
==========================================
+ Hits        45637    45890     +253     
- Misses       7261     7305      +44     
  Partials      167      167

Flag	Coverage Δ
flow	`84.63% <93.45%> (-0.07%)`	⬇️
unit	`52.33% <19.37%> (-0.25%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

src/coord/dist_utils.c

src/coord/dist_aggregate.c

…ommand and response handling

This reverts commit 2843896.

src/coord/dist_utils.c

**Bug Fixes:** * [redis#7385](RediSearch/RediSearch#7385) Fix high temporary memory consumption when loading multiple search indexes from RDB * [redis#7430](RediSearch/RediSearch#7430) Fix a potential deadlock in `FT.HYBRID` in cluster mode during updates. * [redis#7454](RediSearch/RediSearch#7454) Fix a garbage collection performence regression * [redis#7460](RediSearch/RediSearch#7460) Fix potential double-free in Fork GC error paths * [redis#7455](RediSearch/RediSearch#7455) Fix internal cursors not being deleted promptly in cluster mode * [redis#7667](RediSearch/RediSearch#7667) Fix a cursor logical leak upon dropping the index * [redis#7796](RediSearch/RediSearch#7796) Fix a potential use-after-free when removing connections * [redis#7792](RediSearch/RediSearch#7792) Fix string comparison for binary data with embedded NULLs in TOLIST reducer in FT.AGGREGATE * [redis#7823](RediSearch/RediSearch#7823) Update `FT.HYBRID` to accept vector blobs only via parameters * [redis#7903](RediSearch/RediSearch#7903) Fix a memory leak in Hybrid ASM * [redis#8052](RediSearch/RediSearch#8052) Fix `FT.HYBRID` behavior when used with `LOAD *` * [redis#8082](RediSearch/RediSearch#8082) Fix incorrect FULLTEXT field metric counts * [redis#8089](RediSearch/RediSearch#8089) Fix an edge case in `CLUSTERSET` handling * [redis#8152](RediSearch/RediSearch#8152) Fix configuration registration issues **Improvements:** * [redis#7427](RediSearch/RediSearch#7427) Enhance `FT.PROFILE` with vector search execution details * [redis#7431](RediSearch/RediSearch#7431) Ensure full `FT.PROFILE` output is returned on timeout with RETURN policy * [redis#7507](RediSearch/RediSearch#7507) Track timeout warnings and errors in INFO * [redis#7576](RediSearch/RediSearch#7576) Track OOM warnings and errors in INFO * [redis#7612](RediSearch/RediSearch#7612) Track `maxprefixexpansions` warnings and errors in INFO * [redis#7960](RediSearch/RediSearch#7960) Persist query warnings across cursor reads * [redis#7551](RediSearch/RediSearch#7551), [redis#7616](RediSearch/RediSearch#7616), [redis#7622](RediSearch/RediSearch#7622), [redis#7625](RediSearch/RediSearch#7625) Add runtime thread and pending-jobs metrics * [redis#7589](RediSearch/RediSearch#7589) Support multiple slot ranges in `search.CLUSTERSET` * [redis#7707](RediSearch/RediSearch#7707) Add `WITHCOUNT` support to `FT.AGGREGATE` * [redis#7862](RediSearch/RediSearch#7862) Add support for subquery `COUNT` in `FT.HYBRID` * [redis#8087](RediSearch/RediSearch#8087) Add warnings when cursor results may be affected by ASM and expose ASM warnings in `FT.PROFILE` * [redis#8049](RediSearch/RediSearch#8049) Add logging for index-related commands * [redis#8150](RediSearch/RediSearch#8150) Fix shard total profile time reporting in `FT.PROFILE`

**Bug Fixes:** * [#7385](RediSearch/RediSearch#7385) Fix high temporary memory consumption when loading multiple search indexes from RDB * [#7430](RediSearch/RediSearch#7430) Fix a potential deadlock in `FT.HYBRID` in cluster mode during updates. * [#7454](RediSearch/RediSearch#7454) Fix a garbage collection performence regression * [#7460](RediSearch/RediSearch#7460) Fix potential double-free in Fork GC error paths * [#7455](RediSearch/RediSearch#7455) Fix internal cursors not being deleted promptly in cluster mode * [#7667](RediSearch/RediSearch#7667) Fix a cursor logical leak upon dropping the index * [#7796](RediSearch/RediSearch#7796) Fix a potential use-after-free when removing connections * [#7792](RediSearch/RediSearch#7792) Fix string comparison for binary data with embedded NULLs in TOLIST reducer in FT.AGGREGATE * [#7704](RediSearch/RediSearch#7704) Use asynchronous jobs for GC in SVS to accelerate execution * [#7823](RediSearch/RediSearch#7823) Update `FT.HYBRID` to accept vector blobs only via parameters * [#7903](RediSearch/RediSearch#7903) Fix a memory leak in Hybrid ASM * [#8052](RediSearch/RediSearch#8052) Fix `FT.HYBRID` behavior when used with `LOAD *` * [#8082](RediSearch/RediSearch#8082) Fix incorrect FULLTEXT field metric counts * [#8089](RediSearch/RediSearch#8089) Fix an edge case in `CLUSTERSET` handling * [#8152](RediSearch/RediSearch#8152) Fix configuration registration issues **Improvements:** * [#7427](RediSearch/RediSearch#7427) Enhance `FT.PROFILE` with vector search execution details * [#7431](RediSearch/RediSearch#7431) Ensure full `FT.PROFILE` output is returned on timeout with RETURN policy * [#7507](RediSearch/RediSearch#7507) Track timeout warnings and errors in INFO * [#7576](RediSearch/RediSearch#7576) Track OOM warnings and errors in INFO * [#7612](RediSearch/RediSearch#7612) Track `maxprefixexpansions` warnings and errors in INFO * [#7960](RediSearch/RediSearch#7960) Persist query warnings across cursor reads * [#7551](RediSearch/RediSearch#7551), [#7616](RediSearch/RediSearch#7616), [#7622](RediSearch/RediSearch#7622), [#7625](RediSearch/RediSearch#7625) Add runtime thread and pending-jobs metrics * [#7589](RediSearch/RediSearch#7589) Support multiple slot ranges in `search.CLUSTERSET` * [#7707](RediSearch/RediSearch#7707) Add `WITHCOUNT` support to `FT.AGGREGATE` * [#7862](RediSearch/RediSearch#7862) Add support for subquery `COUNT` in `FT.HYBRID` * [#8087](RediSearch/RediSearch#8087) Add warnings when cursor results may be affected by ASM and expose ASM warnings in `FT.PROFILE` * [#8049](RediSearch/RediSearch#8049) Add logging for index-related commands * [#8150](RediSearch/RediSearch#8150) Fix shard total profile time reporting in `FT.PROFILE`

github-actions bot added the size:XL label Dec 9, 2025

cursor bot reviewed Dec 9, 2025

View reviewed changes

src/coord/rpnet.c Outdated Show resolved Hide resolved

nafraf requested a review from oshadmi December 9, 2025 23:07

JoanFM mentioned this pull request Dec 16, 2025

[8.4] [MOD-13049] Do not rely on Shard Idx to decide where to send Command in cluster mode #7815

Merged

nafraf force-pushed the nafraf_backport-7202-to-8.4 branch from 8827599 to 5ac326a Compare December 17, 2025 16:25

cursor bot reviewed Dec 17, 2025

View reviewed changes

src/coord/dist_utils.c Show resolved Hide resolved

nafraf force-pushed the nafraf_backport-7202-to-8.4 branch from 5ac326a to 6d715c2 Compare December 18, 2025 15:38

cursor bot reviewed Dec 18, 2025

View reviewed changes

src/coord/dist_aggregate.c Show resolved Hide resolved

oshadmi reviewed Dec 22, 2025

View reviewed changes

src/coord/dist_aggregate.c Outdated Show resolved Hide resolved

nafraf added 6 commits December 22, 2025 16:43

Backport 7202 to 8.4

e1a46df

Add unstable_features guard

d33ca4f

Fix enable unstable features in tests

1f31c88

Refactor shard handling: replace targetShard with targetShardIdx in c…

e0c1cb1

…ommand and response handling

Fix printAggProfile()

4415b02

Revert "Fix printAggProfile()"

446deff

This reverts commit 2843896.

nafraf force-pushed the nafraf_backport-7202-to-8.4 branch from 61bab08 to 446deff Compare December 22, 2025 21:47

Update _get_cluster_RP_profile() - 8.4 prints 'Total GIL time'

d3973dd

cursor bot reviewed Dec 22, 2025

View reviewed changes

src/coord/dist_utils.c Show resolved Hide resolved

nafraf requested a review from oshadmi December 23, 2025 07:13

oshadmi approved these changes Dec 23, 2025

View reviewed changes

nafraf added this pull request to the merge queue Dec 23, 2025

Merged via the queue into 8.4 with commit 9501de5 Dec 23, 2025
49 of 53 checks passed

nafraf deleted the nafraf_backport-7202-to-8.4 branch December 23, 2025 16:12

alonre24 mentioned this pull request Jan 26, 2026

Update Redisearch module version to v8.4.5 redis/redis#14743

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[8.4] MOD-11751: Support WITHCOUNT keyword in FT.AGGREGATE#7707

[8.4] MOD-11751: Support WITHCOUNT keyword in FT.AGGREGATE#7707
nafraf merged 7 commits into8.4from
nafraf_backport-7202-to-8.4

nafraf commented Dec 9, 2025 •

edited by cursor bot

Loading

Uh oh!

Uh oh!

codecov bot commented Dec 9, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

nafraf commented Dec 9, 2025 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

nafraf commented Dec 9, 2025 •

edited by cursor bot

Loading

codecov bot commented Dec 9, 2025 •

edited

Loading