* remove readies from memcheck

* add colors

* red

* test: update KNN query to use large_k and remove redundant checks

* revert unnecessary changes

* Rust Wildcard Iterator: WIP

* tests

* benchmarks

* Avoid bench ffi overhead with Direct C impl

* Remove overheaded-ffi tests from bench

* Fix build

* rename c bench file

* PR: rename and fixes

* PR: fix and remove redundant test

* PR: provide benchmarks params as test inputs

* PR: include inverted_index only for iterators bencher

* PR: last comments

* PR: set freq to 1

* PR: remove many redundant benchmarks

* PR: scope wildcard iterator result lifetime

* Add guardrail

* test failure on OOM

* Refactor OOM tests to use common setup function

* Update TODO comment for QUERY_EHYBRID_HYBRID_ALIAS in query_error.h

* Apply guy suggestions

(cherry picked from commit fd949e16466a627747cbfd7ab35cf497a3ed5110)

* change goto error to return so no undeclared var issue

add missing lines after feature branch merge

* fix typedef

typedef <type> <name>;

* swap to rust RPCounter, delete C code

* port RPProfile count incrementing behavior

* remove version

* undo import formatting changes

* address comments and (temporarily) fix tests

- adds a `Context::parent` function instead of a `last_processor`
- adds a `RPProfile_IncrementCount` mock function so that the rust
  expecting the C function to exist will still pass

* cargo fmt --all

* WIP

* mock QueryProcessingCtx in Chain

* comment

* use Pin<Box<ffi::QueryProcessingCtx>>

* comment

* add unreachable!() in stub impl, simplify pointer cast

* always set parent in each appended result_processor

* fix miri error: UnsafeCell<*mut QueryProcessingCtx>

* make fmt

* put UnsafeCell inside QueryProcessingCtx

* whoops

* Add GC marker to inverted index

* Add GC marker to reader

* FFI for reader revalidate

* FFI for ii gc marker

These will only be used by some C tests.

* Take mutable parameter for updated variable

…ression fallback (#6914)

* check if alpine

* use distro.name().lower()

* some bug fixes

* add withscores test

* remove print

* fix tests

* code review fixes

* code review fix

* add wait for gc

* remove FDs check

* add wait for gc

* remove FDs check

* Move `RSValue_SendReply` to `reply` module

Moves `RSValue_SendReply` to `reply` module, renaming
it to `RedisModule_Reply_RSValue`. Furthermore,
exposes `RSValue_NumToString` from `value.h`,
as it's used in `RedisModule_Reply_RSValue`,
adding a debug assertion that the pased `RSValue` is
indeed a number, so as to make things safer.

* Remove RSValue_SendReply_Collection

Removes `RSValue_SendReply_Collection`, which
was disabled, removing the last reference to
the `reply` module from `value`

* Clarify requirement of RSValue_NumToString parameter being a number

* Use `RS_ASSERT` instead of `assert` in `RSValue_NumToString`

Co-authored-by: GuyAv46 <[email protected]>

---------

Co-authored-by: GuyAv46 <[email protected]>

* initial commit

* code review fixes

* fix compilation

* code review comment fix + fix cpp tests

* fix tests

* fix tests

* address code review comment

* initial commit

* check folders existence

* revert some changes

* improve the script

* code review comments + group tests in github actions

* code review fixes

* code review fixes

* code review fixes

* Use mutable reference for decoding

* Update tests

* Update benchers

* Update FFI

* Fix safety comments

* Flatten logic

* Combine lifetimes on enum

* Only decoding to return an new instance

* Fix linting issue

* Simplify return

Co-authored-by: GuyAv46 <[email protected]>

* Black box benchmark correctly

---------

Co-authored-by: GuyAv46 <[email protected]>

…okup_GetItem` (#6807)

MOD-11307 implement RLookupRow::get

Co-authored-by: Jonas Kruckenberg <[email protected]>

* try and support yield distance and score

* support score alias in search sub query

* fix compilation

* add tests and fix issues they raised

* fix cpp test

* remove positional argument handling - doesn't fit the flow we are looking for

* try and align branch to requested requirements

* small changes to remove code which is not needed

* code review comments

* fix tests - add error code to arg parser to allow some error handling when combine id dealing with keywords while parsing combine keyword

* code review comment - fix tests

* fix test

* revert YIELD_DISTANCE_AS support, move YIELD_SCORE_AS in combine to be part of RRF/LINEAR arg count

* revert some changes

* fix compilation

* remove arg parse changes

* revert some combine parsing code

* change test names, revert some additional code

* add error when failing to reserve alias

* try and fix tests

* fix some flow tests

* fix flow test + code review comment

* forgot to push a test fix

* review comments

#6885)

* advance ac after finding the right definition

* update the license for arg parser

* if min and max are equal, use slice, otherwise use variable length args

* code review comments

* change how positional arguments are handled - we require the name to match and only then the value

* update license

* initial commit

* update combine handling function

* minor changes to parsing code - changes options types since some don't have count

* fix options

* post rebase fixes

* post rebase fixes

* fix CI

* code review fixes

* bit collision fix

* fix tests

* fix typo

* changing constant arg definition to be a double

* fix timeout parsing

* remove temp variable

* fix error messages

* additional error fixes

* free params dict on error

* avoid double free

* code review comments

* remove constant range validation

* improve error message, try to improve coverage

* test: change test to reflect desired behavior

* default dialect minimum 2 for Hybrid

* test: fix unit test

* fail if DIALECT is part of the query

* do checks in different parts of query

* fix DIALECT error messages

* fix knn and range subqueries

* test: fix tests failing because dialect overriden

* update src/hybrid/parse_hybrid.c

* apply suggestions from code review

Co-authored-by: Omer Shadmi <[email protected]>

* force subqueries same dialect Version as parent

* test: fix tests failing in CI, no DIALECT in query

* fix: fix according to review

* Fix dialect stats, add test with EXPLAINSCORE

* change the error message

* test: fix test that was explicitly setting DIALECT

---------

Co-authored-by: jonathan keinan <[email protected]>
Co-authored-by: Omer Shadmi <[email protected]>
Co-authored-by: oshadmi <[email protected]>

…6930)

* LIMIT and KNN parameters should behave independently

* test: fix tests showing default K independent from Limit

* MOD-11090: Mark DROP commands as touches-arbitrary-keys

* Fix spell check

* changed compatible_redis_version to 8.0.0

* min_redis_version from 8.0.0 to 8.0

* change compatible_redis_version to 8.0.2

* change redis_compatible_version from 8.0.2 to 8.0

* Restore compatible_redis_version 99.99.99

* changed compatible_redis_version to 8.0

* changed redis_compatible_version to 8.2

* compatible_redis_version=8.0

* Restore compatible Redis version to 99.99.99

* Apply phrasing suggestion from @nafraf

Co-authored-by: nafraf <[email protected]>

---------

Co-authored-by: galilev <[email protected]>
Co-authored-by: nafraf <[email protected]>
Co-authored-by: nafraf <[email protected]>

* Use getter for inverted index access

* Only set numeric filter if not null

* Numeric encoder can have duplicate entries

* Raw doc id encoder should use first block id for base

* Make helper inline static

This is to get around multiple definitions compile errors.

* Set wide schema flag on tested index

These tests add records with a wide field mask but were not using the
encoders which supports the wide field mask. So this fixes that.

* Update include path

* Have `NewIndexReader` take an owned decoder ctx

* Remove calls to size of ii and index blocks

These sizeof calls won't be possible once these are in Rust.

* Remove access to block size constants

These won't exist in the Rust move. I also don't know why the tests did
not have these hardcoded in the first place.

* Add comment to panic

* Fix spelling

* Fix tiny negatives numeric encoding

These used to be incorrectly encoded as positive tiny numbers.

* Write numeric records to a numeric index

The Rust code will panic without this.

* FFI for IndexBlock

This is only needed by some C tests.

* Fix comment

Co-authored-by: GuyAv46 <[email protected]>

---------

Co-authored-by: GuyAv46 <[email protected]>

* MOD-10399 encapsulate SearchResult type, removing direct field accesses

* refactor: Make `indexResult` field of `SearchResult` immutable

In sync with the work done in #6846 this change makes `SearchResult` hold onto an immutable point to the `RSIndexResult` instead of a mutable one.
The change does the necessary (limited) cleanup to make this work.

removes `SearchResult_GetIndexResultMut`

* fix: crash revert SearchResult_Override

* upgrade to macos15

* try remove xlarge

* print cpu information

* fix print cpu

* fix

* apply window parameter in Hybrid linear commands

* parse WINDOW when parsing Linear clause

* test: test the behavior of windows setting w.r.t limit

* test: add tests for parsing WINDOW in Linear

* code review changes

* test: fix tests

* limit K by WINDOW also for Linear

* update src/hybrid/parse_hybrid.c

* update tests/cpptests/test_cpp_hybrid_defaults.cpp

Co-authored-by: Itzikvaknin <[email protected]>

---------

Co-authored-by: Itzikvaknin <[email protected]>

* fix total_result accuracy when using FILTER

* correctly handle responses with errors

* access total_result using recursive_index

* code review fixes

* another cursor code review comment fix

* fix test

* fix tests

* code review comments

* rebase + fix test

* call RLookup_AddKeysFrom(sourceLookups[i], tailLookup) after consumiong that upstream

* CR comment

* CR comment

* apply window parameter in Hybrid linear commands

* parse WINDOW when parsing Linear clause

* test: test the behavior of windows setting w.r.t limit

* test: add tests for parsing WINDOW in Linear

* code review changes

* test: fix tests

* limit K by WINDOW also for Linear

* update src/hybrid/parse_hybrid.c

* update tests/cpptests/test_cpp_hybrid_defaults.cpp

Co-authored-by: Itzikvaknin <[email protected]>

* propagate RRF and LINEAR window parameters to arrange step limit in search subquery.

* Fix hybrid.py:_validate_results

* Fix unit tests

* Increase coverage

---------

Co-authored-by: Joan Fontanals Martinez <[email protected]>

upgrade hiredis

* Hide internals of `RSValue`

Ensure code outside of the `value` C module
does not access internals of the `RSValue`
type, but instead uses several newly created
constructors, setters, and getters.

The `RS_DUOVAL_OTHER2VAL` is to be eliminated
in a later commit that renames the `Duo`
variant of `RSValueType` to `Trio`.

`RSVALUE_MAP_KEYPOS` and `RSVALUE_MAP_VALUEPOS`
will be eliminated in a redesign of the map creation
functionality of `RSValue`

* Rename `RSValue` `Duo` type to `Trio`

In the past, `RSValue` has been adapted such
that its `Duo` field actually holds three
values. In preparation for the migration
to Rust of the `value` module, this type
is renamed to `Trio`, eliminating the
`RS_DUOVAL_OTHER2VAL` macro in the process.

* Refactor `RSValue` map API

Add types `RSValueMap` and `RSValueMapEntry`,
representing an `RSValue` map variant and
its entries. Add functions `RSValueMap_Create_Uninit`,
`RSValueMap_SetEntry`, and `RSValue_NewMap`
to allow for creating maps and wrapping them
in `RSValue`s.

* Move `APIVERSION_RETURN_MULTI_CMP_FIRST` from `value.h` to `search_ctx.h`

* Eliminate macros `RSVALUE_REPLACE` and `RSVALUE_CLEARVAR`

* Rename `RSValue_Decref` to `RSValue_DecrRef`

For consistency with `RSValue_IncrRef`,
renames `RSValue_Decref` to `RSValue_DecrRef`.

Also removes and updates outdated comments.

* test: add tests for filter hybrid behavior

* test: improve testing

* fix: fix test

* test: test behavior of double filter pre and post

* complete test with 2 filters no combine

* Add 1 more test

* test: clean comments

* fix: fix problem after merge with master

* change asserts

---------

Co-authored-by: nafraf <[email protected]>

* register FT.ADD, FT.GET and FT.DEL with key

* fix all test to correctly use ft.add ft.get and ft.del

* revert test deletion

* fix spellings

* fix ACL and add a test

* test SAFEADD as well

…SValue` (#7001)

* Initial type layout for Rust `RsValue` and `SharedRsValue`

- Move `RSValueTrait` and `RSValueFFI` in the `value` crate
  to a separate module, and put them behind a feature gate
  as the module depends on the `ffi` crate which conflicts
  with cbindgen when it attempts to generate bindings for
  the `value` crate. This feature is to be removed
  when `RsValueFFI` is eliminated in a later stage of the
  porting process.
- Define `RsValue` and `SharedRsValue` types, both still
  incomplete and unoptimized, so that we can generate bindings
  referencing these types. `RsValue` is a stack-based type,
  whereas `SharedRsValue` is heap-allocated and reference
  counted.
- Define `RsValueMap`, `RsValueTrio`, as well as `RsValueInternal`
  to represent the internals of `RsValue` and `SharedRsValue`.

Signed-off-by: Henk Oordt <[email protected]>

* Create `value_ffi` crate as the FFI specification for `value`

Create `value_ffi` to mimic the C `value.h` interface.
`RSValue_NewWithType`, `RSValue_WithType`, and
`RSValue_NewStringWithType` are omitted intentionally,
and are to be replaced with specialized constructors
that fully initialized the produced value.

Signed-off-by: Henk Oordt <[email protected]>

---------

Signed-off-by: Henk Oordt <[email protected]>

fix goto

* fix bugs regarding misalignment and missing symbols

- redis allocator mocks had misalignment issues
- symbols were missing (e.g. RedisModule_Alloc)
- ptrs vs objects

This was causing issues with 'cargo test' on
rqe_iterators_bencher on Linux, but it was all good
on MacOS.

* restore previous comments

* fix license header

* Add explanation about h1

* apply feedback from @LukeMathWalker

…policies (#7017)

* test: add test validating crash from different timeout policies

* test: improve test as per comments

* test: improve spped by using pipeline

* handle smaller amount of docs

* Update tests/pytests/test_cluster_aggregate_timeout.py

Co-authored-by: GuyAv46 <[email protected]>

---------

Co-authored-by: GuyAv46 <[email protected]>

* ci: show weird test failure

* fix: initialize keyspace notifciations when starting test

* revert change on github

* ci: add back MAX_WORKER_THREADS option to Makefile

* ci: test in CMakeCache.txt

* revert changes on actions

* preperations for the new version

* remove any code path for JAPI<6 only

* update API file

* align with new API

* some more cleanup

* update JSON target branch when building

* fix API usage

* Apply API alignment

* WIP: Register Register DistHybridCommand()

* WIP: Integrate hybrid_dispatcher

* MOD-11172: Add Argument Parser Struct To Improve Parsing Experience (#6856)

* advance ac after finding the right definition

* update the license for arg parser

* if min and max are equal, use slice, otherwise use variable length args

* code review comments

* change how positional arguments are handled - we require the name to match and only then the value

* update license

* Add dispatcher to rpnet

* WIP: HybridRequest_BuildDistributedDepletionPipeline

* WIP: Fixing HybridRequest_buildDistRPChain()

* Intermediate work on coordinator FT.HYBRID

* try to fix HybridRequest_buildMRCommand

* Test HybridRequest_buildMRCommand()

* Minor fixes

* Intermediate work on dispatcher refactor

* Update parseHybridCommand call to include internal arg

* Intermediate work on dispatcher

* Intermediate work on hybrid_cursor_mappings

* initial commit

* Intermediate work on RPnetNext in FT.HYBRID
Timeout case is disabled at the moment

* Working coordinator in case  __key is loaded,
Still very missing solution

* Create dist_utils.c

* Add timeout initialization for each subquery

* Handle erros in processCursorMappingCallback()

* Fix vector subquery flag

* Fixing memory leak

* Fixing HybridRequest_prepareForExecution()

* Fixing memory leaks

* Cleanup

* MOD-11654: Deserialize Score If It Exists (#6968)

* Update Feature Branch With Master (#6970)

* MOD-11654: FT.HYBRID - Merging Requires __key To Exists (#6941)

* MOD-11660: Coordinator upstreams sorters heap size (#6969)

* Move coordinator subqueries areqs construction to hybridRequestSetupCoordinatorSubqueriesRequests

* Fix hybridRequestSetupCoordinatorSubqueriesRequests implementation
Add a flag indicating is coordinator subquery so getArrangeRP will add a sorter as expected

* Small refactoring of RPHybridMerger_Accum (#6974)

* MOD-11693: FT.HYBRID - Coordinator Gracefully Handle Errors (#6981)

* initial commit

* code review  fixes

* remove newline

* code review fixes

* fix typos

* initialize count with current response count in case we already got some responses.

* small fix

* rephrase error messages

* Update src/coord/hybrid/hybrid_cursor_mappings.c

Typo

---------

Co-authored-by: Itzikvaknin <[email protected]>

* MOD-11685: FT.HYBRID Fix Command Line Lifetime (#6977)

* WIP: load key

* initial commit

* hybrid request - clone command line for lifetime correctness

* small fix

* additional fixes

* remove logs

* code review comments

* sync the rpnet lookup with that of the merger

* revert some changes

* code review comments

* code review comment

* MOD-11685: Add a test for _ft.hybrid with the LOAD clause (#6976)

* remove load clause handling from MR command

* revert rlookup related code

* additional revert of code

* code review  comment

* small revert

* code review comment

---------

Co-authored-by: nafraf <[email protected]>
Co-authored-by: Itzikvaknin <[email protected]>

* MOD-11243: Change SORTBY 0 to NOSORT  (#7009)

* Change SORTBY 0 -> NOSORT in FT.HYBRID

* New line EOF

* Fix comments

* Remove redundent callback

* Add testSortbyNotEnoughArguments

* Newline EOF

* CR comment

* Fix hybrid timeout (#7028)

This PR includes several improvements and fixes:
- Proper initialization of the time field in the search context for hybrid request flows.
- Explicit setup of command properties in HybridRequest_prepareForExecution.
- Updated iterStartCb private data to use a ref-counted data instead of a raw void*.
- Fixed several memory leaks.
Co-authored-by: Itzikvaknin <[email protected]>

* Change TestRealTimeouts queries to heavier ones (#7038)

Change TestRealTimeouts_queries to heavier ones

* MOD-11004: init hybrid areq time (#7051)

* Add more timeout tests

* Init time in HybridRequest_StartCursors()

* UpdateTime in HybridRequest_BuildPipeline()

* Clean up

* Move setuptime to hybridCommandHandler

* Remove unneeded SetupTime

* Clean up test_hybrid_internal.py

* Update RPDepleterTest

* Merge master into feature-coordinator-hybrid (#7068)

* MOD-11701: FT.HYBRID - Synchronize RLookup Moving Parts (#7006)

* WIP: load key

* initial commit

* additional fixes

* remove logs

* code review comments

* sync the rpnet lookup with that of the merger

* revert some changes

* code review comments

* code review comment

* remove load clause handling from MR command

* initial commit

* revert OpenLoadKeys function

* bug fixes

* fix typo

* move back to Write in AddKeys, open docKey as read before initializing the hybrid lookup context

* move sorting changes to only be in coordinator after distribute phase

* remove MakeDefaultHybridUpstreams

* cursor code review comments

* code review comments

* rename variable

* fix compilation

* modify aggpln serializatrion flow

* fix buildMRCommand flow regarding adding __key to the LOAD clause

* additional fixes

* simplify the approach a bit - translate the result row as soon as we receive it - this allows us to use our own doc key to access the document key
no need for a key array

* rever some code to original state

* code review fix

* fix typos

* code review comment

* fix alias for search rlookup key not being opened in PLN_Distribute

* fix cpp test compilation

* small fix

* fix

* fix build mr tests

* fix test leak

* only free the array

* fix aggregate flow

* fix row leak

* fix to fill upstream info function

* handle cases where LOAD is already specified - since previous assert kept firiing

* simulate previous behaviour to get tests to pass

* fix

* place the new load step at the end of the step arguments

* fix typo

* revert some of the code to a simpler way of serializing the additional steps

* looks like LOAD can be specified multiple times

* fix crash

* remove free

* change score key opening to write

* fix compilation

* fix cpptests - fallback to opening key as read if it already exists

* open score as read for now to ensure consistent behaviour - less optimized

* open the score key outside the unresolved ok scope

* further refactoring of lookup flow

* code review fixes

* Fix depleter push to the coordinator upstream pipelines

* Small fix to pushRP

* Fix pointer address
Fix typo

---------

Co-authored-by: nafraf <[email protected]>
Co-authored-by: [email protected] <[email protected]>

* [MOD-11664] Handle _INDEX_PREFIXES in hybrid distributed queries (#7021)

* feat: handle _INDEX_PREFIXES in hybrid distributed queries

* Apply suggestions from code review

* test: add test for the feature

* fix: free allocated memory

* test: fix unit tests

* do not add empty _INDEX_PREFIXES to internal command if not needed

* fix: do not directly cast args, use ArgCursor to traverse args cleanly

* fix potential problem when no spec

* initialize prefixes offsets for AREQ

* fix _INDEX_PREFIXES handling and parsing

* Fix memory issues

* Prevent double-free

* Fix mem leak

* free prefixes in success path

* Revert "free prefixes in success path"

This reverts commit 2181baa.

* fix: fix FGC tests

* fix: empty prefix should be valid during the complete function scope

* Code review fixes

* fix by review comments

* handle comments on review

* use std vector as prefixes when working in tests

* test: add happy path test

* test: fix test

---------

Co-authored-by: [email protected] <[email protected]>
Co-authored-by: nafraf <[email protected]>
Co-authored-by: nafraf <[email protected]>

---------

Co-authored-by: nafraf <[email protected]>
Co-authored-by: kei-nan <[email protected]>
Co-authored-by: [email protected] <[email protected]>
Co-authored-by: Itzikvaknin <[email protected]>
Co-authored-by: nafraf <[email protected]>
Co-authored-by: Joan Fontanals <[email protected]>

… (#7080)

* Move load step handling in parseHybridCommand to a function handleLoadStepForHybridPipelines

Support multiple load clauses

* Add test for two load steps in FT.HYBRID

* feat: add default scorer in configuration

* fix and test config setters and getters

* test: add flow tests checking scorer behavior

* fix: fix as per cursor comments

* fix: fix as per code review

* fix: change the init config code

* test: fix the tests

* test: refactor tests

* fix memory leak in getter

* test: add explicit test BM25STD is the default since start

* validate default scorer at module init

* fix validation against extensions

* apply changes as per review

* Update tests/cpptests/test_cpp_forkgc.cpp

* refactor small setter

* try to fix sanitizer leak

* test: Known leaks when onModuleLoad fails

* Update src/module-init/module-init.c

Co-authored-by: kei-nan <[email protected]>

---------

Co-authored-by: kei-nan <[email protected]>

* Add simple test

* Add test with APPLY, GROUPBY, YIELD_SCORE_AS

---------

Co-authored-by: nafraf <[email protected]>

* remove incorrect assertion

* sort shards in new topology RedisCluster_GetTopology, to have a stable order when the topology don't actually changes

* Repair an index block with deleted records

* Repair an empty index block

* Repair block when nothing was deleted

* Repair block with some deletions

* Remove num deletes

* Simpler block creation

* Have repair split big deltas correctly

* Remove unwraps

* Add GC scan to inverted index

* Add comments

* Test for scan

* Add GC apply for ii

* Add last block details to scan results

* Skip the last block if new entries were added

* Report on apply results

* Derive Copy for unit structs

* Use `smallvec` instead

* Better name

Co-authored-by: Luca Palmieri <[email protected]>

* Use more of a zip / union for apply

* Use `impl Fn` instead

This will allow the use of closures at the FFI layer.

* Add GC to wrappers and track unique docs correctly

* Add repair callback

* Clippy suggestions

* Increment GC marker on apply

* Improve comment

Co-authored-by: Luca Palmieri <[email protected]>

* Enable `union` feature for `smallvec`

* Make it easier to see encoded block entries

* Remove unneeded 0 entries check

A block can never have zero entries outside of garbage collection so
this check will never be hit.

* Don't test empty blocks

Nothing will result in empty block anyway.

* Improve variable name

Co-authored-by: meiravgri <[email protected]>

* Check for last block only once

* Ignore apply if deltas is empty

* Use `u8` for delta

* Document number in test

* Make GC info fields public

This will allow other tests (iterators) to access this information too.

* Clippy suggestion

* Add benchmark for GC patterns

* Use variable to track changes

* Faster unwrap

This appears to be slightly faster than a pure `expect()`.

* Run bench quicker

This is done by setting custom measurement and warmup times.

* Add an integration test

This does a full cycle add_record, read, gc scan, and gc apply
operations and tests the expected results.

* Fix spelling

---------

Co-authored-by: Luca Palmieri <[email protected]>
Co-authored-by: meiravgri <[email protected]>

* Fix ACL test to validate index creation

* Address review

* Remove redundant test

* copy test from razmon-fix_ACL_test_feature

---------

Co-authored-by: nafraf <[email protected]>