Skip to content

Reduce merge queue master#7568

Merged
alonre24 merged 25 commits intomasterfrom
reduce_merge_queue_master
Dec 3, 2025
Merged

Reduce merge queue master#7568
alonre24 merged 25 commits intomasterfrom
reduce_merge_queue_master

Conversation

@alonre24
Copy link
Collaborator

@alonre24 alonre24 commented Nov 28, 2025

Describe the changes in the pull request

As an attempt to strike a better balance between continuous testing and velocity, we reduce the platforms that we are testing on upon merging every PR to master, leaving only ubuntu-latest and macos for both x86 and arm under the test matrix. Other jobs remain as is.

Also - removing macOS on Intel, as it is obsolete and will not be supported soon, and having all test flows running full test suites on the merge queue (removing ones that were running QUICK mode).
And - freeing disk space in container jobs (as we are already exceeding the limits)

Mark if applicable

  • This PR introduces API changes
  • This PR introduces serialization changes

Note

Reduce merge-queue platforms, remove macOS x86_64, run full (non-QUICK) tests, add aggressive disk cleanup, and support comma-separated platform filters.

  • CI workflows:
    • event-merge-to-queue.yml: Limit test matrix to ubuntu:noble, macos, alpine:3; switch job-test-config from QUICK=1 to full tests; minor TODO update.
    • flow-intel.yml: Run full tests (test-config: '' instead of QUICK=1).
    • flow-temp.yml: Replace Intel job with merge-queue workflow invocation.
  • Matrix generation (generate-matrix.yml):
    • Support comma-separated platform filter; update filtering logic accordingly.
    • Remove macos x86_64 from available combinations (keep macos aarch64).
  • Configuration (task-get-config.yml):
    • Drop macOS x86_64 config; keep macos aarch64.
  • Test flow (task-test.yml):
    • Enhance container spec with explicit image and mount options.
    • Add disk space cleanup for container and non-container jobs; add disk usage checks after build and in log summary.
    • Minor naming tweaks (e.g., job names).

Written by Cursor Bugbot for commit 200bb11. This will update automatically on new commits. Configure here.

@codecov
Copy link

codecov bot commented Nov 28, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.71%. Comparing base (81173b7) to head (200bb11).
⚠️ Report is 5 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #7568      +/-   ##
==========================================
- Coverage   84.71%   84.71%   -0.01%     
==========================================
  Files         351      351              
  Lines       54198    54197       -1     
  Branches    14506    14506              
==========================================
- Hits        45916    45912       -4     
- Misses       8089     8092       +3     
  Partials      193      193              
Flag Coverage Δ
flow 85.17% <ø> (-0.03%) ⬇️
unit 52.02% <ø> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@alonre24 alonre24 requested review from GuyAv46 and removed request for GuyAv46 December 1, 2025 07:23
Copy link
Collaborator

@GuyAv46 GuyAv46 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@alonre24 alonre24 requested a review from GuyAv46 December 1, 2025 13:19
alonre24 and others added 3 commits December 3, 2025 11:41
* fix: Avoid Rust cache contamination across platforms. (#7569)

* [MOD-12170] Implement ASM State Machine on notifications (#7331)

* first commit

* fix: handle lock in two phases

* inform ownership in regular command process handlers

* fix: handle proper init

* fix: use conditional variable

* fix: add some comments

* test: add test proving deadlock

* fix: changes as per comments

* fix: remove this locking from conn

* remove code where certainity is not so high

* fix potential issue

* test: add first unit tests version

* test: add proper testing with the value proposition

* test: improve testing

* alternative using auxiliary lock

* simplify thpool a bit

* make sure RedisModule_Yield is protected

* fix: fix issue raised by cursor

* fix: handle potential deadlock in drain also

* fix: fix potential TOCTOU concurrency bug

* fix: fix order of release

* fix: fix potential overflow issue

* fix tests

* fix: fix counts in all variables

* small refactor

* some refactoring of Shared Exclusive Lock

* simplify

* protect GILOwned simple bool

* clarify some comments

* set GILAlternativeLockHeld to true properly

* fix spelling

* add assertion

* fix: fix import RS_LOG_ASSERT

* test: add more conditions to testing

* add some more logic

* improve testing to proper signal main thread can finish while other threads may be waiting for the Shared Lock in the loop

* test: handl test properly

* add another pattern of tests

* fix comments from cursor

* fix concurrency bug

* fix: fix potential race condition at release lock time

* fix: add condition

* test: add more testing

* force testing further to capture more potential errors

* parametrize tests

* test: make tests a little faster

* test: add micro benchmark

* fix compile microbenchmarks

* fix: avoid potential reentrant deadlocks

* fix: avoid potential reentrant deadlocks

* test: avoid leak in test

* test: avoid leak in test

* fix: fix assertion

* fix: fix assertion

* Simplify Shared Lock internals (#7267)

* simplify shared lock

* small improvement to set_timeout

* fix comment

* fix nd improve comments

* condition fix

* remove lock type from release API

* Add lock type back to the release API

* remove Unlocked from enum and handle clock init for macOS

* adapt to use new API

* chnange according to comments

* handle PR comments

* handle PR comments

* fix: take shared lock in other cases

* change as PR comments

* test: simplify test, do not allow query errors

* fix tests as per comments

* fix: handle number of high priority jobs running

* fix: fix test comment

* add ASM to help slot tracking in notifications

* test: add some testing idea

* change as per PR comments

* compile and link test fix

* change draining method to drain high priority

* add ctests for ASM State Machine

* test: add ASM tests

* test: complete tests

* change as per PR comments

* add micro benchmarks with jobs in threads

* remove changes not wanted

* checkout redis feature branch in task test

* remove draining

* move atomic to new header

* remove _internal naming

* fix formatting

---------

Co-authored-by: GuyAv46 <[email protected]>

* [MOD-12627] Add Debug Support for `FT.PROFILE` Command (#7510)

* imp debug profile for SA:

introduce in mocule.h: RSProfileCommandImp
RSProfileCommand calls RSProfileCommandImp(isDebug = false) for regular execution
ProfileCommandCommand_DebugWrapper mcalls it with isDebug=true and skips _FT.DEBUG

introduce entrypoint for _FT.DEBUG FT.PROFILE in debug_commands:
ProfileCommandCommand_DebugWrapper

RSProfileCommandImp calls DEBUG_execCommandCommon is its debug

_recursiveProfilePrint skips printing debug RP

* pass is debug instead of extracting:

module.h:
replace declaration: DistAggregateCommand
DistSearchCommand
with Imp version that receives isDebug

expose ProfileCommandHandlerImp

align debug_commands

introducr _FT.DEBUG _FT.PROFILE

* add test for cluster

* return res

* augi fixes

* fix spell check

* fix for real

* fix test

* skip tests according to env

* revrt test_profile changes

* reove changes from internal_only

* [MOD-12694] [MOD-12069] Add active_coord_threads metric (#7546)

* Add multi-threading statistics tracking for active I/O threads

* fix comment

* add cpp test

* fix spelling

* address comment

* remove unnecessary nre line

* add "active_worker_threads" metric

* fix comment
imp tests

* test cleanups

* add test comne about the num queries and cleanups

* fix declartion

* remove coord threads

* add active_coord_threads

expose ConcurrentSearchPool_WorkingThreadCount

* make the tests run...

* add "active_worker_threads" metric

* fix declartion

* remove coord threads

* make the tests run...

* introduce workersThreadPool_isInitialized
assert is initizlied in GlobalStats_GetMultiThreadingStats

* cleanup

* rename workersThreadPool_isCreated

* introduce ConcurrentSearchPool_IsCreated

* fix test

* we dont need workers

* remove ConcurrentSearchPool_IsCreated and workersThreadPool_isInitialized

* fix merge

* [MOD-12789] test: fix flaky thpool test (#7581)

test: fix flaky thpool test

* Support Multiple Slot Ranges in search.CLUSTERSET - [MOD-11657] (#7508)

* support multiple slot ranges

* implement and move around helpers

* add tests

* ignore shards with no slots

* improve parsing

* better error message

* fix flow tests

* fix tests

* sort by node id

* improve error testing

* cover missing cases

* more error messages improvements

* address AI review

* stabilize Unexpected argument error path

* stabilize more error paths

* fix tests accordingly

* last fix

* add logs to cluster set command

* add more logs per @alonre24 request

* rename MetricIterator to Metric (#7586)

The other iterators are not suffixed with 'Iterator'.

* [MOD-12701] Split the execution of Rust and C/C++ unit tests across two different CI steps. (#7587)

Split the execution of Rust and C/C++ unit tests across two different CI steps.

* [MOD-12519] implement skip multi in II iterators (#7426)

* remove it->skipMulti

No longer needed as the full iterators have been removed.
The query ones always set it to true.

* implement skip multi

II query iterators are supposed to skip results having the same ids.

Test ported from test_cpp_iterator_index.cpp

* inline read() and skip_to()

* test more ii iterator edge cases

Port of GetCorrectValue and EOFAfterFiltering from test_cpp_iterator_index.cpp

* Compress layers prior to exporting them to the Docker layer cache (#7529)

* Compress Docker layers prior to exporting them.

* Ignore boost subfolders recursively

* [MOD-12417] Track maxprefixexpansions errors and warnings in info (#7570)

track maxprefixexpanions

* [MOD-12409]: Port DocumentType enum to Rust (#7590)

Port DocumentType enum to Rust

Port DocumentType enum to Rust as `document::DocumentType`, removing it from `redisearch.h`.
Use `document::DocumentType` in `rlookup` instead of `rlookup::bindings::DocumentType`.

* [MOD-12069] Add `*_pending_jobs` metrics (#7556)

* align info/* to active_coord

* add APIs to get queues length

* add to info

* fix

* test

* fix test

* catch general error

* rename

* fix moduleArgs

* rename

* rename test_active_worker_threads

* rename to wworketrs

* [MOD-12392] Remove numDocs parameter from non-optimized Wildcard iterator (#7602)

Remove numDocs parameter from non-optimized Wildcard iterator

* [MOD-12701] Enforce a per-test timeout in the C++ rstest suite using ctest (#7588)

* Enforce a per-test timeout in the C++ rstest suite using ctest

* Disable problematic tests

* Raise timeout to 60s

* Raise timeout

* Add a timeout for coordinator tests too

* Skip ActivateIoThreadsMetric test

* Register Cursor Sub-Commands as such - [MOD-12807, MOD-12808] (#7571)

* split cursor command

* fix and improve tests

* cover error cases

* fix cursor leaks

* Add "TODO: run hybrid cursor" back

* remove new empty line

* small test improvement

* fix FT.CURSOR GC

* de-flake test

* make CURSOR PROFILE internal only

* test the free

* Keep just the prints

* test on macos and noble as well

* Move the free

* Remove

* Add to non container

* Back to regular test

* Remove spaces

* Moved the remove and add the repo size

* Moved it again

* Move it to after repo build

* Move print after repo build

* test all

* run the temp flow

* add mount to container

* Fix curly

---------

Co-authored-by: Luca Palmieri <[email protected]>
Co-authored-by: Joan Fontanals <[email protected]>
Co-authored-by: GuyAv46 <[email protected]>
Co-authored-by: meiravgri <[email protected]>
Co-authored-by: Guillaume Desmottes <[email protected]>
Co-authored-by: lerman25 <[email protected]>
Co-authored-by: Henk Oordt <[email protected]>
Co-authored-by: alonre24 <[email protected]>
Copy link

@jit-ci jit-ci bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❌ Jit has detected 1 important finding in this PR that you should review.


Jit encountered an internal error and cannot comment on each finding.

You can ask a Jit admin to comment #jit_ignore_all on this PR to ignore the findings.

Here are the findings in this PR:

  • Security Control: Static Code Analysis Semgrep Pro
    • Type: Yaml.Github-Actions.Security.Run-Shell-Injection.Run-Shell-Injection
    • Description: Using variable interpolation ${{...}} with github context data in a run: step could allow an attacker to inject their own code into the runner. This would allow them to steal secrets and code. github context data can have arbitrary user input and should be treated as untrusted. Instead, use an intermediate environment variable with env: to store the data and use the environment variable in the run: script. Be sure to use double-quotes the environment variable, like this: "$ENVVAR".
    • Severity: HIGH
    • Learn More: Link
    • Filename: .github/workflows/generate-matrix.yml
    • Lines: 43-114

@alonre24
Copy link
Collaborator Author

alonre24 commented Dec 3, 2025

dor-forer
dor-forer previously approved these changes Dec 3, 2025
# Documentation: https://redislabs.atlassian.net/wiki/spaces/DX/pages/3967844669/RediSearch+CI+refactor

on:
workflow_call:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove leftovers

@alonre24 alonre24 requested a review from GuyAv46 December 3, 2025 16:30
@alonre24 alonre24 enabled auto-merge December 3, 2025 16:32
GuyAv46
GuyAv46 previously approved these changes Dec 3, 2025
@alonre24 alonre24 added this pull request to the merge queue Dec 3, 2025
Merged via the queue into master with commit f217727 Dec 3, 2025
25 checks passed
@alonre24 alonre24 deleted the reduce_merge_queue_master branch December 3, 2025 19:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants