Skip to content

Build without readies [MOD-5722]#6017

Merged
alonre24 merged 70 commits intomasterfrom
build_without_readies
May 4, 2025
Merged

Build without readies [MOD-5722]#6017
alonre24 merged 70 commits intomasterfrom
build_without_readies

Conversation

@alonre24
Copy link
Collaborator

@alonre24 alonre24 commented Apr 25, 2025

Describe the changes in the pull request

(PR started by @DvirDukhan, continue by @alonre24). Original PR was merged and reverted due to packing failures #6003)
The goal is to purge https://github.com/RedisLabsModules/readies from our build system.
In this PR we introduce a new script .build.sh which is an alternative entry point for building the module using cmake without going through the makefile.
This includes refactoring of our CMakeLists.txt so it will account for the rules and flags that are passed via readies.

Now we can to the following operations without going through readies:

  • build the module and the unit tests by calling ./build.sh [TESTS]
  • run flow tests by calling ./build.sh RUN_PYTEST
  • build and run the sanitizer with flow tests by calling ./build.sh [RUN_PYTEST]

This is still an intermediate step on the way to purge readies, and building the module in the "old way" using makefile rules is still possible (and will probably be available in the future as well, but internally will call to ./build.sh - TBD).

As part of that, I also:

  1. Removed redundant files related to build, including the docker template (we are not using it anymore).
  2. Removed the option to build redisearch as a static library and redisearch lite since these are no longer in use (light as of redis 8)

** Benchmarks flow does not show any regression **

Left to do (in upcoming PRs):

  • Build and run coverage without readies - currently this still works via make coverage call only.
  • Refactor unit tests flow that is still using readies internally (will also require some refactoring regardless) - currently requires calling make unit-tests
  • Apply the linter that brings an informative message upon setting up the module about missing dependencies.
  • Refactor pack and upload-artifacts flows

Mark if applicable

  • This PR introduces API changes
  • This PR introduces serialization changes

DvirDukhan
DvirDukhan previously approved these changes Apr 27, 2025
@alonre24 alonre24 added this pull request to the merge queue Apr 27, 2025
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Apr 27, 2025
@github-actions
Copy link

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

@alonre24
Copy link
Collaborator Author

@alonre24 alonre24 requested review from DvirDukhan and GuyAv46 and removed request for GuyAv46 April 29, 2025 19:31
@alonre24 alonre24 added this pull request to the merge queue May 4, 2025
Merged via the queue into master with commit bd56920 May 4, 2025
15 of 16 checks passed
@alonre24 alonre24 deleted the build_without_readies branch May 4, 2025 08:47
@redisearch-backport-pull-request
Copy link
Contributor

Backport failed for 2.8, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin 2.8
git worktree add -d .worktree/backport-6017-to-2.8 origin/2.8
cd .worktree/backport-6017-to-2.8
git switch --create backport-6017-to-2.8
git cherry-pick -x bd569209494b0afa76256e9211f20ebf1ce353d7

@redisearch-backport-pull-request
Copy link
Contributor

Backport failed for 2.6, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin 2.6
git worktree add -d .worktree/backport-6017-to-2.6 origin/2.6
cd .worktree/backport-6017-to-2.6
git switch --create backport-6017-to-2.6
git cherry-pick -x bd569209494b0afa76256e9211f20ebf1ce353d7

@redisearch-backport-pull-request
Copy link
Contributor

Backport failed for 2.10, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin 2.10
git worktree add -d .worktree/backport-6017-to-2.10 origin/2.10
cd .worktree/backport-6017-to-2.10
git switch --create backport-6017-to-2.10
git cherry-pick -x bd569209494b0afa76256e9211f20ebf1ce353d7

@redisearch-backport-pull-request
Copy link
Contributor

Backport failed for 8.0, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin 8.0
git worktree add -d .worktree/backport-6017-to-8.0 origin/8.0
cd .worktree/backport-6017-to-8.0
git switch --create backport-6017-to-8.0
git cherry-pick -x bd569209494b0afa76256e9211f20ebf1ce353d7

nafraf pushed a commit that referenced this pull request May 13, 2025
* purge readeis from cmake. Use new build script instead of makefile

* call script in CI

* libuv

* restore build folder

* libuv

* unit tests

* pretty build.sh

* pytest

* fixed extension build error

* Add profile flags + try pass linker flag properly (wip)

* fix build and linkage + pytest + adjust json

* small cleanup

* Add pytz dep

* bump googletest version, use same variant name as before

* remove ld libs

* Fix executable linker flags - build unit test properly.
run unit tests with old "make unit-tests" way (until this is fixed via ./build.sh)

* fix rust build via build.sh

* fix san build

* Set coverage flags and use the same bin dir for sanitizer (build it for debug)

* update benchmark image for regression test

* remove leftover

* Build hiredis static

* fix json env flag in CI

* CR changes

* Restore deleted files until unit tests are done as well, use boolean params

* restore vecsim version

* fix profile and fix build error

* fix for profile

* try fix mac build

* fix unit tests for arm

* Define boost dir

* remove the policies

* try fix the binroot for unit-tests

* change dir name for arm

* remove policy from hash as well

* use clang in mac

* try to set hardcoded clang

* try to mimic readies in macos includes

* try set CMAKE_OSX_DEPLOYMENT_TARGET

* whitespace formatting

* Set CC to clang in apple

* try to export llvm

* try set compiler in build script

* use bin in path

* try EXPORT properly with homebrew

* remove setting bad path to clang

* set clang path

* update to llvm@18

* update c++ path

* set compiler with LLVM env var

* fix?

* set proper link flags for mac

* remove bsymbolic from hiredis

* revert hiredis in non macos to how it was

* clean stuff

* cleanups

* Fix the extract debug symbols command so it will work as before (required for packing properly)

* update deps

* fix for alpine

---------

Co-authored-by: DvirDukhan <[email protected]>
Co-authored-by: GuyAv46 <[email protected]>
@alonre24 alonre24 changed the title Build without readies [MOD-5711] Build without readies [MOD-5722] May 18, 2025
JoanFM pushed a commit that referenced this pull request May 27, 2025
* purge readeis from cmake. Use new build script instead of makefile

* call script in CI

* libuv

* restore build folder

* libuv

* unit tests

* pretty build.sh

* pytest

* fixed extension build error

* Add profile flags + try pass linker flag properly (wip)

* fix build and linkage + pytest + adjust json

* small cleanup

* Add pytz dep

* bump googletest version, use same variant name as before

* remove ld libs

* Fix executable linker flags - build unit test properly.
run unit tests with old "make unit-tests" way (until this is fixed via ./build.sh)

* fix rust build via build.sh

* fix san build

* Set coverage flags and use the same bin dir for sanitizer (build it for debug)

* update benchmark image for regression test

* remove leftover

* Build hiredis static

* fix json env flag in CI

* CR changes

* Restore deleted files until unit tests are done as well, use boolean params

* restore vecsim version

* fix profile and fix build error

* fix for profile

* try fix mac build

* fix unit tests for arm

* Define boost dir

* remove the policies

* try fix the binroot for unit-tests

* change dir name for arm

* remove policy from hash as well

* use clang in mac

* try to set hardcoded clang

* try to mimic readies in macos includes

* try set CMAKE_OSX_DEPLOYMENT_TARGET

* whitespace formatting

* Set CC to clang in apple

* try to export llvm

* try set compiler in build script

* use bin in path

* try EXPORT properly with homebrew

* remove setting bad path to clang

* set clang path

* update to llvm@18

* update c++ path

* set compiler with LLVM env var

* fix?

* set proper link flags for mac

* remove bsymbolic from hiredis

* revert hiredis in non macos to how it was

* clean stuff

* cleanups

* Fix the extract debug symbols command so it will work as before (required for packing properly)

* update deps

* fix for alpine

---------

Co-authored-by: DvirDukhan <[email protected]>
Co-authored-by: GuyAv46 <[email protected]>
JoanFM pushed a commit that referenced this pull request May 27, 2025
* purge readeis from cmake. Use new build script instead of makefile

* call script in CI

* libuv

* restore build folder

* libuv

* unit tests

* pretty build.sh

* pytest

* fixed extension build error

* Add profile flags + try pass linker flag properly (wip)

* fix build and linkage + pytest + adjust json

* small cleanup

* Add pytz dep

* bump googletest version, use same variant name as before

* remove ld libs

* Fix executable linker flags - build unit test properly.
run unit tests with old "make unit-tests" way (until this is fixed via ./build.sh)

* fix rust build via build.sh

* fix san build

* Set coverage flags and use the same bin dir for sanitizer (build it for debug)

* update benchmark image for regression test

* remove leftover

* Build hiredis static

* fix json env flag in CI

* CR changes

* Restore deleted files until unit tests are done as well, use boolean params

* restore vecsim version

* fix profile and fix build error

* fix for profile

* try fix mac build

* fix unit tests for arm

* Define boost dir

* remove the policies

* try fix the binroot for unit-tests

* change dir name for arm

* remove policy from hash as well

* use clang in mac

* try to set hardcoded clang

* try to mimic readies in macos includes

* try set CMAKE_OSX_DEPLOYMENT_TARGET

* whitespace formatting

* Set CC to clang in apple

* try to export llvm

* try set compiler in build script

* use bin in path

* try EXPORT properly with homebrew

* remove setting bad path to clang

* set clang path

* update to llvm@18

* update c++ path

* set compiler with LLVM env var

* fix?

* set proper link flags for mac

* remove bsymbolic from hiredis

* revert hiredis in non macos to how it was

* clean stuff

* cleanups

* Fix the extract debug symbols command so it will work as before (required for packing properly)

* update deps

* fix for alpine

---------

Co-authored-by: DvirDukhan <[email protected]>
Co-authored-by: GuyAv46 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants