Skip to content

Implement least_used load balancing algorithm for disks inside volume#36686

Merged
serxa merged 9 commits intoClickHouse:masterfrom
azat:disk-least-used
May 20, 2022
Merged

Implement least_used load balancing algorithm for disks inside volume#36686
serxa merged 9 commits intoClickHouse:masterfrom
azat:disk-least-used

Conversation

@azat
Copy link
Copy Markdown
Member

@azat azat commented Apr 26, 2022 •
edited by alexey-milovidov
Loading

Changelog category (leave one):

  • Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Implement least_used load balancing algorithm for disks inside volume (multi disk configuration)

@robot-ch-test-poll robot-ch-test-poll added the pr-improvement Pull request with some product improvements label Apr 26, 2022
@azat azat force-pushed the disk-least-used branch from 8d8da9b to 9df119c Compare April 26, 2022 20:22
@alexey-milovidov alexey-milovidov changed the title Implement lead_used load balancing algorithm for disks inside volume Implement least_used load balancing algorithm for disks inside volume Apr 27, 2022
@azat azat marked this pull request as draft April 27, 2022 07:27
@azat azat force-pushed the disk-least-used branch from 9df119c to 042cf0c Compare April 27, 2022 11:03
@azat
Copy link
Copy Markdown
Member Author

azat commented Apr 27, 2022

Stateless tests (debug, actions) [2/3] — fail: 1, passed: 1338, skipped: 18

@azat
Copy link
Copy Markdown
Member Author

azat commented Apr 27, 2022

Stress test (debug, actions) — Fatal message in clickhouse-server.log (see fatal_messages.txt)

Details 
[ 6355 ] {} <Fatal> BaseDaemon: ########################################
[ 6355 ] {} <Fatal> BaseDaemon: (version 22.5.1.1, build id: 9E1B85662356D46A) (from thread 1770) (query_id: 28c829fa-5b6a-4e93-99cf-7c46b28ab3d4) (query: SELECT sum(AdvEngineID), count(), avg(ResolutionWidth) FROM test.hits_s3 ;) Received signal Aborted (6)
[ 6355 ] {} <Fatal> BaseDaemon: 
[ 6355 ] {} <Fatal> BaseDaemon: Stack trace: 0x7fb65f38703b 0x7fb65f366859 0x7fb65f366729 0x7fb65f378006 0x25891121 0x258913ca 0x26240927 0x262408f5 0x262408d5 0x262408b5 0x26240895 0x26240875 0x26240855 0x26240221 0x26221907 0x26221895 0x26221879 0x2622166e 0x17b24431 0x17b243d9 0x2588424c 0x26203131 0x262033ea 0x2620334b 0x262038ca 0x17b24431 0x17b243d9 0x2625462c 0x262543dd 0x26254499 0x261b988c 0x261b97fc 0x261b9799 0x261b9335 0x261b93ae 0x261b93d9 0x17db95ec 0x17db955c 0x17db9019 0x17db8e63 0x25eabe59
[ 6355 ] {} <Fatal> BaseDaemon: 4. gsignal @ 0x7fb65f38703b in ?
[ 6355 ] {} <Fatal> BaseDaemon: 5. abort @ 0x7fb65f366859 in ?
[ 6355 ] {} <Fatal> BaseDaemon: 6. ? @ 0x7fb65f366729 in ?
[ 6355 ] {} <Fatal> BaseDaemon: 7. ? @ 0x7fb65f378006 in ?
[ 6355 ] {} <Fatal> BaseDaemon: 8. /build/build_docker/../src/Common/FileSegment.cpp:655: DB::FileSegment::assertDetachedStatus() const @ 0x25891121 in /usr/bin/clickhouse
[ 6355 ] {} <Fatal> BaseDaemon: 9. /build/build_docker/../src/Common/FileSegment.cpp:694: DB::FileSegmentsHolder::~FileSegmentsHolder() @ 0x258913ca in /usr/bin/clickhouse
[ 6355 ] {} <Fatal> BaseDaemon: 10. /build/build_docker/../contrib/libcxx/include/optional:239: std::__1::__optional_destruct_base<DB::FileSegmentsHolder, false>::~__optional_destruct_base() @ 0x26240927 in /usr/bin/clickhouse
[ 6355 ] {} <Fatal> BaseDaemon: 11. /build/build_docker/../contrib/libcxx/include/optional:312: std::__1::__optional_storage_base<DB::FileSegmentsHolder, false>::~__optional_storage_base() @ 0x262408f5 in /usr/bin/clickhouse
[ 6355 ] {} <Fatal> BaseDaemon: 12. /build/build_docker/../contrib/libcxx/include/optional:485: std::__1::__optional_copy_base<DB::FileSegmentsHolder, false>::~__optional_copy_base() @ 0x262408d5 in /usr/bin/clickhouse
[ 6355 ] {} <Fatal> BaseDaemon: 13. /build/build_docker/../contrib/libcxx/include/optional:513: std::__1::__optional_move_base<DB::FileSegmentsHolder, false>::~__optional_move_base() @ 0x262408b5 in /usr/bin/clickhouse
[ 6355 ] {} <Fatal> BaseDaemon: 14. /build/build_docker/../contrib/libcxx/include/optional:546: std::__1::__optional_copy_assign_base<DB::FileSegmentsHolder, false>::~__optional_copy_assign_base() @ 0x26240895 in /usr/bin/clickhouse
[ 6355 ] {} <Fatal> BaseDaemon: 15. /build/build_docker/../contrib/libcxx/include/optional:578: std::__1::__optional_move_assign_base<DB::FileSegmentsHolder, false>::~__optional_move_assign_base() @ 0x26240875 in /usr/bin/clickhouse
[ 6355 ] {} <Fatal> BaseDaemon: 16. /build/build_docker/../contrib/libcxx/include/optional:621: std::__1::optional<DB::FileSegmentsHolder>::~optional() @ 0x26240855 in /usr/bin/clickhouse
[ 6355 ] {} <Fatal> BaseDaemon: 17. /build/build_docker/../src/Disks/IO/CachedReadBufferFromRemoteFS.h:12: DB::CachedReadBufferFromRemoteFS::~CachedReadBufferFromRemoteFS() @ 0x26240221 in /usr/bin/clickhouse
[ 6355 ] {} <Fatal> BaseDaemon: 18. /build/build_docker/../contrib/libcxx/include/__memory/construct_at.h:57: void std::__1::__destroy_at<DB::CachedReadBufferFromRemoteFS, 0>(DB::CachedReadBufferFromRemoteFS*) @ 0x26221907 in /usr/bin/clickhouse
[ 6355 ] {} <Fatal> BaseDaemon: 19. /build/build_docker/../contrib/libcxx/include/__memory/construct_at.h:82: void std::__1::destroy_at<DB::CachedReadBufferFromRemoteFS, 0>(DB::CachedReadBufferFromRemoteFS*) @ 0x26221895 in /usr/bin/clickhouse
[ 6355 ] {} <Fatal> BaseDaemon: 20. /build/build_docker/../contrib/libcxx/include/__memory/allocator_traits.h:321: void std::__1::allocator_traits<std::__1::allocator<DB::CachedReadBufferFromRemoteFS> >::destroy<DB::CachedReadBufferFromRemoteFS, void, void>(std::__1::allocator<DB::CachedReadBufferFromRemoteFS>&, DB::CachedReadBufferFromRemoteFS*) @ 0x26221879 in /usr/bin/clickhouse
[ 6355 ] {} <Fatal> BaseDaemon: 21. /build/build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:310: std::__1::__shared_ptr_emplace<DB::CachedReadBufferFromRemoteFS, std::__1::allocator<DB::CachedReadBufferFromRemoteFS> >::__on_zero_shared() @ 0x2622166e in /usr/bin/clickhouse
[ 6355 ] {} <Fatal> BaseDaemon: 22. /build/build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:175: std::__1::__shared_count::__release_shared() @ 0x17b24431 in /usr/bin/clickhouse
[ 6355 ] {} <Fatal> BaseDaemon: 23. /build/build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:216: std::__1::__shared_weak_count::__release_shared() @ 0x17b243d9 in /usr/bin/clickhouse
[ 6355 ] {} <Fatal> BaseDaemon: 24. /build/build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:704: std::__1::shared_ptr<DB::SeekableReadBuffer>::~shared_ptr() @ 0x2588424c in /usr/bin/clickhouse
[ 6355 ] {} <Fatal> BaseDaemon: 25. /build/build_docker/../src/Disks/IO/ReadBufferFromRemoteFSGather.h:23: DB::ReadBufferFromRemoteFSGather::~ReadBufferFromRemoteFSGather() @ 0x26203131 in /usr/bin/clickhouse

@kssenii you may be interesting, and it does not looks like it will be fixed by #36685 ?

@azat azat force-pushed the disk-least-used branch from 042cf0c to 3771e7f Compare April 27, 2022 14:47
@azat
Copy link
Copy Markdown
Member Author

azat commented Apr 27, 2022

https://s3.amazonaws.com/clickhouse-test-reports/36686/042cf0c76444e8738eb2481ae21a135f05b4c990/integration_tests__thread__actions__[2/4].html

Incorrect report, test_jbod_ha/test.py::test_jbod_ha failed only in the first run.

@azat azat marked this pull request as ready for review April 27, 2022 14:50
@kssenii
Copy link
Copy Markdown
Member

kssenii commented Apr 27, 2022

Stress test (debug, actions) — Fatal message in clickhouse-server.log (see fatal_messages.txt)
@kssenii you may be interesting, and it does not looks like it will be fixed by #36685 ?

By this one: #36660

@azat azat mentioned this pull request May 1, 2022
Merged
@serxa serxa self-assigned this May 3, 2022
@azat azat force-pushed the disk-least-used branch from 3771e7f to 276c9d7 Compare May 3, 2022 09:22
@azat
Copy link
Copy Markdown
Member Author

azat commented May 3, 2022 •
edited
Loading

Stress test (address, actions) — Backward compatibility check: Error message in clickhouse-server.log (see bc_check_error_messages.txt)

Stress test (memory, actions) — Hung check failed

2022.05.03 14:42:52.991418 [ 1095 ] {} <Error> static void DB::WriteBufferFromS3::finalizeCacheIfNeeded(std::optional<FileSegmentsHolder> &): Code: 49. DB::Exception: Operation not allowed, file segment is detached. (LOGICAL_ERROR), Stack trace (when copying this message, always include the lines below):

@azat
Copy link
Copy Markdown
Member Author

azat commented May 12, 2022

@serxa Hi. Did you managed to find time to take a look?

@serxa
Copy link
Copy Markdown
Member

serxa commented May 13, 2022

Hi! A was going to do that on Monday.

serxa
serxa reviewed May 15, 2022
View reviewed changes
@azat azat force-pushed the disk-least-used branch from 276c9d7 to d63c701 Compare May 18, 2022 14:10
serxa
serxa requested changes May 19, 2022
View reviewed changes
@azat azat force-pushed the disk-least-used branch from d63c701 to 15936dc Compare May 19, 2022 10:54
azat added 9 commits May 19, 2022 14:10
@azat
Add storage_configuration example into config.xml
456444e 
Signed-off-by: Azat Khuzhin <[email protected]>
@azat
doc: document perform_ttl_move_on_insert
590dfe6 
Signed-off-by: Azat Khuzhin <[email protected]>
@azat
doc/mergetree: fix aligment in documentation
dad31fe 
Signed-off-by: Azat Khuzhin <[email protected]>
@azat
Disks: Remove mentions about RAID1
9d734bd 
Signed-off-by: Azat Khuzhin <[email protected]>
@azat
Disks: Remove unused src/Disks/SingleDiskVolume.cpp
4bc849b 
Signed-off-by: Azat Khuzhin <[email protected]>
@azat
Implement lead_used load balancing algorithm for disks inside volume
ba26b3c 
v2: rebase on top removed raid1
Signed-off-by: Azat Khuzhin <[email protected]>
@azat
Disks: Reduce number of statfs() calls for least_used disk load balan…
1d98913 
…cing policy

Signed-off-by: Azat Khuzhin <[email protected]>
@azat
Disks: Introduce IReservation::getUnreservedSpace()
757894b 
Signed-off-by: Azat Khuzhin <[email protected]>
@azat
Disks: More accurate free space calculation for least_used
a350991 
Signed-off-by: Azat Khuzhin <[email protected]>
@azat azat force-pushed the disk-least-used branch from 15936dc to a350991 Compare May 19, 2022 11:10
serxa
serxa approved these changes May 19, 2022
View reviewed changes
Copy link
Copy Markdown
Member

@serxa serxa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@serxa
Copy link
Copy Markdown
Member

serxa commented May 20, 2022

failed tests look unrelated, so merge

@serxa serxa merged commit 2450ab9 into ClickHouse:master May 20, 2022
@azat azat deleted the disk-least-used branch May 20, 2022 07:45
@den-crane den-crane mentioned this pull request Jun 25, 2022
Open
@azat azat mentioned this pull request Oct 25, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@serxa serxa serxa approved these changes

Assignees

@serxa serxa

Labels

pr-improvement Pull request with some product improvements

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@azat @kssenii @serxa @robot-ch-test-poll