Skip to content

Fix restore S3 disk without detached directories#33730

Merged
Avogar merged 1 commit intoClickHouse:masterfrom
ianton-ru:MDB-16115-2
Jan 19, 2022
Merged

Fix restore S3 disk without detached directories#33730
Avogar merged 1 commit intoClickHouse:masterfrom
ianton-ru:MDB-16115-2

Conversation

@ianton-ru
Copy link
Copy Markdown
Contributor

@ianton-ru ianton-ru commented Jan 18, 2022

Changelog category (leave one):

  • Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Create parent directories in DiskS3::restoreFileOperations method.

Detailed description / Documentation draft:
During restart S3 disk procedure ClckHouse reads operations from log on S3 and reproduses stored operations on files. In this log only file operations, so directories like 'detached' can be missed after that procedure when no one operation uses this directory. In this case trying to restore local metafiles fails with error:

2022.01.10 11:55:06.078012 [ 255 ] {24ad8cc5-1ed4-49af-8ee4-9ad72b65d34d} <Error> executeQuery: std::exception. Code: 1001, type: std::__1::__fs::filesystem::filesystem_error, e.what() = filesystem error: in rename: No such file or directory [/var/lib/clickhouse/disks/s3/store/ff5/ff55d5a3-d684-423b-bf55-d5a3d684223b/0_1_1_0/] [/var/lib/clickhouse/disks/s3/store/ff5/ff55d5a3-d684-423b-bf55-d5a3d684223b/detached/0_1_1_0]
Cannot print extra info for Poco::Exception (version 21.12.3.32 (official build)) (from 10.10.249.8:48008) (in query: SYSTEM RESTART DISK s3), Stack trace (when copying this message, always include the lines below):

0. std::runtime_error::runtime_error(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0x1924062d in ?
1. std::__1::system_error::system_error(std::__1::error_code, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0x1924a297 in ?
2. ? @ 0x191ef602 in ?
3. ? @ 0x191eeedd in ?
4. ? @ 0x191f22f8 in ?
5. std::__1::__fs::filesystem::__rename(std::__1::__fs::filesystem::path const&, std::__1::__fs::filesystem::path const&, std::__1::error_code*) @ 0x191f89b5 in ?
6. DB::DiskLocal::moveDirectory(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0x12bb780f in /usr/bin/clickhouse
7. DB::DiskS3::restoreFileOperations(DB::DiskS3::RestoreInformation const&) @ 0x12bd67d9 in /usr/bin/clickhouse
8. DB::DiskS3::restore() @ 0x12bc8ecf in /usr/bin/clickhouse
9. DB::DiskS3::startup() @ 0x12bc7635 in /usr/bin/clickhouse
10. DB::DiskRestartProxy::restart() @ 0x12c0271c in /usr/bin/clickhouse
11. DB::InterpreterSystemQuery::restartDisk(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&) @ 0x132e9b91 in /usr/bin/clickhouse
12. DB::InterpreterSystemQuery::execute() @ 0x132dfcb2 in /usr/bin/clickhouse
13. ? @ 0x135289f0 in /usr/bin/clickhouse
14. DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::__1::shared_ptr<DB::Context>, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>, std::__1::optional<DB::FormatSettings> const&) @ 0x1352be7c in /usr/bin/clickhouse
15. DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::__1::optional<DB::CurrentThread::QueryScope>&) @ 0x13d8e5da in /usr/bin/clickhouse
16. DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&) @ 0x13d92ce7 in /usr/bin/clickhouse
17. DB::HTTPServerConnection::run() @ 0x13ff5eca in /usr/bin/clickhouse
18. Poco::Net::TCPServerConnection::start() @ 0x16f3d6af in /usr/bin/clickhouse
19. Poco::Net::TCPServerDispatcher::run() @ 0x16f3fb01 in /usr/bin/clickhouse
20. Poco::PooledThread::run() @ 0x1704e889 in /usr/bin/clickhouse
21. Poco::ThreadImpl::runnableEntry(void*) @ 0x1704bf80 in /usr/bin/clickhouse
22. start_thread @ 0x76db in /lib/x86_64-linux-gnu/libpthread-2.27.so
23. clone @ 0x12171f in /lib/x86_64-linux-gnu/libc-2.27.so

@Avogar Avogar self-assigned this Jan 18, 2022
@Avogar Avogar merged commit 3501b56 into ClickHouse:master Jan 19, 2022
@Avogar Avogar added the pr-must-backport Pull request should be backported intentionally. Use this label with great care! label Jan 19, 2022
Avogar added a commit that referenced this pull request Jan 20, 2022
Backport #33730 to 22.1: Fix restore S3 disk without detached directories
Avogar added a commit that referenced this pull request Jan 20, 2022
Backport #33730 to 21.12: Fix restore S3 disk without detached directories
@Felixoid Felixoid added the pr-backports-created Backport PRs are successfully created, it won't be processed by CI script anymore label Jul 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-backports-created Backport PRs are successfully created, it won't be processed by CI script anymore pr-must-backport Pull request should be backported intentionally. Use this label with great care!

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants