Too many open files issue #25994
Description
Describe the unexpected behaviour
For unknown reasons i've started receiving errors.
Setup is a 2 node cluster with replication.
The open file limit is set to 1048576
At the moment i'm only getting this on 1 of the 2 nodes which makes this very confusing.
I get the errors pretty quickly after a restart.
Most information I've been able to find is to increase the openfile limit but since it's already set much higher than is advices i'm a bit lost on how to solve this.
How to reproduce
Clickhouse 21.3.13.9
Error message and/or stacktrace
2021.07.05 12:22:03.990986 [ 143 ] {} <Error> auto DB::IBackgroundJobExecutor::jobExecutingTask()::(anonymous class)::operator()() const: Code: 76, e.displayText() = DB::ErrnoException: Cannot open file /var/lib/clickhouse/store/5e0/5e034cc5-a5a2-4684-9052-f4494e183702/tmp_merge_202107_158521_161678_140/CurrentMetric_DiskSpaceReservedForMerge.mrk2, errno: 23, strerror: Too many open files in system, Stack trace (when copying this message, always include the lines below):
--
Mon, Jul 5 2021 1:22:03 pm |
Mon, Jul 5 2021 1:22:03 pm | 0. DB::ErrnoException::ErrnoException(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, int, std::__1::optional<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > const&) @ 0x84fb42b in /usr/bin/clickhouse
Mon, Jul 5 2021 1:22:03 pm | 1. DB::throwFromErrnoWithPath(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, int) @ 0x84fb7a4 in /usr/bin/clickhouse
Mon, Jul 5 2021 1:22:03 pm | 2. DB::WriteBufferFromFile::WriteBufferFromFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned long, int, unsigned int, char*, unsigned long) @ 0x85f019e in /usr/bin/clickhouse
Mon, Jul 5 2021 1:22:03 pm | 3. DB::DiskLocal::writeFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned long, DB::WriteMode) @ 0xe7afd75 in /usr/bin/clickhouse
Mon, Jul 5 2021 1:22:03 pm | 4. DB::MergeTreeDataPartWriterOnDisk::Stream::Stream(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::IDisk>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::ICompressionCodec> const&, unsigned long) @ 0xf438cfb in /usr/bin/clickhouse
Mon, Jul 5 2021 1:22:03 pm | 5. ? @ 0xf44152b in /usr/bin/clickhouse
Mon, Jul 5 2021 1:22:03 pm | 6. DB::MergeTreeDataPartWriterWide::addStreams(DB::NameAndTypePair const&, std::__1::shared_ptr<DB::IAST> const&) @ 0xf43c08d in /usr/bin/clickhouse
Mon, Jul 5 2021 1:22:03 pm | 7. DB::MergeTreeDataPartWriterWide::MergeTreeDataPartWriterWide(std::__1::shared_ptr<DB::IMergeTreeDataPart const> const&, DB::NamesAndTypesList const&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::__1::vector<std::__1::shared_ptr<DB::IMergeTreeIndex const>, std::__1::allocator<std::__1::shared_ptr<DB::IMergeTreeIndex const> > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::ICompressionCodec> const&, DB::MergeTreeWriterSettings const&, DB::MergeTreeIndexGranularity const&) @ 0xf43bec3 in /usr/bin/clickhouse
Mon, Jul 5 2021 1:22:03 pm | 8. DB::MergeTreeDataPartWide::getWriter(DB::NamesAndTypesList const&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::__1::vector<std::__1::shared_ptr<DB::IMergeTreeIndex const>, std::__1::allocator<std::__1::shared_ptr<DB::IMergeTreeIndex const> > > const&, std::__1::shared_ptr<DB::ICompressionCodec> const&, DB::MergeTreeWriterSettings const&, DB::MergeTreeIndexGranularity const&) const @ 0xf42dc20 in /usr/bin/clickhouse
Mon, Jul 5 2021 1:22:03 pm | 9. DB::MergedBlockOutputStream::MergedBlockOutputStream(std::__1::shared_ptr<DB::IMergeTreeDataPart const> const&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::NamesAndTypesList const&, std::__1::vector<std::__1::shared_ptr<DB::IMergeTreeIndex const>, std::__1::allocator<std::__1::shared_ptr<DB::IMergeTreeIndex const> > > const&, std::__1::shared_ptr<DB::ICompressionCodec>, bool) @ 0xf503bea in /usr/bin/clickhouse
Mon, Jul 5 2021 1:22:03 pm | 10. DB::MergeTreeDataMergerMutator::mergePartsToTemporaryPart(DB::FutureMergedMutatedPart const&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::BackgroundProcessListEntry<DB::MergeListElement, DB::MergeInfo>&, std::__1::shared_ptr<DB::RWLockImpl::LockHolderImpl>&, long, DB::Context const&, std::__1::unique_ptr<DB::IReservation, std::__1::default_delete<DB::IReservation> > const&, bool, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0xf3ef9a0 in /usr/bin/clickhouse
Mon, Jul 5 2021 1:22:03 pm | 11. DB::StorageMergeTree::mergeSelectedParts(std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, bool, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, DB::StorageMergeTree::MergeMutateSelectedEntry&, std::__1::shared_ptr<DB::RWLockImpl::LockHolderImpl>&) @ 0xf168fa8 in /usr/bin/clickhouse
Mon, Jul 5 2021 1:22:03 pm | 12. ? @ 0xf17b49c in /usr/bin/clickhouse
Mon, Jul 5 2021 1:22:03 pm | 13. ? @ 0xf3443d7 in /usr/bin/clickhouse
Mon, Jul 5 2021 1:22:03 pm | 14. ThreadPoolImpl<ThreadFromGlobalPool>::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) @ 0x853cf08 in /usr/bin/clickhouse
Mon, Jul 5 2021 1:22:03 pm | 15. ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...)::'lambda'()::operator()() @ 0x853e91f in /usr/bin/clickhouse
Mon, Jul 5 2021 1:22:03 pm | 16. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0x853a55f in /usr/bin/clickhouse
Mon, Jul 5 2021 1:22:03 pm | 17. ? @ 0x853d9d3 in /usr/bin/clickhouse
Mon, Jul 5 2021 1:22:03 pm | 18. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
Mon, Jul 5 2021 1:22:03 pm | 19. __clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
Mon, Jul 5 2021 1:22:03 pm | (version 21.3.13.9 (official build))