-
Notifications
You must be signed in to change notification settings - Fork 8.3k
Gorilla compression fails on merge #6398
Description
Describe the bug
We test clickhouse with carbon-clickhouse as replacement for the graphite database.
When we tried to use Gorilla compression for the metric values, but after a while the clickhouse server error log fills with error messages.
Version:
version 19.11.6.31 (official build)
Error message:
2019.08.08 11:17:30.152682 [ 42 ] {909d41bd-3856-4de7-aae8-47cb91d0e56f} executeQuery: Code: 71, e.displayText() = DB::Exception: Cannot write after end of buffer. (version 19.11.6.31 (official build)) (from 127.0.0.1:51176) (in query: INSERT INTO graphite.data (Path, Value, Time, Date, Timestamp) FORMAT RowBinary ), Stack trace:
- clickhouse-server(StackTrace::StackTrace()+0x30) [0x7d258d0]
- clickhouse-server(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, int)+0x25) [0x3bb8205]
- clickhouse-server(DB::WriteBuffer::nextImpl()+0x55) [0x38dc68d]
- clickhouse-server() [0x708609f]
- clickhouse-server(DB::CompressionCodecGorilla::doCompressData(char const*, unsigned int, char*) const+0x122) [0x7086202]
- clickhouse-server(DB::ICompressionCodec::compress(char const*, unsigned int, char*) const+0x33) [0x6ac6083]
- clickhouse-server(DB::CompressionCodecMultiple::doCompressData(char const*, unsigned int, char*) const+0x202) [0x7088e02]
- clickhouse-server(DB::ICompressionCodec::compress(char const*, unsigned int, char*) const+0x33) [0x6ac6083]
- clickhouse-server(DB::CompressedWriteBuffer::nextImpl()+0xae) [0x6ac0dfe]
- clickhouse-server(DB::IMergedBlockOutputStream::ColumnStream::finalize()+0x68) [0x72fbf88]
- clickhouse-server(DB::MergedBlockOutputStream::writeSuffixAndFinalizePart(std::shared_ptrDB::MergeTreeDataPart&, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*)+0x8c9) [0x6f66399]
- clickhouse-server(DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&)+0x1447) [0x6f348d7]
- clickhouse-server(DB::MergeTreeBlockOutputStream::write(DB::Block const&)+0xe4) [0x6ed41b4]
- clickhouse-server(DB::PushingToViewsBlockOutputStream::write(DB::Block const&)+0x51) [0x710d551]
- clickhouse-server(DB::SquashingBlockOutputStream::finalize()+0x143) [0x7115d03]
- clickhouse-server(DB::SquashingBlockOutputStream::writeSuffix()+0x11) [0x7115f31]
- clickhouse-server(DB::copyData(DB::IBlockInputStream&, DB::IBlockOutputStream&, std::atomic*)+0x707) [0x6aea047]
- clickhouse-server(DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, DB::Context&, std::function<void (std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)>, std::function<void (std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)>)+0x1da) [0x6d39a7a]
- clickhouse-server(DB::HTTPHandler::processQuery(Poco::Net::HTTPServerRequest&, HTMLForm&, Poco::Net::HTTPServerResponse&, DB::HTTPHandler::Output&)+0x197e) [0x3bcd00e]
- clickhouse-server(DB::HTTPHandler::handleRequest(Poco::Net::HTTPServerRequest&, Poco::Net::HTTPServerResponse&)+0x52b) [0x3bcf42b]
- clickhouse-server(Poco::Net::HTTPServerConnection::run()+0x2a9) [0x7705979]
- clickhouse-server(Poco::Net::TCPServerConnection::start()+0x10) [0x7700750]
- clickhouse-server(Poco::Net::TCPServerDispatcher::run()+0xed) [0x7700e6d]
- clickhouse-server(Poco::PooledThread::run()+0x81) [0x7e24621]
- clickhouse-server(Poco::ThreadImpl::runnableEntry(void*)+0x3c) [0x7e223cc]
- clickhouse-server() [0xb7f7b40]
- /lib64/libpthread.so.0() [0x306fe07aa1]
- /lib64/libc.so.6(clone+0x6d) [0x306fae8bdd]
On another server we also get ABORTs:
2019.08.08 11:10:06.829438 [ 4 ] {} graphite.data: Removing temporary directory /var/lib/clickhouse/data/graphite/data/tmp_insert_201908_2736_2736_0
2019.08.08 11:10:30.247196 [ 43 ] {} BaseDaemon: (version 19.11.6.31 (official build)) (from thread 41) Terminate called for uncaught exception:
Code: 71, e.displayText() = DB::Exception: Cannot write after end of buffer., Stack trace:
- clickhouse-server(StackTrace::StackTrace()+0x30) [0x7d258d0]
- clickhouse-server(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, int)+0x25) [0x3bb8205]
- clickhouse-server(DB::WriteBuffer::nextImpl()+0x55) [0x38dc68d]
- clickhouse-server(DB::BitWriter::doFlush()+0x165) [0x707eab5]
- clickhouse-server() [0x39a5d4f]
- clickhouse-server(DB::CompressionCodecGorilla::doCompressData(char const*, unsigned int, char*) const+0x122) [0x7086202]
- clickhouse-server(DB::ICompressionCodec::compress(char const*, unsigned int, char*) const+0x33) [0x6ac6083]
- clickhouse-server(DB::CompressionCodecMultiple::doCompressData(char const*, unsigned int, char*) const+0x202) [0x7088e02]
- clickhouse-server(DB::ICompressionCodec::compress(char const*, unsigned int, char*) const+0x33) [0x6ac6083]
- clickhouse-se
2019.08.08 11:10:30.249130 [ 43 ] {} BaseDaemon: ########################################
2019.08.08 11:10:30.249177 [ 43 ] {} BaseDaemon: (version 19.11.6.31 (official build)) (from thread 41) Received signal Aborted (6).
2019.08.08 11:10:30.249194 [ 43 ] {} BaseDaemon:
2019.08.08 11:10:30.362874 [ 43 ] {} BaseDaemon: 0. clickhouse-server(StackTrace::StackTrace(ucontext_t const&)+0x31) [0x7d25911] - clickhouse-server() [0x3cd5a5e]
- /lib64/libpthread.so.0() [0x389ca0f7e0]
- /lib64/libc.so.6(gsignal+0x35) [0x389c232495]
- /lib64/libc.so.6(abort+0x175) [0x389c233c75]
- clickhouse-server() [0x376a23c]
- clickhouse-server(__cxxabiv1::__terminate(void (*)())+0x6) [0xb773436]
- clickhouse-server() [0xb80f759]
- clickhouse-server(__gxx_personality_v0+0x294) [0xb772e14]
- clickhouse-server() [0xb81e169]
- clickhouse-server(_Unwind_Resume+0xa7) [0xb81e3c7]
- clickhouse-server() [0x707eb67]
- clickhouse-server() [0x39a5d4f]
- clickhouse-server(DB::CompressionCodecGorilla::doCompressData(char const*, unsigned int, char*) const+0x122) [0x7086202]
- clickhouse-server(DB::ICompressionCodec::compress(char const*, unsigned int, char*) const+0x33) [0x6ac6083]
- clickhouse-server(DB::CompressionCodecMultiple::doCompressData(char const*, unsigned int, char*) const+0x202) [0x7088e02]
- clickhouse-server(DB::ICompressionCodec::compress(char const*, unsigned int, char*) const+0x33) [0x6ac6083]
- clickhouse-server(DB::CompressedWriteBuffer::nextImpl()+0xae) [0x6ac0dfe]
- clickhouse-server(DB::IMergedBlockOutputStream::ColumnStream::finalize()+0x68) [0x72fbf88]
- clickhouse-server(DB::MergedBlockOutputStream::writeSuffixAndFinalizePart(std::shared_ptrDB::MergeTreeDataPart&, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*)+0x8c9) [0x6f66399]
- clickhouse-server(DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&)+0x1447) [0x6f348d7]
- clickhouse-server(DB::MergeTreeBlockOutputStream::write(DB::Block const&)+0xe4) [0x6ed41b4]
- clickhouse-server(DB::PushingToViewsBlockOutputStream::write(DB::Block const&)+0x51) [0x710d551]
- clickhouse-server(DB::SquashingBlockOutputStream::finalize()+0x143) [0x7115d03]
- clickhouse-server(DB::SquashingBlockOutputStream::writeSuffix()+0x11) [0x7115f31]
- clickhouse-server(DB::copyData(DB::IBlockInputStream&, DB::IBlockOutputStream&, std::atomic*)+0x707) [0x6aea047]
- clickhouse-server(DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, DB::Context&, std::function<void (std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)>, std::function<void (std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)>)+0x1da) [0x6d39a7a]
- clickhouse-server(DB::HTTPHandler::processQuery(Poco::Net::HTTPServerRequest&, HTMLForm&, Poco::Net::HTTPServerResponse&, DB::HTTPHandler::Output&)+0x197e) [0x3bcd00e]
- clickhouse-server(DB::HTTPHandler::handleRequest(Poco::Net::HTTPServerRequest&, Poco::Net::HTTPServerResponse&)+0x52b) [0x3bcf42b]
- clickhouse-server(Poco::Net::HTTPServerConnection::run()+0x2a9) [0x7705979]
- clickhouse-server(Poco::Net::TCPServerConnection::start()+0x10) [0x7700750]
- clickhouse-server(Poco::Net::TCPServerDispatcher::run()+0xed) [0x7700e6d]
Table definition for graphite.data
ATTACH TABLE data
(
Path String CODEC(ZSTD(1)),
Value Float64 CODEC(Gorilla, LZ4),
Time UInt32 CODEC(DoubleDelta, LZ4),
Date Date,
Timestamp UInt32 CODEC(Delta(4), ZSTD(1))
)
ENGINE = GraphiteMergeTree('graphite_rollup')
PARTITION BY toYYYYMM(Date)
ORDER BY (Path, Time)
SETTINGS index_granularity = 8192