Skip to content

Gorilla compression fails on merge #6398

@chbuescher

Description

@chbuescher

Describe the bug
We test clickhouse with carbon-clickhouse as replacement for the graphite database.
When we tried to use Gorilla compression for the metric values, but after a while the clickhouse server error log fills with error messages.

Version:
version 19.11.6.31 (official build)

Error message:
2019.08.08 11:17:30.152682 [ 42 ] {909d41bd-3856-4de7-aae8-47cb91d0e56f} executeQuery: Code: 71, e.displayText() = DB::Exception: Cannot write after end of buffer. (version 19.11.6.31 (official build)) (from 127.0.0.1:51176) (in query: INSERT INTO graphite.data (Path, Value, Time, Date, Timestamp) FORMAT RowBinary ), Stack trace:

  1. clickhouse-server(StackTrace::StackTrace()+0x30) [0x7d258d0]
  2. clickhouse-server(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, int)+0x25) [0x3bb8205]
  3. clickhouse-server(DB::WriteBuffer::nextImpl()+0x55) [0x38dc68d]
  4. clickhouse-server() [0x708609f]
  5. clickhouse-server(DB::CompressionCodecGorilla::doCompressData(char const*, unsigned int, char*) const+0x122) [0x7086202]
  6. clickhouse-server(DB::ICompressionCodec::compress(char const*, unsigned int, char*) const+0x33) [0x6ac6083]
  7. clickhouse-server(DB::CompressionCodecMultiple::doCompressData(char const*, unsigned int, char*) const+0x202) [0x7088e02]
  8. clickhouse-server(DB::ICompressionCodec::compress(char const*, unsigned int, char*) const+0x33) [0x6ac6083]
  9. clickhouse-server(DB::CompressedWriteBuffer::nextImpl()+0xae) [0x6ac0dfe]
  10. clickhouse-server(DB::IMergedBlockOutputStream::ColumnStream::finalize()+0x68) [0x72fbf88]
  11. clickhouse-server(DB::MergedBlockOutputStream::writeSuffixAndFinalizePart(std::shared_ptrDB::MergeTreeDataPart&, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*)+0x8c9) [0x6f66399]
  12. clickhouse-server(DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&)+0x1447) [0x6f348d7]
  13. clickhouse-server(DB::MergeTreeBlockOutputStream::write(DB::Block const&)+0xe4) [0x6ed41b4]
  14. clickhouse-server(DB::PushingToViewsBlockOutputStream::write(DB::Block const&)+0x51) [0x710d551]
  15. clickhouse-server(DB::SquashingBlockOutputStream::finalize()+0x143) [0x7115d03]
  16. clickhouse-server(DB::SquashingBlockOutputStream::writeSuffix()+0x11) [0x7115f31]
  17. clickhouse-server(DB::copyData(DB::IBlockInputStream&, DB::IBlockOutputStream&, std::atomic*)+0x707) [0x6aea047]
  18. clickhouse-server(DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, DB::Context&, std::function<void (std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)>, std::function<void (std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)>)+0x1da) [0x6d39a7a]
  19. clickhouse-server(DB::HTTPHandler::processQuery(Poco::Net::HTTPServerRequest&, HTMLForm&, Poco::Net::HTTPServerResponse&, DB::HTTPHandler::Output&)+0x197e) [0x3bcd00e]
  20. clickhouse-server(DB::HTTPHandler::handleRequest(Poco::Net::HTTPServerRequest&, Poco::Net::HTTPServerResponse&)+0x52b) [0x3bcf42b]
  21. clickhouse-server(Poco::Net::HTTPServerConnection::run()+0x2a9) [0x7705979]
  22. clickhouse-server(Poco::Net::TCPServerConnection::start()+0x10) [0x7700750]
  23. clickhouse-server(Poco::Net::TCPServerDispatcher::run()+0xed) [0x7700e6d]
  24. clickhouse-server(Poco::PooledThread::run()+0x81) [0x7e24621]
  25. clickhouse-server(Poco::ThreadImpl::runnableEntry(void*)+0x3c) [0x7e223cc]
  26. clickhouse-server() [0xb7f7b40]
  27. /lib64/libpthread.so.0() [0x306fe07aa1]
  28. /lib64/libc.so.6(clone+0x6d) [0x306fae8bdd]

On another server we also get ABORTs:
2019.08.08 11:10:06.829438 [ 4 ] {} graphite.data: Removing temporary directory /var/lib/clickhouse/data/graphite/data/tmp_insert_201908_2736_2736_0
2019.08.08 11:10:30.247196 [ 43 ] {} BaseDaemon: (version 19.11.6.31 (official build)) (from thread 41) Terminate called for uncaught exception:
Code: 71, e.displayText() = DB::Exception: Cannot write after end of buffer., Stack trace:

  1. clickhouse-server(StackTrace::StackTrace()+0x30) [0x7d258d0]
  2. clickhouse-server(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, int)+0x25) [0x3bb8205]
  3. clickhouse-server(DB::WriteBuffer::nextImpl()+0x55) [0x38dc68d]
  4. clickhouse-server(DB::BitWriter::doFlush()+0x165) [0x707eab5]
  5. clickhouse-server() [0x39a5d4f]
  6. clickhouse-server(DB::CompressionCodecGorilla::doCompressData(char const*, unsigned int, char*) const+0x122) [0x7086202]
  7. clickhouse-server(DB::ICompressionCodec::compress(char const*, unsigned int, char*) const+0x33) [0x6ac6083]
  8. clickhouse-server(DB::CompressionCodecMultiple::doCompressData(char const*, unsigned int, char*) const+0x202) [0x7088e02]
  9. clickhouse-server(DB::ICompressionCodec::compress(char const*, unsigned int, char*) const+0x33) [0x6ac6083]
  10. clickhouse-se
    2019.08.08 11:10:30.249130 [ 43 ] {} BaseDaemon: ########################################
    2019.08.08 11:10:30.249177 [ 43 ] {} BaseDaemon: (version 19.11.6.31 (official build)) (from thread 41) Received signal Aborted (6).
    2019.08.08 11:10:30.249194 [ 43 ] {} BaseDaemon:
    2019.08.08 11:10:30.362874 [ 43 ] {} BaseDaemon: 0. clickhouse-server(StackTrace::StackTrace(ucontext_t const&)+0x31) [0x7d25911]
  11. clickhouse-server() [0x3cd5a5e]
  12. /lib64/libpthread.so.0() [0x389ca0f7e0]
  13. /lib64/libc.so.6(gsignal+0x35) [0x389c232495]
  14. /lib64/libc.so.6(abort+0x175) [0x389c233c75]
  15. clickhouse-server() [0x376a23c]
  16. clickhouse-server(__cxxabiv1::__terminate(void (*)())+0x6) [0xb773436]
  17. clickhouse-server() [0xb80f759]
  18. clickhouse-server(__gxx_personality_v0+0x294) [0xb772e14]
  19. clickhouse-server() [0xb81e169]
  20. clickhouse-server(_Unwind_Resume+0xa7) [0xb81e3c7]
  21. clickhouse-server() [0x707eb67]
  22. clickhouse-server() [0x39a5d4f]
  23. clickhouse-server(DB::CompressionCodecGorilla::doCompressData(char const*, unsigned int, char*) const+0x122) [0x7086202]
  24. clickhouse-server(DB::ICompressionCodec::compress(char const*, unsigned int, char*) const+0x33) [0x6ac6083]
  25. clickhouse-server(DB::CompressionCodecMultiple::doCompressData(char const*, unsigned int, char*) const+0x202) [0x7088e02]
  26. clickhouse-server(DB::ICompressionCodec::compress(char const*, unsigned int, char*) const+0x33) [0x6ac6083]
  27. clickhouse-server(DB::CompressedWriteBuffer::nextImpl()+0xae) [0x6ac0dfe]
  28. clickhouse-server(DB::IMergedBlockOutputStream::ColumnStream::finalize()+0x68) [0x72fbf88]
  29. clickhouse-server(DB::MergedBlockOutputStream::writeSuffixAndFinalizePart(std::shared_ptrDB::MergeTreeDataPart&, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*)+0x8c9) [0x6f66399]
  30. clickhouse-server(DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&)+0x1447) [0x6f348d7]
  31. clickhouse-server(DB::MergeTreeBlockOutputStream::write(DB::Block const&)+0xe4) [0x6ed41b4]
  32. clickhouse-server(DB::PushingToViewsBlockOutputStream::write(DB::Block const&)+0x51) [0x710d551]
  33. clickhouse-server(DB::SquashingBlockOutputStream::finalize()+0x143) [0x7115d03]
  34. clickhouse-server(DB::SquashingBlockOutputStream::writeSuffix()+0x11) [0x7115f31]
  35. clickhouse-server(DB::copyData(DB::IBlockInputStream&, DB::IBlockOutputStream&, std::atomic*)+0x707) [0x6aea047]
  36. clickhouse-server(DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, DB::Context&, std::function<void (std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)>, std::function<void (std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)>)+0x1da) [0x6d39a7a]
  37. clickhouse-server(DB::HTTPHandler::processQuery(Poco::Net::HTTPServerRequest&, HTMLForm&, Poco::Net::HTTPServerResponse&, DB::HTTPHandler::Output&)+0x197e) [0x3bcd00e]
  38. clickhouse-server(DB::HTTPHandler::handleRequest(Poco::Net::HTTPServerRequest&, Poco::Net::HTTPServerResponse&)+0x52b) [0x3bcf42b]
  39. clickhouse-server(Poco::Net::HTTPServerConnection::run()+0x2a9) [0x7705979]
  40. clickhouse-server(Poco::Net::TCPServerConnection::start()+0x10) [0x7700750]
  41. clickhouse-server(Poco::Net::TCPServerDispatcher::run()+0xed) [0x7700e6d]

Table definition for graphite.data
ATTACH TABLE data
(
Path String CODEC(ZSTD(1)),
Value Float64 CODEC(Gorilla, LZ4),
Time UInt32 CODEC(DoubleDelta, LZ4),
Date Date,
Timestamp UInt32 CODEC(Delta(4), ZSTD(1))
)
ENGINE = GraphiteMergeTree('graphite_rollup')
PARTITION BY toYYYYMM(Date)
ORDER BY (Path, Time)
SETTINGS index_granularity = 8192

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions