Repository navigation
[Bug] replace_if_not_null not found in vectorized compaction #9374
Description
Search before asking
- I had searched in the issues and found no similar issues.
Version
master branch
What's Wrong?
Some thing wrong for REPLACE NOT NULL aggregation type column in vectorized compaction now.
`
1# 0x00007FE7DDEC6400 in /lib64/libc.so.6
2# doris::vectorized::BlockReader::_init_agg_state(doris::TabletReader::ReaderParams const&) at /root/incubator-doris/be/src/vec/olap/block_reader.cpp:107
3# doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) at /root/incubator-doris/be/src/vec/olap/block_reader.cpp:162
4# doris::Merger::vmerge_rowsets(std::shared_ptrdoris::Tablet, doris::ReaderType, std::vector<std::shared_ptrdoris::RowsetReader, std::allocator<std::shared_ptrdoris::RowsetReader > > const&, doris::RowsetWriter*, doris::Merger::Statistics*) at /root/incubator-doris/be/src/olap/merger.cpp:106
5# doris::Compaction::do_compaction_impl(long) at /root/incubator-doris/be/src/olap/compaction.cpp:95
6# doris::Compaction::do_compaction(long) at /root/incubator-doris/be/src/olap/compaction.cpp:57
7# doris::CumulativeCompaction::execute_compact_impl() at /root/incubator-doris/be/src/olap/cumulative_compaction.cpp:75
8# doris::Compaction::execute_compact() at /root/incubator-doris/be/src/olap/compaction.cpp:46
9# doris::Tablet::execute_compaction(doris::CompactionType) at /root/incubator-doris/be/src/olap/tablet.cpp:1407
10# std::_Function_handler<void (), doris::StorageEngine::_submit_compaction_task(std::shared_ptrdoris::Tablet, doris::CompactionType)::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:291
11# doris::ThreadPool::dispatch_thread() at /root/incubator-doris/be/src/util/threadpool.cpp:548
12# doris::Thread::supervise_thread(void*) at /root/incubator-doris/be/src/util/thread.cpp:409
13# start_thread in /lib64/libpthread.so.0
14# clone in /lib64/libc.so.6
`
What You Expected?
fix it
How to Reproduce?
No response
Anything Else?
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
xinyiZzz commentedon May 5, 2022
I also had this problem the other day, but it disappeared after rebase master...
Gabriel39 commentedon May 5, 2022
@xinyiZzz
I reproduce this bug using up-to-date master branch. In fact, if you have a nullable column with
REPLACE NOT NULLagg type and turnenable_vectorized_compactionon and then you can also reproduce it again.In my case, I use a column ``last_update_date
DATETIME REPLACE_IF_NOT_NULL DEFAULT "1970-01-01 00:00:00"to reproduce this bugBiteTheDDDDt commentedon May 5, 2022
Do you have a sql that can reproduce this error?
Gabriel39 commentedon May 5, 2022
@BiteTheDDDDt
I create table using statement:
CREATE TABLE IF NOT EXISTS compaction_tbl
(
user_idLARGEINT NOT NULL COMMENT "用户id",dateDATE NOT NULL COMMENT "数据灌入日期时间",cityVARCHAR(20) COMMENT "用户所在城市",ageSMALLINT COMMENT "用户年龄",sexTINYINT COMMENT "用户性别",last_visit_dateDATETIME REPLACE DEFAULT "1970-01-01 00:00:00" COMMENT "用户最后一次访问时间",last_update_dateDATETIME REPLACE_IF_NOT_NULL DEFAULT "1970-01-01 00:00:00" COMMENT "用户最后一次更新时间",last_visit_date_not_nullDATETIME REPLACE NOT NULL DEFAULT "1970-01-01 00:00:00" COMMENT "用户最后一次访问时间",costBIGINT SUM DEFAULT "0" COMMENT "用户总消费",max_dwell_timeINT MAX DEFAULT "0" COMMENT "用户最大停留时间",min_dwell_timeINT MIN DEFAULT "99999" COMMENT "用户最小停留时间",hll_colHLL HLL_UNION NOT NULL COMMENT "HLL列",bitmap_colBitmap BITMAP_UNION NOT NULL COMMENT "bitmap列")
AGGREGATE KEY(
user_id,date,city,age,sex)DISTRIBUTED BY HASH(user_id) PROPERTIES("replication_num" = "1");
Insert some test data and after a while, background compaction task will core.
xinyiZzz commentedon May 5, 2022
The reason why the bug disappeared after I rebase master was because config::enable_vectorized_compaction was added, and vmerge_rowsets was not used...
My reproduce case is similar.