-
集群架构：5个FE、8个BE，做了haproxy转发，实际上只有3个FE接受流量
-
版本：doris-3.1.4-rc02-7f5ba43de6
-
配置信息：8个BE中有3个与FE合并部署，磁盘空间较小，且配置也相对较差，实际上主要的负载都在剩余的5个BE上，频繁crash的BE也是这几个，配置都是32C，内存在128~256之间，从监控来看没有性能压力
-
问题：几天前平均一天有一个BE会崩溃，最近两天明显频繁了，仅今天0点到现在这5个BE每个都崩溃过，有的崩溃过两次。
-
堆栈信息：
节点1
[libprotobuf ERROR /home/zcp/repo_center/doris_branch-3.1/doris/thirdparty/src/protobuf-21.11/src/google/protobuf/message_lite.cc:133] Can't parse message of type "doris.TabletMetaPB" because it is missing required fields: rs_metas[0].rowset_id
thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=No more data to read.thrift error, reason=No more data to read.thrift error, reason=No more data to read.thrift error, reason=No more data to read.thrift error, reason=No more data to read.*** Query id: fadb0b56f5c34a41-8eff98e64bb7113b ***
*** is nereids: 1 ***
*** tablet id: 0 ***
*** Aborted at 1781160131 (unix time) try "date -d @1781160131" if you are using GNU date ***
*** Current BE git commitID: 7f5ba43de6 ***
*** SIGSEGV address not mapped to object (@0x38) received by PID 774182 (TID 776615 OR 0x7fdb9b1fb640) from PID 56; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:421
1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /opt/apache-doris-2.1.11-bin-x64/java17/lib/server/libjvm.so
2# JVM_handle_linux_signal in /opt/apache-doris-2.1.11-bin-x64/java17/lib/server/libjvm.so
3# 0x00007FE18D23FC30 in /lib64/libc.so.6
4# 0x000055F72ED06710 in /opt/apache-doris-2.1.11-bin-x64/be/lib/doris_be
5# bthread::TaskControl::worker_thread(void*) in /opt/apache-doris-2.1.11-bin-x64/be/lib/doris_be
6# start_thread in /lib64/libc.so.6
7# __GI___clone3 in /lib64/libc.so.6
节点2
# 第一次
thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=No more data to read.thrift error, reason=No more data to read.thrift error, reason=No more data to read.thrift error, reason=No more data to read.thrift error, reason=No more data to read.*** Query id: 0-0 ***
*** is nereids: 0 ***
*** tablet id: 0 ***
*** Aborted at 1781148958 (unix time) try "date -d @1781148958" if you are using GNU date ***
*** Current BE git commitID: 7f5ba43de6 ***
*** SIGSEGV address not mapped to object (@0x0) received by PID 706788 (TID 707748 OR 0x7fedefdf5640) from PID 0; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:421
1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /opt/apache-doris-2.1.11-bin-x64/java17/lib/server/libjvm.so
2# JVM_handle_linux_signal in /opt/apache-doris-2.1.11-bin-x64/java17/lib/server/libjvm.so
3# 0x00007FF035E3FC30 in /lib64/libc.so.6
4# doris::TimestampedVersionTracker::~TimestampedVersionTracker() at /home/zcp/repo_center/doris_release/doris/be/src/olap/version_graph.h:171
5# doris::BaseTablet::~BaseTablet() at /home/zcp/repo_center/doris_release/doris/be/src/olap/base_tablet.cpp:144
6# doris::TabletManager::start_trash_sweep() at /home/zcp/repo_center/doris_release/doris/be/src/olap/tablet_manager.cpp:1214
7# doris::StorageEngine::start_trash_sweep(double*, bool) in /opt/apache-doris-2.1.11-bin-x64/be/lib/doris_be
8# doris::StorageEngine::_garbage_sweeper_thread_callback() at /home/zcp/repo_center/doris_release/doris/be/src/olap/olap_server.cpp:400
9# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_release/doris/be/src/util/thread.cpp:499
10# start_thread in /lib64/libc.so.6
11# __GI___clone3 in /lib64/libc.so.6
# 第二次
thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=No more data to read.*** Query id: 724d1aec84377ae7-ce6eca491825f191 ***
*** is nereids: 0 ***
*** tablet id: 1780317202352 ***
*** Aborted at 1781163113 (unix time) try "date -d @1781163113" if you are using GNU date ***
*** Current BE git commitID: 7f5ba43de6 ***
*** SIGSEGV invalid permissions for mapped object (@0x7f522ad6100b) received by PID 863276 (TID 922344 OR 0x7f2df6aba640) from PID 718671883; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:421
1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /opt/apache-doris-2.1.11-bin-x64/java17/lib/server/libjvm.so
2# JVM_handle_linux_signal in /opt/apache-doris-2.1.11-bin-x64/java17/lib/server/libjvm.so
3# 0x00007F5702A3FC30 in /lib64/libc.so.6
4# doris::vectorized::JsonbSerializeUtil::jsonb_to_block(std::vector<std::shared_ptr<doris::vectorized::DataTypeSerDe>, std::allocator<std::shared_ptr<doris::vectorized::DataTypeSerDe> > > const&, char const*, unsigned long, std::unordered_map<unsigned int, unsigned int, std::hash<unsigned int>, std::equal_to<unsigned int>, std::allocator<std::pair<unsigned int const, unsigned int> > > const&, doris::vectorized::Block&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::unordered_set<int, std::hash<int>, std::equal_to<int>, std::allocator<int> > const&) at /home/zcp/repo_center/doris_release/doris/be/src/vec/jsonb/serialize.cpp:101
5# doris::vectorized::JsonbSerializeUtil::jsonb_to_block(std::vector<std::shared_ptr<doris::vectorized::DataTypeSerDe>, std::allocator<std::shared_ptr<doris::vectorized::DataTypeSerDe> > > const&, doris::vectorized::ColumnStr<unsigned int> const&, std::unordered_map<unsigned int, unsigned int, std::hash<unsigned int>, std::equal_to<unsigned int>, std::allocator<std::pair<unsigned int const, unsigned int> > > const&, doris::vectorized::Block&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::unordered_set<int, std::hash<int>, std::equal_to<int>, std::allocator<int> > const&) in /opt/apache-doris-2.1.11-bin-x64/be/lib/doris_be
6# doris::BaseTablet::fetch_value_through_row_column(std::shared_ptr<doris::Rowset>, doris::TabletSchema const&, unsigned int, std::vector<unsigned int, std::allocator<unsigned int> > const&, std::vector<unsigned int, std::allocator<unsigned int> > const&, doris::vectorized::Block&) in /opt/apache-doris-2.1.11-bin-x64/be/lib/doris_be
7# doris::FlexibleReadPlan::read_columns_by_plan(doris::TabletSchema const&, std::vector<unsigned int, std::allocator<unsigned int> > const&, std::map<doris::RowsetId, std::shared_ptr<doris::Rowset>, std::less<doris::RowsetId>, std::allocator<std::pair<doris::RowsetId const, std::shared_ptr<doris::Rowset> > > > const&, doris::vectorized::Block&, std::map<unsigned int, unsigned int, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, unsigned int> > >*) const at /home/zcp/repo_center/doris_release/doris/be/src/olap/partial_update_info.cpp:520
8# doris::FlexibleReadPlan::fill_non_primary_key_columns_for_row_store(doris::RowsetWriterContext*, std::map<doris::RowsetId, std::shared_ptr<doris::Rowset>, std::less<doris::RowsetId>, std::allocator<std::pair<doris::RowsetId const, std::shared_ptr<doris::Rowset> > > > const&, doris::TabletSchema const&, std::vector<unsigned int, std::allocator<unsigned int> > const&, doris::vectorized::Block&, std::vector<COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>, std::allocator<COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn> > >&, std::vector<bool, std::allocator<bool> > const&, bool, unsigned long, unsigned long, doris::vectorized::Block const*, std::vector<doris::BitmapValue, std::allocator<doris::BitmapValue> >*) const at /home/zcp/repo_center/doris_release/doris/be/src/olap/partial_update_info.cpp:687
9# doris::FlexibleReadPlan::fill_non_primary_key_columns(doris::RowsetWriterContext*, std::map<doris::RowsetId, std::shared_ptr<doris::Rowset>, std::less<doris::RowsetId>, std::allocator<std::pair<doris::RowsetId const, std::shared_ptr<doris::Rowset> > > > const&, doris::TabletSchema const&, doris::vectorized::Block&, std::vector<bool, std::allocator<bool> > const&, bool, unsigned long, unsigned long, doris::vectorized::Block const*, std::vector<doris::BitmapValue, std::allocator<doris::BitmapValue> >*) const at /home/zcp/repo_center/doris_release/doris/be/src/olap/partial_update_info.cpp:551
10# doris::segment_v2::VerticalSegmentWriter::_append_block_with_flexible_partial_content(doris::segment_v2::RowsInBlock&, doris::vectorized::Block&) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:735
11# doris::segment_v2::VerticalSegmentWriter::write_batch() at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1035
12# doris::SegmentFlusher::_add_rows(std::unique_ptr<doris::segment_v2::VerticalSegmentWriter, std::default_delete<doris::segment_v2::VerticalSegmentWriter> >&, doris::vectorized::Block const*, unsigned long, unsigned long) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_creator.cpp:134
13# doris::SegmentFlusher::flush_single_block(doris::vectorized::Block const*, int, long*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_creator.cpp:75
14# doris::SegmentCreator::flush_single_block(doris::vectorized::Block const*, int, long*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_creator.cpp:416
15# doris::BaseBetaRowsetWriter::flush_memtable(doris::vectorized::Block*, int, long*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/beta_rowset_writer.cpp:750
16# doris::FlushToken::_do_flush_memtable(doris::MemTable*, int, long*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/memtable_flush_executor.cpp:160
17# doris::FlushToken::_flush_memtable(std::shared_ptr<doris::MemTable>, int, long) in /opt/apache-doris-2.1.11-bin-x64/be/lib/doris_be
18# doris::MemtableFlushTask::run() at /home/zcp/repo_center/doris_release/doris/be/src/olap/memtable_flush_executor.cpp:60
19# doris::ThreadPool::dispatch_thread() at /home/zcp/repo_center/doris_release/doris/be/src/util/threadpool.cpp:609
20# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_release/doris/be/src/util/thread.cpp:499
21# start_thread in /lib64/libc.so.6
22# __GI___clone3 in /lib64/libc.so.6
节点3
[libprotobuf ERROR /home/zcp/repo_center/doris_branch-3.1/doris/thirdparty/src/protobuf-21.11/src/google/protobuf/message_lite.cc:133] Can't parse message of type "doris.TabletMetaPB" because it is missing required fields: rs_metas[2].load_id.hi, rs_metas[2].load_id.lo, rs_metas[2].tablet_uid.hi, rs_metas[2].tablet_uid.lo
*** Query id: 0-0 ***
*** is nereids: 0 ***
*** tablet id: 0 ***
*** Aborted at 1781129325 (unix time) try "date -d @1781129325" if you are using GNU date ***
*** Current BE git commitID: 7f5ba43de6 ***
*** SIGSEGV address not mapped to object (@0x8) received by PID 713876 (TID 716515 OR 0x7f458fbfe640) from PID 8; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:421
1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /opt/apache-doris-2.1.11-bin-x64/java17/lib/server/libjvm.so
2# JVM_handle_linux_signal in /opt/apache-doris-2.1.11-bin-x64/java17/lib/server/libjvm.so
3# 0x00007F4C0023FC30 in /lib64/libc.so.6
4# google::protobuf::internal::RepeatedPtrFieldBase::DestroyProtos() in /opt/apache-doris-2.1.11-bin-x64/be/lib/doris_be
5# doris::segment_v2::SegmentFooterPB::SharedDtor() at /home/zcp/repo_center/doris_release/doris/gensrc/build/gen_cpp/segment_v2.pb.cc:5864
6# doris::segment_v2::SegmentFooterPB::~SegmentFooterPB() at /home/zcp/repo_center/doris_release/doris/gensrc/build/gen_cpp/segment_v2.pb.cc:5859
7# doris::MemoryTrackedPageBase<std::shared_ptr<doris::segment_v2::SegmentFooterPB> >::~MemoryTrackedPageBase() at /home/zcp/repo_center/doris_release/doris/be/src/olap/page_cache.h:48
8# doris::MemoryTrackedPageWithPagePtr<doris::segment_v2::SegmentFooterPB>::~MemoryTrackedPageWithPagePtr() at /home/zcp/repo_center/doris_release/doris/be/src/olap/page_cache.cpp:69
9# doris::LRUCache::prune_if(std::function<bool (doris::LRUHandle const*)>, bool) at /home/zcp/repo_center/doris_release/doris/be/src/olap/lru_cache.cpp:615
10# doris::ShardedLRUCache::prune_if(std::function<bool (doris::LRUHandle const*)>, bool) in /opt/apache-doris-2.1.11-bin-x64/be/lib/doris_be
11# doris::LRUCachePolicy::prune_stale() in /opt/apache-doris-2.1.11-bin-x64/be/lib/doris_be
12# doris::CacheManager::for_each_cache_prune_stale_wrap(std::function<void (doris::CachePolicy*)>, doris::RuntimeProfile*) at /home/zcp/repo_center/doris_release/doris/be/src/runtime/memory/cache_manager.cpp:35
13# doris::CacheManager::for_each_cache_prune_stale(doris::RuntimeProfile*) at /home/zcp/repo_center/doris_release/doris/be/src/runtime/memory/cache_manager.cpp:45
14# doris::Daemon::cache_prune_stale_thread() at /home/zcp/repo_center/doris_release/doris/be/src/common/daemon.cpp:523
15# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_release/doris/be/src/util/thread.cpp:499
16# start_thread in /lib64/libc.so.6
17# __GI___clone3 in /lib64/libc.so.6
节点4
thrift error, reason=No more data to read.thrift error, reason=No more data to read.thrift error, reason=No more data to read.thrift error, reason=No more data to read.thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=No more data to read.thrift error, reason=No more data to read.thrift error, reason=No more data to read.thrift error, reason=No more data to read.thrift error, reason=No more data to read.thrift error, reason=No more data to read.F20260611 13:01:01.634954 671800 stream.cpp:113] Check failed: 0 == Socket::Address(fake_sock_id, &ptr) (0 vs. -1)
*** Check failure stack trace: ***
@ 0x55d429b665e6 google::LogMessage::SendToLog()
@ 0x55d429b63030 google::LogMessage::Flush()
@ 0x55d429b66e29 google::LogMessageFatal::~LogMessageFatal()
@ 0x55d42b7b5c4b brpc::Stream::Create()
@ 0x55d42b7b5f92 brpc::StreamAccept()
@ 0x55d41f373a68 std::_Function_handler<>::_M_invoke()
@ 0x55d41f38f340 doris::WorkThreadPool<>::work_thread()
@ 0x55d42ccfba70 execute_native_thread_routine
@ 0x7fd8e928b2ea start_thread
@ 0x7fd8e93103c0 __clone3
@ (nil) (unknown)
*** Query id: 872bd4b71323454b-af5691e13e359c36 ***
*** is nereids: 0 ***
*** tablet id: 0 ***
*** Aborted at 1781154061 (unix time) try "date -d @1781154061" if you are using GNU date ***
*** Current BE git commitID: 7f5ba43de6 ***
*** SIGABRT unknown detail explain (@0xa3b8f) received by PID 670607 (TID 671800 OR 0x7fd6179cd640) from PID 670607; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:421
1# 0x00007FD8E923FC30 in /lib64/libc.so.6
2# __pthread_kill_implementation in /lib64/libc.so.6
3# __GI_raise in /lib64/libc.so.6
4# __GI_abort in /lib64/libc.so.6
5# 0x000055D429B70EBD in /opt/apache-doris-2.1.11-bin-x64/be/lib/doris_be
6# 0x000055D429B634FA in /opt/apache-doris-2.1.11-bin-x64/be/lib/doris_be
7# google::LogMessage::SendToLog() in /opt/apache-doris-2.1.11-bin-x64/be/lib/doris_be
8# google::LogMessage::Flush() in /opt/apache-doris-2.1.11-bin-x64/be/lib/doris_be
9# google::LogMessageFatal::~LogMessageFatal() in /opt/apache-doris-2.1.11-bin-x64/be/lib/doris_be
10# brpc::Stream::Create(brpc::StreamOptions const&, brpc::StreamSettings const*, unsigned long*) in /opt/apache-doris-2.1.11-bin-x64/be/lib/doris_be
11# brpc::StreamAccept(unsigned long*, brpc::Controller&, brpc::StreamOptions const*) in /opt/apache-doris-2.1.11-bin-x64/be/lib/doris_be
12# std::_Function_handler<void (), doris::PInternalService::open_load_stream(google::protobuf::RpcController*, doris::POpenLoadStreamRequest const*, doris::POpenLoadStreamResponse*, google::protobuf::Closure*)::$_0>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
13# doris::WorkThreadPool<false>::work_thread(int) at /home/zcp/repo_center/doris_release/doris/be/src/util/work_thread_pool.hpp:159
14# execute_native_thread_routine at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
15# start_thread in /lib64/libc.so.6
16# __GI___clone3 in /lib64/libc.so.6
节点5
[libprotobuf ERROR /home/zcp/repo_center/doris_branch-3.1/doris/thirdparty/src/protobuf-21.11/src/google/protobuf/message_lite.cc:133] Can't parse message of type "doris.TabletMetaPB" because it is missing required fields: rs_metas[0].rowset_id
thrift error, reason=No more data to read.thrift error, reason=No more data to read.thrift error, reason=No more data to read.thrift error, reason=No more data to read.thrift error, reason=THRIFT_EAGAIN (timed out)thrift error, reason=No more data to read.thrift error, reason=No more data to read.thrift error, reason=No more data to read.thrift error, reason=No more data to read.thrift error, reason=No more data to read.thrift error, reason=No more data to read.thrift error, reason=No more data to read.thrift error, reason=No more data to read.thrift error, reason=No more data to read.thrift error, reason=No more data to read.thrift error, reason=No more data to read.thrift error, reason=No more data to read.thrift error, reason=No more data to read.thrift error, reason=No more data to read.thrift error, reason=No more data to read.thrift error, reason=No more data to read.thrift error, reason=No more data to read.thrift error, reason=No more data to read.thrift error, reason=No more data to read.thrift error, reason=No more data to read.thrift error, reason=No more data to read.F20260611 12:12:34.385785 633133 bitmap_value.h:1928] Check failed: count == _set.size() (unsigned char value 23 vs. 17) bitmap value with incorrect set count
*** Check failure stack trace: ***
@ 0x555c715d45e6 google::LogMessage::SendToLog()
@ 0x555c715d1030 google::LogMessage::Flush()
@ 0x555c715d4e29 google::LogMessageFatal::~LogMessageFatal()
@ 0x555c68f847fd doris::BitmapValue::deserialize()
@ 0x555c6b2b8800 doris::vectorized::DataTypeBitMapSerDe::read_one_cell_from_jsonb()
@ 0x555c70540df9 doris::vectorized::JsonbSerializeUtil::jsonb_to_block()
@ 0x555c70540bd4 doris::vectorized::JsonbSerializeUtil::jsonb_to_block()
@ 0x555c65e30816 doris::BaseTablet::fetch_value_through_row_column()
@ 0x555c66682e0d doris::FlexibleReadPlan::read_columns_by_plan()
@ 0x555c6668415f doris::FlexibleReadPlan::fill_non_primary_key_columns_for_row_store()
@ 0x555c66683152 doris::FlexibleReadPlan::fill_non_primary_key_columns()
@ 0x555c6693682f doris::segment_v2::VerticalSegmentWriter::_append_block_with_flexible_partial_content()
@ 0x555c6693a646 doris::segment_v2::VerticalSegmentWriter::write_batch()
@ 0x555c666e714e doris::SegmentFlusher::_add_rows()
@ 0x555c666e625c doris::SegmentFlusher::flush_single_block()
@ 0x555c666e9932 doris::SegmentCreator::flush_single_block()
@ 0x555c666b9185 doris::BaseBetaRowsetWriter::flush_memtable()
@ 0x555c665fbe87 doris::FlushToken::_do_flush_memtable()
@ 0x555c665fc40b doris::FlushToken::_flush_memtable()
@ 0x555c665ff6a7 doris::MemtableFlushTask::run()
@ 0x555c66f0eb1a doris::ThreadPool::dispatch_thread()
@ 0x555c66f044f1 doris::Thread::supervise_thread()
@ 0x7f98c768b2ea start_thread
@ 0x7f98c77103c0 __clone3
@ (nil) (unknown)
*** Query id: 89408fdccd70780e-aeb65d1e2010518a ***
*** is nereids: 0 ***
*** tablet id: 1780317202520 ***
*** Aborted at 1781151154 (unix time) try "date -d @1781151154" if you are using GNU date ***
*** Current BE git commitID: 7f5ba43de6 ***
*** SIGABRT unknown detail explain (@0xa98b) received by PID 43403 (TID 633133 OR 0x7e6bdc3fe640) from PID 43403; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:421
1# 0x00007F98C763FC30 in /lib64/libc.so.6
2# __pthread_kill_implementation in /lib64/libc.so.6
3# __GI_raise in /lib64/libc.so.6
4# __GI_abort in /lib64/libc.so.6
5# 0x0000555C715DEEBD in /opt/apache-doris-2.1.11-bin-x64/be/lib/doris_be
6# 0x0000555C715D14FA in /opt/apache-doris-2.1.11-bin-x64/be/lib/doris_be
7# google::LogMessage::SendToLog() in /opt/apache-doris-2.1.11-bin-x64/be/lib/doris_be
8# google::LogMessage::Flush() in /opt/apache-doris-2.1.11-bin-x64/be/lib/doris_be
9# google::LogMessageFatal::~LogMessageFatal() in /opt/apache-doris-2.1.11-bin-x64/be/lib/doris_be
10# doris::BitmapValue::deserialize(char const*) at /home/zcp/repo_center/doris_release/doris/be/src/util/bitmap_value.h:1928
11# doris::vectorized::DataTypeBitMapSerDe::read_one_cell_from_jsonb(doris::vectorized::IColumn&, doris::JsonbValue const*) const in /opt/apache-doris-2.1.11-bin-x64/be/lib/doris_be
12# doris::vectorized::JsonbSerializeUtil::jsonb_to_block(std::vector<std::shared_ptr<doris::vectorized::DataTypeSerDe>, std::allocator<std::shared_ptr<doris::vectorized::DataTypeSerDe> > > const&, char const*, unsigned long, std::unordered_map<unsigned int, unsigned int, std::hash<unsigned int>, std::equal_to<unsigned int>, std::allocator<std::pair<unsigned int const, unsigned int> > > const&, doris::vectorized::Block&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::unordered_set<int, std::hash<int>, std::equal_to<int>, std::allocator<int> > const&) in /opt/apache-doris-2.1.11-bin-x64/be/lib/doris_be
13# doris::vectorized::JsonbSerializeUtil::jsonb_to_block(std::vector<std::shared_ptr<doris::vectorized::DataTypeSerDe>, std::allocator<std::shared_ptr<doris::vectorized::DataTypeSerDe> > > const&, doris::vectorized::ColumnStr<unsigned int> const&, std::unordered_map<unsigned int, unsigned int, std::hash<unsigned int>, std::equal_to<unsigned int>, std::allocator<std::pair<unsigned int const, unsigned int> > > const&, doris::vectorized::Block&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::unordered_set<int, std::hash<int>, std::equal_to<int>, std::allocator<int> > const&) in /opt/apache-doris-2.1.11-bin-x64/be/lib/doris_be
14# doris::BaseTablet::fetch_value_through_row_column(std::shared_ptr<doris::Rowset>, doris::TabletSchema const&, unsigned int, std::vector<unsigned int, std::allocator<unsigned int> > const&, std::vector<unsigned int, std::allocator<unsigned int> > const&, doris::vectorized::Block&) in /opt/apache-doris-2.1.11-bin-x64/be/lib/doris_be
15# doris::FlexibleReadPlan::read_columns_by_plan(doris::TabletSchema const&, std::vector<unsigned int, std::allocator<unsigned int> > const&, std::map<doris::RowsetId, std::shared_ptr<doris::Rowset>, std::less<doris::RowsetId>, std::allocator<std::pair<doris::RowsetId const, std::shared_ptr<doris::Rowset> > > > const&, doris::vectorized::Block&, std::map<unsigned int, unsigned int, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, unsigned int> > >*) const at /home/zcp/repo_center/doris_release/doris/be/src/olap/partial_update_info.cpp:520
16# doris::FlexibleReadPlan::fill_non_primary_key_columns_for_row_store(doris::RowsetWriterContext*, std::map<doris::RowsetId, std::shared_ptr<doris::Rowset>, std::less<doris::RowsetId>, std::allocator<std::pair<doris::RowsetId const, std::shared_ptr<doris::Rowset> > > > const&, doris::TabletSchema const&, std::vector<unsigned int, std::allocator<unsigned int> > const&, doris::vectorized::Block&, std::vector<COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>, std::allocator<COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn> > >&, std::vector<bool, std::allocator<bool> > const&, bool, unsigned long, unsigned long, doris::vectorized::Block const*, std::vector<doris::BitmapValue, std::allocator<doris::BitmapValue> >*) const at /home/zcp/repo_center/doris_release/doris/be/src/olap/partial_update_info.cpp:687
17# doris::FlexibleReadPlan::fill_non_primary_key_columns(doris::RowsetWriterContext*, std::map<doris::RowsetId, std::shared_ptr<doris::Rowset>, std::less<doris::RowsetId>, std::allocator<std::pair<doris::RowsetId const, std::shared_ptr<doris::Rowset> > > > const&, doris::TabletSchema const&, doris::vectorized::Block&, std::vector<bool, std::allocator<bool> > const&, bool, unsigned long, unsigned long, doris::vectorized::Block const*, std::vector<doris::BitmapValue, std::allocator<doris::BitmapValue> >*) const at /home/zcp/repo_center/doris_release/doris/be/src/olap/partial_update_info.cpp:551
18# doris::segment_v2::VerticalSegmentWriter::_append_block_with_flexible_partial_content(doris::segment_v2::RowsInBlock&, doris::vectorized::Block&) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:735
19# doris::segment_v2::VerticalSegmentWriter::write_batch() at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:1035
20# doris::SegmentFlusher::_add_rows(std::unique_ptr<doris::segment_v2::VerticalSegmentWriter, std::default_delete<doris::segment_v2::VerticalSegmentWriter> >&, doris::vectorized::Block const*, unsigned long, unsigned long) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_creator.cpp:134
21# doris::SegmentFlusher::flush_single_block(doris::vectorized::Block const*, int, long*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_creator.cpp:75
22# doris::SegmentCreator::flush_single_block(doris::vectorized::Block const*, int, long*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_creator.cpp:416
23# doris::BaseBetaRowsetWriter::flush_memtable(doris::vectorized::Block*, int, long*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/beta_rowset_writer.cpp:750
24# doris::FlushToken::_do_flush_memtable(doris::MemTable*, int, long*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/memtable_flush_executor.cpp:160
25# doris::FlushToken::_flush_memtable(std::shared_ptr<doris::MemTable>, int, long) in /opt/apache-doris-2.1.11-bin-x64/be/lib/doris_be
26# doris::MemtableFlushTask::run() at /home/zcp/repo_center/doris_release/doris/be/src/olap/memtable_flush_executor.cpp:60
27# doris::ThreadPool::dispatch_thread() at /home/zcp/repo_center/doris_release/doris/be/src/util/threadpool.cpp:609
28# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_release/doris/be/src/util/thread.cpp:499
29# start_thread in /lib64/libc.so.6
30# __GI___clone3 in /lib64/libc.so.6
