doris BE 经常重启-2.1.7

Viewed 46
2.1 cluster

环境

  • os: rockylinux 9.5
  • arch: amd64
  • doris: 2.1.7
  • jdk: 1.8.0_121
  • be: 3 node
  • fe: 3 ndoe(1 follower, 2 observer)
  • network: 10G

问题

基本情况

我们现在将 doris 投入生产环境已经半年多了，在机房建立了6节点的高性能集群。但是在我们日常使用的时候，经常出现 BE 的问题，这个过程中也查找大量资料，从使用实践和参数优化上解决了一些问题。最近依然遇到 BE 重启的情况，我们尝试了很多方法还是没办法解决，希望社区能给我们提供一些帮助。

具体描述

最近在我们刷写数据的时候会重启，在日常查询的时候一会重启，这会导致业务使用时经常出现问题。

be.out 日志

截取了三个 6月1日到现在的 be.out 日志


StdoutLogger 2025-05-31 01:14:47,541 Start time: Sat May 31 01:14:47 AM CST 2025
INFO: java_cmd /usr/java/jdk8/bin/java
INFO: jdk_version 8
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/java_extensions/preload-extensions/preload-extensions-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/java_extensions/java-udf/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
*** Query id: 0-0 ***
*** is nereids: 0 ***
*** tablet id: 0 ***
*** Aborted at 1748727647 (unix time) try "date -d @1748727647" if you are using GNU date ***
*** Current BE git commitID: 443e87e203 ***
*** SIGSEGV address not mapped to object (@0x8) received by PID 3193097 (TID 3196264 OR 0x7f0501632640) from PID 8; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:421
 1# os::Linux::chained_handler(int, siginfo*, void*) in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 3# signalHandler(int, siginfo*, void*) in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 4# 0x00007F0B98A3E730 in /lib64/libc.so.6
 5# doris::LRUCache::prune_if(std::function<bool (doris::LRUHandle const*)>, bool) at /home/zcp/repo_center/doris_release/doris/be/src/olap/lru_cache.cpp:500
 6# doris::ShardedLRUCache::prune_if(std::function<bool (doris::LRUHandle const*)>, bool) in /opt/module/doris/be/lib/doris_be
 7# doris::LRUCachePolicy::prune_stale() in /opt/module/doris/be/lib/doris_be
 8# doris::CacheManager::for_each_cache_prune_stale_wrap(std::function<void (doris::CachePolicy*)>, doris::RuntimeProfile*) at /home/zcp/repo_center/doris_release/doris/be/src/runtime/memory/cache_manager.cpp:35
 9# doris::CacheManager::for_each_cache_prune_stale(doris::RuntimeProfile*) at /home/zcp/repo_center/doris_release/doris/be/src/runtime/memory/cache_manager.cpp:45
10# doris::Daemon::cache_prune_stale_thread() at /home/zcp/repo_center/doris_release/doris/be/src/common/daemon.cpp:386
11# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_release/doris/be/src/util/thread.cpp:499
12# start_thread in /lib64/libc.so.6
13# __clone3 in /lib64/libc.so.6

StdoutLogger 2025-06-01 05:41:02,605 Start time: Sun Jun  1 05:41:02 AM CST 2025
INFO: java_cmd /usr/java/jdk8/bin/java
INFO: jdk_version 8
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/java_extensions/preload-extensions/preload-extensions-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/java_extensions/java-udf/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
*** Query id: 6ad80117093a43c5-bd594960273895ae ***
*** is nereids: 1 ***
*** tablet id: 0 ***
*** Aborted at 1748732965 (unix time) try "date -d @1748732965" if you are using GNU date ***
*** Current BE git commitID: 443e87e203 ***
*** SIGSEGV address not mapped to object (@0x0) received by PID 3434690 (TID 3437938 OR 0x7f8b9b20a640) from PID 0; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:421
 1# os::Linux::chained_handler(int, siginfo*, void*) in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 3# signalHandler(int, siginfo*, void*) in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 4# 0x00007F9207E3E730 in /lib64/libc.so.6
 5# doris::ColumnPredicate* doris::create_comparison_predicate<(doris::PredicateType)1>(doris::TabletColumn const&, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, doris::vectorized::Arena*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/predicate_creator.h:278
 6# doris::parse_to_predicate(doris::TabletColumn const&, unsigned int, doris::TCondition const&, doris::vectorized::Arena*, bool) in /opt/module/doris/be/lib/doris_be
 7# doris::Status doris::DeleteHandler::_parse_column_pred<doris::DeleteSubPredicatePB>(std::shared_ptr<doris::TabletSchema>, std::shared_ptr<doris::TabletSchema>, google::protobuf::RepeatedPtrField<doris::DeleteSubPredicatePB> const&, doris::DeleteConditions*) in /opt/module/doris/be/lib/doris_be
 8# doris::DeleteHandler::init(std::shared_ptr<doris::TabletSchema>, std::vector<std::shared_ptr<doris::RowsetMeta>, std::allocator<std::shared_ptr<doris::RowsetMeta> > > const&, long) at /home/zcp/repo_center/doris_release/doris/be/src/olap/delete_handler.cpp:405
 9# doris::TabletReader::_init_delete_condition(doris::TabletReader::ReaderParams const&) at /home/zcp/repo_center/doris_release/doris/be/src/olap/tablet_reader.cpp:628
10# doris::TabletReader::_init_params(doris::TabletReader::ReaderParams const&) at /home/zcp/repo_center/doris_release/doris/be/src/olap/tablet_reader.cpp:295
11# doris::TabletReader::init(doris::TabletReader::ReaderParams const&) at /home/zcp/repo_center/doris_release/doris/be/src/olap/tablet_reader.cpp:126
12# doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/block_reader.cpp:215
13# doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:228
14# doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:259
15# std::_Function_handler<void (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
16# doris::ThreadPool::dispatch_thread() in /opt/module/doris/be/lib/doris_be
17# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_release/doris/be/src/util/thread.cpp:499
18# start_thread in /lib64/libc.so.6
19# __clone3 in /lib64/libc.so.6

StdoutLogger 2025-06-01 07:09:32,518 Start time: Sun Jun  1 07:09:32 AM CST 2025
INFO: java_cmd /usr/java/jdk8/bin/java
INFO: jdk_version 8
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/java_extensions/preload-extensions/preload-extensions-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/java_extensions/java-udf/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
*** Query id: 1516c1a64724c9d-870b36893a4b8034 ***
*** is nereids: 1 ***
*** tablet id: 0 ***
*** Aborted at 1748735701 (unix time) try "date -d @1748735701" if you are using GNU date ***
*** Current BE git commitID: 443e87e203 ***
*** SIGSEGV address not mapped to object (@0x38) received by PID 3452001 (TID 3454789 OR 0x7fefa6456640) from PID 56; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:421
 1# os::Linux::chained_handler(int, siginfo*, void*) in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 3# signalHandler(int, siginfo*, void*) in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 4# 0x00007FF50BE3E730 in /lib64/libc.so.6
 5# 0x00005573C98F18C0 in /opt/module/doris/be/lib/doris_be
 6# bthread::TaskControl::worker_thread(void*) in /opt/module/doris/be/lib/doris_be
 7# start_thread in /lib64/libc.so.6
 8# __clone3 in /lib64/libc.so.6

StdoutLogger 2025-06-01 07:55:17,631 Start time: Sun Jun  1 07:55:17 AM CST 2025
INFO: java_cmd /usr/java/jdk8/bin/java
INFO: jdk_version 8
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/java_extensions/preload-extensions/preload-extensions-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/java_extensions/java-udf/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
*** Query id: 0-0 ***
*** is nereids: 0 ***
*** tablet id: 0 ***
*** Aborted at 1748738945 (unix time) try "date -d @1748738945" if you are using GNU date ***
*** Current BE git commitID: 443e87e203 ***
*** SIGSEGV address not mapped to object (@0x8) received by PID 3463379 (TID 3465156 OR 0x7fe87ad42640) from PID 8; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:421
 1# os::Linux::chained_handler(int, siginfo*, void*) in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 3# signalHandler(int, siginfo*, void*) in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 4# 0x00007FEBD323E730 in /lib64/libc.so.6
 5# std::_Rb_tree<int, std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > >, std::_Select1st<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > >, std::less<int>, std::allocator<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > > >::_M_erase(std::_Rb_tree_node<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > >*) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:1889
 6# std::_Rb_tree<int, std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > >, std::_Select1st<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > >, std::less<int>, std::allocator<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > > >::_M_erase(std::_Rb_tree_node<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > >*) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:1888
 7# std::_Rb_tree<int, std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > >, std::_Select1st<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > >, std::less<int>, std::allocator<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > > >::_M_erase(std::_Rb_tree_node<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > >*) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:1888
 8# std::_Rb_tree<int, std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > >, std::_Select1st<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > >, std::less<int>, std::allocator<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > > >::_M_erase(std::_Rb_tree_node<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > >*) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:1888
 9# doris::segment_v2::Segment::~Segment() at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/segment.cpp:102
10# std::_Sp_counted_ptr<doris::segment_v2::Segment*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:348
11# doris::SegmentCache::CacheValue::~CacheValue() at /home/zcp/repo_center/doris_release/doris/be/src/olap/segment_loader.h:78
12# doris::SegmentCache::CacheValue::~CacheValue() at /home/zcp/repo_center/doris_release/doris/be/src/olap/segment_loader.h:78
13# doris::LRUCache::erase(doris::CacheKey const&, unsigned int) at /home/zcp/repo_center/doris_release/doris/be/src/olap/lru_cache.cpp:431
14# doris::SegmentLoader::erase_segments(doris::RowsetId const&, long) at /home/zcp/repo_center/doris_release/doris/be/src/olap/segment_loader.cpp:95
15# doris::Rowset::clear_cache() at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/rowset.cpp:130
16# doris::Tablet::clear_cache() at /home/zcp/repo_center/doris_release/doris/be/src/olap/tablet.cpp:4048
17# doris::TabletManager::_drop_tablet_unlocked(long, long, bool, bool) at /home/zcp/repo_center/doris_release/doris/be/src/olap/tablet_manager.cpp:569
18# doris::TabletManager::drop_tablet(long, long, bool) at /home/zcp/repo_center/doris_release/doris/be/src/olap/tablet_manager.cpp:536
19# doris::drop_tablet_callback(doris::StorageEngine&, doris::TAgentTaskRequest const&) in /opt/module/doris/be/lib/doris_be
20# std::_Function_handler<void (), doris::TaskWorkerPool::submit_task(doris::TAgentTaskRequest const&)::$_0::operator()<doris::TAgentTaskRequest const&>(doris::TAgentTaskRequest const&) const::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
21# doris::ThreadPool::dispatch_thread() in /opt/module/doris/be/lib/doris_be
22# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_release/doris/be/src/util/thread.cpp:499
23# start_thread in /lib64/libc.so.6
24# __clone3 in /lib64/libc.so.6

StdoutLogger 2025-06-01 08:49:17,627 Start time: Sun Jun  1 08:49:17 AM CST 2025
INFO: java_cmd /usr/java/jdk8/bin/java
INFO: jdk_version 8
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/java_extensions/preload-extensions/preload-extensions-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/java_extensions/java-udf/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]




###################################################################################

StdoutLogger 2025-05-31 00:04:00,853 Start time: Sat May 31 12:04:00 AM CST 2025
INFO: java_cmd /usr/java/jdk8/bin/java
INFO: jdk_version 8
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/java_extensions/preload-extensions/preload-extensions-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/java_extensions/java-udf/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
*** Query id: 0-0 ***
*** is nereids: 0 ***
*** tablet id: 0 ***
*** Aborted at 1748625909 (unix time) try "date -d @1748625909" if you are using GNU date ***
*** Current BE git commitID: 443e87e203 ***
*** SIGSEGV address not mapped to object (@0x0) received by PID 2189117 (TID 2191877 OR 0x7fcd1fdad640) from PID 0; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:421
 1# os::Linux::chained_handler(int, siginfo*, void*) in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 3# signalHandler(int, siginfo*, void*) in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 4# 0x00007FD27063E730 in /lib64/libc.so.6

StdoutLogger 2025-05-31 01:25:30,855 Start time: Sat May 31 01:25:30 AM CST 2025
INFO: java_cmd /usr/java/jdk8/bin/java
INFO: jdk_version 8
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/java_extensions/preload-extensions/preload-extensions-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/java_extensions/java-udf/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
*** Query id: 0-0 ***
*** is nereids: 0 ***
*** tablet id: 0 ***
*** Aborted at 1748627749 (unix time) try "date -d @1748627749" if you are using GNU date ***
*** Current BE git commitID: 443e87e203 ***
*** SIGSEGV address not mapped to object (@0x8) received by PID 2223360 (TID 2226443 OR 0x7fb285628640) from PID 8; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:421
 1# os::Linux::chained_handler(int, siginfo*, void*) in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 3# signalHandler(int, siginfo*, void*) in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 4# 0x00007FB89BA3E730 in /lib64/libc.so.6
 5# doris::segment_v2::ColumnReader::~ColumnReader() at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/column_reader.cpp:220
 6# doris::segment_v2::ColumnReader::~ColumnReader() at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/column_reader.cpp:217
 7# std::_Rb_tree<int, std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > >, std::_Select1st<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > >, std::less<int>, std::allocator<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > > >::_M_erase(std::_Rb_tree_node<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > >*) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:1889
 8# std::_Rb_tree<int, std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > >, std::_Select1st<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > >, std::less<int>, std::allocator<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > > >::_M_erase(std::_Rb_tree_node<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > >*) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:1888
 9# std::_Rb_tree<int, std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > >, std::_Select1st<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > >, std::less<int>, std::allocator<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > > >::_M_erase(std::_Rb_tree_node<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > >*) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:1888
10# std::_Rb_tree<int, std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > >, std::_Select1st<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > >, std::less<int>, std::allocator<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > > >::_M_erase(std::_Rb_tree_node<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > >*) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:1888
11# std::_Rb_tree<int, std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > >, std::_Select1st<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > >, std::less<int>, std::allocator<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > > >::_M_erase(std::_Rb_tree_node<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > >*) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:1888
12# std::_Rb_tree<int, std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > >, std::_Select1st<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > >, std::less<int>, std::allocator<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > > >::_M_erase(std::_Rb_tree_node<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > >*) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:1888
13# std::_Rb_tree<int, std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > >, std::_Select1st<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > >, std::less<int>, std::allocator<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > > >::_M_erase(std::_Rb_tree_node<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > >*) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:1888
14# doris::segment_v2::Segment::~Segment() at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/segment.cpp:102
15# std::_Sp_counted_ptr<doris::segment_v2::Segment*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:348
16# doris::SegmentCache::CacheValue::~CacheValue() at /home/zcp/repo_center/doris_release/doris/be/src/olap/segment_loader.h:78
17# doris::SegmentCache::CacheValue::~CacheValue() at /home/zcp/repo_center/doris_release/doris/be/src/olap/segment_loader.h:78
18# doris::LRUCache::prune_if(std::function<bool (doris::LRUHandle const*)>, bool) at /home/zcp/repo_center/doris_release/doris/be/src/olap/lru_cache.cpp:500
19# doris::ShardedLRUCache::prune_if(std::function<bool (doris::LRUHandle const*)>, bool) in /opt/module/doris/be/lib/doris_be
20# doris::LRUCachePolicy::prune_stale() in /opt/module/doris/be/lib/doris_be
21# doris::CacheManager::for_each_cache_prune_stale_wrap(std::function<void (doris::CachePolicy*)>, doris::RuntimeProfile*) at /home/zcp/repo_center/doris_release/doris/be/src/runtime/memory/cache_manager.cpp:35
22# doris::CacheManager::for_each_cache_prune_stale(doris::RuntimeProfile*) at /home/zcp/repo_center/doris_release/doris/be/src/runtime/memory/cache_manager.cpp:45
23# doris::Daemon::cache_prune_stale_thread() at /home/zcp/repo_center/doris_release/doris/be/src/common/daemon.cpp:386
24# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_release/doris/be/src/util/thread.cpp:499
25# start_thread in /lib64/libc.so.6
26# __clone3 in /lib64/libc.so.6

StdoutLogger 2025-05-31 01:56:00,850 Start time: Sat May 31 01:56:00 AM CST 2025
INFO: java_cmd /usr/java/jdk8/bin/java
INFO: jdk_version 8
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/java_extensions/preload-extensions/preload-extensions-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/java_extensions/java-udf/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
*** Query id: a3275f3b966f4fa1-a67aadc806a532e9 ***
*** is nereids: 1 ***
*** tablet id: 0 ***
*** Aborted at 1748629930 (unix time) try "date -d @1748629930" if you are using GNU date ***
*** Current BE git commitID: 443e87e203 ***
*** SIGSEGV address not mapped to object (@0x18) received by PID 2232758 (TID 2236293 OR 0x7f0f6eaed640) from PID 24; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:421
 1# os::Linux::chained_handler(int, siginfo*, void*) in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 3# signalHandler(int, siginfo*, void*) in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 4# 0x00007F1639E3E730 in /lib64/libc.so.6
 5# doris::segment_v2::OrdinalIndexReader::_load(bool, bool, std::unique_ptr<doris::segment_v2::OrdinalIndexPB, std::default_delete<doris::segment_v2::OrdinalIndexPB> >) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/ordinal_page_index.cpp:83
 6# doris::segment_v2::OrdinalIndexReader::load(bool, bool) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/ordinal_page_index.cpp:76
 7# doris::segment_v2::ColumnReader::seek_at_or_before(unsigned long, doris::segment_v2::OrdinalPageIndexIterator*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/column_reader.cpp:629
 8# doris::segment_v2::FileColumnIterator::seek_to_ordinal(unsigned long) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/column_reader.cpp:1065
 9# doris::segment_v2::FileColumnIterator::read_by_rowids(unsigned int const*, unsigned long, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>&) in /opt/module/doris/be/lib/doris_be
10# doris::segment_v2::SegmentIterator::_read_columns_by_index(unsigned int, unsigned int&, bool) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:1729
11# doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2070
12# doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:1908
13# doris::segment_v2::LazyInitSegmentIterator::next_batch(doris::vectorized::Block*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/lazy_init_segment_iterator.h:45
14# doris::BetaRowsetReader::next_block(doris::vectorized::Block*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/beta_rowset_reader.cpp:342
15# doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row() at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.cpp:508
16# doris::vectorized::VCollectIterator::Level0Iterator::ensure_first_row_ref() at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.cpp:483
17# doris::vectorized::VCollectIterator::Level1Iterator::ensure_first_row_ref() in /opt/module/doris/be/lib/doris_be
18# doris::vectorized::VCollectIterator::build_heap(std::vector<std::shared_ptr<doris::RowsetReader>, std::allocator<std::shared_ptr<doris::RowsetReader> > >&) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.cpp:186
19# doris::vectorized::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/block_reader.cpp:162
20# doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/block_reader.cpp:235
21# doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:228
22# doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:259
23# std::_Function_handler<void (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
24# doris::ThreadPool::dispatch_thread() in /opt/module/doris/be/lib/doris_be
25# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_release/doris/be/src/util/thread.cpp:499
26# start_thread in /lib64/libc.so.6
27# __clone3 in /lib64/libc.so.6

StdoutLogger 2025-05-31 02:32:30,860 Start time: Sat May 31 02:32:30 AM CST 2025
INFO: java_cmd /usr/java/jdk8/bin/java
INFO: jdk_version 8
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/java_extensions/preload-extensions/preload-extensions-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/java_extensions/java-udf/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
*** Query id: b0b4b96241aa4e3f-b26fd00c4e78386d ***
*** is nereids: 1 ***
*** tablet id: 0 ***
*** Aborted at 1748637468 (unix time) try "date -d @1748637468" if you are using GNU date ***
*** Current BE git commitID: 443e87e203 ***
*** SIGSEGV address not mapped to object (@0x38) received by PID 2243118 (TID 2245831 OR 0x7f5c4c8f2640) from PID 56; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:421
 1# os::Linux::chained_handler(int, siginfo*, void*) in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 3# signalHandler(int, siginfo*, void*) in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 4# 0x00007F61A103E730 in /lib64/libc.so.6
 5# 0x0000560657B547FA in /opt/module/doris/be/lib/doris_be
 6# bthread::TaskGroup::task_runner(long) in /opt/module/doris/be/lib/doris_be
 7# bthread_make_fcontext in /opt/module/doris/be/lib/doris_be

StdoutLogger 2025-05-31 04:38:00,849 Start time: Sat May 31 04:38:00 AM CST 2025
INFO: java_cmd /usr/java/jdk8/bin/java
INFO: jdk_version 8
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/java_extensions/preload-extensions/preload-extensions-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/java_extensions/java-udf/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
*** Query id: 0-0 ***
*** is nereids: 0 ***
*** tablet id: 0 ***
*** Aborted at 1748740607 (unix time) try "date -d @1748740607" if you are using GNU date ***
*** Current BE git commitID: 443e87e203 ***
*** SIGSEGV address not mapped to object (@0x1a8) received by PID 2265537 (TID 2267054 OR 0x7fbf7b5e4640) from PID 424; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:421
 1# os::Linux::chained_handler(int, siginfo*, void*) in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 3# signalHandler(int, siginfo*, void*) in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 4# 0x00007FC26DE3E730 in /lib64/libc.so.6
 5# std::_Function_handler<void (std::shared_ptr<doris::Tablet> const&), doris::TabletManager::find_best_tablet_to_compaction(doris::CompactionType, doris::DataDir*, std::unordered_set<std::shared_ptr<doris::Tablet>, std::hash<std::shared_ptr<doris::Tablet> >, std::equal_to<std::shared_ptr<doris::Tablet> >, std::allocator<std::shared_ptr<doris::Tablet> > > const&, unsigned int*, std::unordered_map<std::basic_string_view<char, std::char_traits<char> >, std::shared_ptr<doris::CumulativeCompactionPolicy>, std::hash<std::basic_string_view<char, std::char_traits<char> > >, std::equal_to<std::basic_string_view<char, std::char_traits<char> > >, std::allocator<std::pair<std::basic_string_view<char, std::char_traits<char> > const, std::shared_ptr<doris::CumulativeCompactionPolicy> > > > const&)::$_0>::_M_invoke(std::_Any_data const&, std::shared_ptr<doris::Tablet> const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
 6# doris::TabletManager::for_each_tablet(std::function<void (std::shared_ptr<doris::Tablet> const&)>&&, std::function<bool (doris::Tablet*)>&&) at /home/zcp/repo_center/doris_release/doris/be/src/olap/tablet_manager.cpp:624
 7# doris::TabletManager::find_best_tablet_to_compaction(doris::CompactionType, doris::DataDir*, std::unordered_set<std::shared_ptr<doris::Tablet>, std::hash<std::shared_ptr<doris::Tablet> >, std::equal_to<std::shared_ptr<doris::Tablet> >, std::allocator<std::shared_ptr<doris::Tablet> > > const&, unsigned int*, std::unordered_map<std::basic_string_view<char, std::char_traits<char> >, std::shared_ptr<doris::CumulativeCompactionPolicy>, std::hash<std::basic_string_view<char, std::char_traits<char> > >, std::equal_to<std::basic_string_view<char, std::char_traits<char> > >, std::allocator<std::pair<std::basic_string_view<char, std::char_traits<char> > const, std::shared_ptr<doris::CumulativeCompactionPolicy> > > > const&) at /home/zcp/repo_center/doris_release/doris/be/src/olap/tablet_manager.cpp:804
 8# doris::StorageEngine::_generate_compaction_tasks(doris::CompactionType, std::vector<doris::DataDir*, std::allocator<doris::DataDir*> >&, bool) at /home/zcp/repo_center/doris_release/doris/be/src/olap/olap_server.cpp:901
 9# doris::StorageEngine::_compaction_tasks_producer_callback() at /home/zcp/repo_center/doris_release/doris/be/src/olap/olap_server.cpp:621
10# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_release/doris/be/src/util/thread.cpp:499
11# start_thread in /lib64/libc.so.6
12# __clone3 in /lib64/libc.so.6

StdoutLogger 2025-06-01 09:17:00,845 Start time: Sun Jun  1 09:17:00 AM CST 2025
INFO: java_cmd /usr/java/jdk8/bin/java
INFO: jdk_version 8
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/java_extensions/preload-extensions/preload-extensions-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/java_extensions/java-udf/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
*** Query id: 0-0 ***
*** is nereids: 0 ***
*** tablet id: 0 ***
*** Aborted at 1748794232 (unix time) try "date -d @1748794232" if you are using GNU date ***
*** Current BE git commitID: 443e87e203 ***
*** SIGSEGV address not mapped to object (@0x0) received by PID 2502280 (TID 2504083 OR 0x7f353997a640) from PID 0; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:421
 1# os::Linux::chained_handler(int, siginfo*, void*) in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 3# signalHandler(int, siginfo*, void*) in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 4# 0x00007F390263E730 in /lib64/libc.so.6
 5# doris::TabletColumn::to_schema_pb(doris::ColumnPB*) const at /home/zcp/repo_center/doris_release/doris/be/src/olap/tablet_schema.cpp:600
 6# doris::TabletSchema::to_schema_pb(doris::TabletSchemaPB*) const at /home/zcp/repo_center/doris_release/doris/be/src/olap/tablet_schema.cpp:1183
 7# doris::TabletMeta::to_meta_pb(doris::TabletMetaPB*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/tablet_meta.cpp:704
 8# doris::TabletMeta::serialize(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/tablet_meta.cpp:500
 9# doris::TabletMeta::_save_meta(doris::DataDir*) in /opt/module/doris/be/lib/doris_be
10# doris::TabletMeta::save_meta(doris::DataDir*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/tablet_meta.cpp:469
11# doris::Tablet::save_meta() at /home/zcp/repo_center/doris_release/doris/be/src/olap/tablet.cpp:351
12# doris::TabletManager::_drop_tablet_unlocked(long, long, bool, bool) in /opt/module/doris/be/lib/doris_be
13# doris::TabletManager::drop_tablet(long, long, bool) at /home/zcp/repo_center/doris_release/doris/be/src/olap/tablet_manager.cpp:536
14# doris::drop_tablet_callback(doris::StorageEngine&, doris::TAgentTaskRequest const&) in /opt/module/doris/be/lib/doris_be
15# std::_Function_handler<void (), doris::TaskWorkerPool::submit_task(doris::TAgentTaskRequest const&)::$_0::operator()<doris::TAgentTaskRequest const&>(doris::TAgentTaskRequest const&) const::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
16# doris::ThreadPool::dispatch_thread() in /opt/module/doris/be/lib/doris_be
17# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_release/doris/be/src/util/thread.cpp:499
18# start_thread in /lib64/libc.so.6
19# __clone3 in /lib64/libc.so.6

StdoutLogger 2025-06-02 00:10:45,877 Start time: Mon Jun  2 12:10:45 AM CST 2025
INFO: java_cmd /usr/java/jdk8/bin/java
INFO: jdk_version 8
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/java_extensions/preload-extensions/preload-extensions-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/java_extensions/java-udf/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]





###################################################################################

StdoutLogger 2025-05-31 02:00:37,673 Start time: Sat May 31 02:00:37 AM CST 2025
INFO: java_cmd /usr/java/jdk8/bin/java
INFO: jdk_version 8
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/java_extensions/preload-extensions/preload-extensions-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/java_extensions/java-udf/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
*** Query id: 0-0 ***
*** is nereids: 0 ***
*** tablet id: 0 ***
*** Aborted at 1748636456 (unix time) try "date -d @1748636456" if you are using GNU date ***
*** Current BE git commitID: 443e87e203 ***
*** SIGSEGV address not mapped to object (@0x8) received by PID 164180 (TID 167299 OR 0x7fe4be9bc640) from PID 8; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:421
 1# os::Linux::chained_handler(int, siginfo*, void*) in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 3# signalHandler(int, siginfo*, void*) in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 4# 0x00007FEB0AA3E730 in /lib64/libc.so.6
 5# doris::segment_v2::ColumnReader::~ColumnReader() at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/column_reader.cpp:220
 6# doris::segment_v2::ColumnReader::~ColumnReader() at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/column_reader.cpp:217
 7# std::_Rb_tree<int, std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > >, std::_Select1st<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > >, std::less<int>, std::allocator<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > > >::_M_erase(std::_Rb_tree_node<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > >*) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:1889
 8# std::_Rb_tree<int, std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > >, std::_Select1st<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > >, std::less<int>, std::allocator<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > > >::_M_erase(std::_Rb_tree_node<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > >*) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:1888
 9# std::_Rb_tree<int, std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > >, std::_Select1st<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > >, std::less<int>, std::allocator<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > > >::_M_erase(std::_Rb_tree_node<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > >*) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:1888
10# std::_Rb_tree<int, std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > >, std::_Select1st<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > >, std::less<int>, std::allocator<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > > >::_M_erase(std::_Rb_tree_node<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > >*) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:1888
11# std::_Rb_tree<int, std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > >, std::_Select1st<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > >, std::less<int>, std::allocator<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > > >::_M_erase(std::_Rb_tree_node<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > >*) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:1888
12# std::_Rb_tree<int, std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > >, std::_Select1st<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > >, std::less<int>, std::allocator<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > > >::_M_erase(std::_Rb_tree_node<std::pair<int const, std::unique_ptr<doris::segment_v2::ColumnReader, std::default_delete<doris::segment_v2::ColumnReader> > > >*) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:1888
13# doris::segment_v2::Segment::~Segment() at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/segment.cpp:102
14# std::_Sp_counted_ptr<doris::segment_v2::Segment*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:348
15# doris::SegmentCache::CacheValue::~CacheValue() at /home/zcp/repo_center/doris_release/doris/be/src/olap/segment_loader.h:78
16# doris::SegmentCache::CacheValue::~CacheValue() at /home/zcp/repo_center/doris_release/doris/be/src/olap/segment_loader.h:78
17# doris::LRUCache::prune_if(std::function<bool (doris::LRUHandle const*)>, bool) at /home/zcp/repo_center/doris_release/doris/be/src/olap/lru_cache.cpp:500
18# doris::ShardedLRUCache::prune_if(std::function<bool (doris::LRUHandle const*)>, bool) in /opt/module/doris/be/lib/doris_be
19# doris::LRUCachePolicy::prune_stale() in /opt/module/doris/be/lib/doris_be
20# doris::CacheManager::for_each_cache_prune_stale_wrap(std::function<void (doris::CachePolicy*)>, doris::RuntimeProfile*) at /home/zcp/repo_center/doris_release/doris/be/src/runtime/memory/cache_manager.cpp:35
21# doris::CacheManager::for_each_cache_prune_stale(doris::RuntimeProfile*) at /home/zcp/repo_center/doris_release/doris/be/src/runtime/memory/cache_manager.cpp:45
22# doris::Daemon::cache_prune_stale_thread() at /home/zcp/repo_center/doris_release/doris/be/src/common/daemon.cpp:386
23# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_release/doris/be/src/util/thread.cpp:499
24# start_thread in /lib64/libc.so.6
25# __clone3 in /lib64/libc.so.6

StdoutLogger 2025-05-31 04:21:07,680 Start time: Sat May 31 04:21:07 AM CST 2025
INFO: java_cmd /usr/java/jdk8/bin/java
INFO: jdk_version 8
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/java_extensions/preload-extensions/preload-extensions-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/java_extensions/java-udf/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
*** Query id: 0-0 ***
*** is nereids: 0 ***
*** tablet id: 19201686 ***
*** Aborted at 1748729718 (unix time) try "date -d @1748729718" if you are using GNU date ***
*** Current BE git commitID: 443e87e203 ***
*** SIGSEGV address not mapped to object (@0x10) received by PID 189544 (TID 190972 OR 0x7feca33fe640) from PID 16; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:421
 1# os::Linux::chained_handler(int, siginfo*, void*) in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 3# signalHandler(int, siginfo*, void*) in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 4# 0x00007FEFBAC3E730 in /lib64/libc.so.6
 5# doris::DeleteHandler::parse_condition(doris::DeleteSubPredicatePB const&, doris::TCondition*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/delete_handler.cpp:291
 6# doris::Status doris::DeleteHandler::_parse_column_pred<doris::DeleteSubPredicatePB>(std::shared_ptr<doris::TabletSchema>, std::shared_ptr<doris::TabletSchema>, google::protobuf::RepeatedPtrField<doris::DeleteSubPredicatePB> const&, doris::DeleteConditions*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/delete_handler.cpp:359
 7# doris::DeleteHandler::init(std::shared_ptr<doris::TabletSchema>, std::vector<std::shared_ptr<doris::RowsetMeta>, std::allocator<std::shared_ptr<doris::RowsetMeta> > > const&, long) at /home/zcp/repo_center/doris_release/doris/be/src/olap/delete_handler.cpp:405
 8# doris::TabletReader::_init_delete_condition(doris::TabletReader::ReaderParams const&) at /home/zcp/repo_center/doris_release/doris/be/src/olap/tablet_reader.cpp:628
 9# doris::TabletReader::_init_params(doris::TabletReader::ReaderParams const&) at /home/zcp/repo_center/doris_release/doris/be/src/olap/tablet_reader.cpp:295
10# doris::TabletReader::init(doris::TabletReader::ReaderParams const&) at /home/zcp/repo_center/doris_release/doris/be/src/olap/tablet_reader.cpp:126
11# doris::vectorized::VerticalBlockReader::init(doris::TabletReader::ReaderParams const&, doris::CompactionSampleInfo*) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vertical_block_reader.cpp:224
12# doris::Merger::vertical_compact_one_group(std::shared_ptr<doris::Tablet>, doris::ReaderType, std::shared_ptr<doris::TabletSchema>, bool, std::vector<unsigned int, std::allocator<unsigned int> > const&, doris::vectorized::RowSourcesBuffer*, std::vector<std::shared_ptr<doris::RowsetReader>, std::allocator<std::shared_ptr<doris::RowsetReader> > > const&, doris::RowsetWriter*, long, doris::Merger::Statistics*, std::vector<unsigned int, std::allocator<unsigned int> >, long, doris::CompactionSampleInfo*) in /opt/module/doris/be/lib/doris_be
13# doris::Merger::vertical_merge_rowsets(std::shared_ptr<doris::Tablet>, doris::ReaderType, std::shared_ptr<doris::TabletSchema>, std::vector<std::shared_ptr<doris::RowsetReader>, std::allocator<std::shared_ptr<doris::RowsetReader> > > const&, doris::RowsetWriter*, long, long, doris::Merger::Statistics*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/merger.cpp:445
14# doris::Compaction::do_compaction_impl(long) at /home/zcp/repo_center/doris_release/doris/be/src/olap/compaction.cpp:387
15# doris::Compaction::do_compaction(long) at /home/zcp/repo_center/doris_release/doris/be/src/olap/compaction.cpp:137
16# doris::BaseCompaction::execute_compact_impl() at /home/zcp/repo_center/doris_release/doris/be/src/olap/base_compaction.cpp:79
17# doris::Compaction::execute_compact() at /home/zcp/repo_center/doris_release/doris/be/src/olap/compaction.cpp:119
18# doris::Tablet::execute_compaction(doris::Compaction&) at /home/zcp/repo_center/doris_release/doris/be/src/olap/tablet.cpp:2069
19# std::_Function_handler<void (), doris::StorageEngine::_submit_compaction_task(std::shared_ptr<doris::Tablet>, doris::CompactionType, bool)::$_0>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
20# doris::ThreadPool::dispatch_thread() in /opt/module/doris/be/lib/doris_be
21# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_release/doris/be/src/util/thread.cpp:499
22# start_thread in /lib64/libc.so.6
23# __clone3 in /lib64/libc.so.6

StdoutLogger 2025-06-01 06:15:37,765 Start time: Sun Jun  1 06:15:37 AM CST 2025
INFO: java_cmd /usr/java/jdk8/bin/java
INFO: jdk_version 8
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/java_extensions/preload-extensions/preload-extensions-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/java_extensions/java-udf/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
*** Query id: aa1823c224a94326-84f48640d21146ec ***
*** is nereids: 1 ***
*** tablet id: 0 ***
*** Aborted at 1748799438 (unix time) try "date -d @1748799438" if you are using GNU date ***
*** Current BE git commitID: 443e87e203 ***
*** SIGSEGV address not mapped to object (@0x0) received by PID 408528 (TID 411796 OR 0x7f8879179640) from PID 0; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:421
 1# os::Linux::chained_handler(int, siginfo*, void*) in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 3# signalHandler(int, siginfo*, void*) in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 4# 0x00007F8ED6C3E730 in /lib64/libc.so.6
 5# doris::vectorized::NewOlapScanner::_collect_profile_before_close() at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:676
 6# doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:333
 7# std::_Function_handler<void (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
 8# doris::ThreadPool::dispatch_thread() in /opt/module/doris/be/lib/doris_be
 9# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_release/doris/be/src/util/thread.cpp:499
10# start_thread in /lib64/libc.so.6
11# __clone3 in /lib64/libc.so.6

StdoutLogger 2025-06-02 01:37:37,765 Start time: Mon Jun  2 01:37:37 AM CST 2025
INFO: java_cmd /usr/java/jdk8/bin/java
INFO: jdk_version 8
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/java_extensions/preload-extensions/preload-extensions-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/java_extensions/java-udf/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
*** Query id: 4b2053dd850c40b6-a83a83b2e20e891a ***
*** is nereids: 0 ***
*** tablet id: 0 ***
*** Aborted at 1748826012 (unix time) try "date -d @1748826012" if you are using GNU date ***
*** Current BE git commitID: 443e87e203 ***
*** SIGSEGV address not mapped to object (@0x0) received by PID 580550 (TID 582974 OR 0x7f81648d7640) from PID 0; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:421
 1# os::Linux::chained_handler(int, siginfo*, void*) in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 3# signalHandler(int, siginfo*, void*) in /usr/java/jdk8/jre/lib/amd64/server/libjvm.so
 4# 0x00007F8634A3E730 in /lib64/libc.so.6
 5# brpc::Socket::Create(brpc::SocketOptions const&, unsigned long*) in /opt/module/doris/be/lib/doris_be
 6# brpc::Stream::Create(brpc::StreamOptions const&, brpc::StreamSettings const*, unsigned long*) in /opt/module/doris/be/lib/doris_be
 7# brpc::StreamAccept(unsigned long*, brpc::Controller&, brpc::StreamOptions const*) in /opt/module/doris/be/lib/doris_be
 8# std::_Function_handler<void (), doris::PInternalServiceImpl::open_load_stream(google::protobuf::RpcController*, doris::POpenLoadStreamRequest const*, doris::POpenLoadStreamResponse*, google::protobuf::Closure*)::$_0>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
 9# doris::WorkThreadPool<false>::work_thread(int) at /home/zcp/repo_center/doris_release/doris/be/src/util/work_thread_pool.hpp:159
10# execute_native_thread_routine at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
11# start_thread in /lib64/libc.so.6
12# __clone3 in /lib64/libc.so.6

StdoutLogger 2025-06-02 09:00:22,775 Start time: Mon Jun  2 09:00:22 AM CST 2025
INFO: java_cmd /usr/java/jdk8/bin/java
INFO: jdk_version 8
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/java_extensions/preload-extensions/preload-extensions-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/java_extensions/java-udf/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/doris/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]

希望社区的小伙伴帮我分析解决一下问题。

2 Answers

GPT 搜索了一下，供参考。
Doris 崩溃日志分析总结

从提供的多个崩溃日志来看，Doris BE（Backend）节点频繁重启的主要原因是 内存访问异常导致的 SIGSEGV 段错误。这些错误集中在几个关键模块，以下是核心问题和解决方向：

主要问题分类及原因分析

  1. 缓存管理模块的内存释放问题（高频问题）
    • 崩溃位置：

    ◦ doris::LRUCache::prune_if

    ◦ doris::ShardedLRUCache::prune_if

    ◦ doris::Segment::~Segment()

    • 原因：

    ◦ 后台线程 cache_prune_stale_thread 清理过期缓存时，访问已释放内存。

    ◦ 可能因 缓存对象生命周期管理缺陷（如提前释放对象后仍被访问）或 多线程竞争（清理线程与查询线程同时操作缓存）导致。

    • 日志示例：

    *** SIGSEGV address not mapped to object (@0x8) ... 
5# doris::LRUCache::prune_if(...) at lru_cache.cpp:500
7# doris::CacheManager::for_each_cache_prune_stale(...)
10# doris::Daemon::cache_prune_stale_thread()

  2. 删除条件解析逻辑缺陷（高频问题）
    • 崩溃位置：

    ◦ doris::DeleteHandler::init

    ◦ doris::parse_to_predicate

    • 原因：

    ◦ 解析表的删除条件（Delete Condition）时出现空指针或无效内存访问。

    ◦ 可能与 Schema 变更后旧版删除条件未兼容 或 PB 数据解析异常 有关。

    • 日志示例：

    *** SIGSEGV address not mapped to object (@0x0) ...
5# doris::ColumnPredicate* doris::create_comparison_predicate<...>(...)
6# doris::parse_to_predicate(...)
7# doris::DeleteHandler::init(...)
9# doris::TabletReader::_init_delete_condition(...)

  3. Segment 和 ColumnReader 析构异常
    • 崩溃位置：

    ◦ doris::segment_v2::Segment::~Segment()

    ◦ doris::segment_v2::ColumnReader::~ColumnReader()

    • 原因：

    ◦ Segment 或 ColumnReader 析构时访问无效内存（如已被释放的 RB-Tree 节点）。

    ◦ 可能因 多线程下对象析构顺序错误（如缓存清理与查询线程并发操作 Segment）导致。

  4. Compaction 任务逻辑缺陷
    • 崩溃位置：

    ◦ doris::Merger::vertical_compact_one_group

    ◦ doris::Compaction::do_compaction_impl

    • 原因：

    ◦ 数据压缩过程中处理删除条件或读取行集（Rowset）时内存访问异常。

根本原因总结
• 内存管理缺陷：LRU 缓存清理、对象析构等场景存在 释放后使用（Use-After-Free） 或 空指针解引用 问题。

• 并发冲突：后台线程（缓存清理、Compaction）与查询线程并发操作共享对象（如 Segment、缓存条目），缺少同步保护。

• 数据兼容性问题：旧版删除条件解析逻辑未能处理 Schema 变更后的异常情况。

解决建议

  1. 修复缓存管理模块
    • 生命周期加固：使用 std::shared_ptr 管理缓存对象引用计数，确保清理时不访问已释放对象。

    • 同步机制：为缓存操作（如 prune_if）添加读写锁（如 boost::shared_mutex），避免与查询线程并发冲突。

    • 日志增强：在 LRUCache::prune_if 中打印被清理对象的 Key，便于追踪问题缓存条目。

  2. 修复删除条件解析
    • 空指针检查：在 DeleteHandler::init 中增加对 TabletSchema 和 DeleteSubPredicatePB 的健壮性检查：

    if (tablet_schema == nullptr || !delete_condition_pb.has_column_name()) {
    return Status::InvalidArgument("Invalid delete condition");
}

    • Schema 兼容性：检测旧版删除条件，必要时跳过解析或执行转换。

  3. 修复 Segment 析构逻辑
    • 资源释放顺序：确保 ColumnReader 在 Segment 析构前释放其持有的资源（如 RB-Tree 节点）。

    • 日志增强：在 Segment::~Segment() 中打印 Segment ID，确认是否因重复释放触发崩溃。

  4. 升级版本
    • 当前版本 CommitID 为 443e87e203，需检查 Doris 社区 Issue 是否已有相关修复：

    ◦ 搜索关键词：SIGSEGV prune_if、DeleteHandler init crash。

    • 建议升级到最新稳定版（如 1.2.7+），已知类似问题可能已在后续版本修复。

临时缓解措施
• 关闭缓存清理：修改 BE 配置项 cache_clean_interval_sec 为 0（停止后台清理线程）。

• 减少并发压力：调整 compaction_thread_num 和 scanner_thread_pool_thread_num，降低线程竞争。

• 监控日志：通过 grep "SIGSEGV" be.INFO 持续收集崩溃上下文，缩小问题范围。

注意：若问题集中在删除操作时发生，可临时禁用数据删除功能（如避免 DELETE FROM 或分区删除），定位到具体修复方案后再重新启用。

通过上述措施，应能显著降低崩溃频率。建议优先排查缓存和删除处理模块的代码缺陷，这两个模块是当前崩溃的主因。

Dogbertmis13

建议把jdk升级到1.8.0_2xx版本试试

tagawa7
Related Questions
doris 偶发查询很慢
Spark读取Doris报错
1 answers
跪求大佬指导一下，doris be节点其中一个磁盘故障后更换新磁盘后无法重启be服务
1 answers
broker load大表导数较慢
doris报错被截断 doris版本2.1.6
1 answers
doris 2.1.5。有时候执行ddl 时间特别长
1 answers