Doris2.1.6升级为Doris2.1.11后全部be节点进程频繁停止运行

Viewed 11
2.1

Doris2.1.6升级为Doris2.1.11遇到大规模计算任务各个be节点频繁停止运行，高峰使用期间几乎达到了每3到5分钟be就要停止一次，并且产生了大量的gc日志，同样的场景之前的Doris2.1.6就没有出现这个问题，以下是其中一个节点be.out日志信息

截图20260915101328.png
StdoutLogger 2026-09-14 09:52:15,251 Start time: Mon Sep 14 09:52:15 CST 2026
INFO: java_cmd /opt/jdk8u504-b01/bin/java
INFO: jdk_version 8
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/ssd/doris216/doris/be/lib/java_extensions/preload-extensions/preload-extensions-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/ssd/doris216/doris/be/lib/java_extensions/java-udf/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/ssd/doris216/doris/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
*** Query id: d40f2c54d0924879-a7ed4eac341c14ba ***
*** is nereids: 1 ***
*** tablet id: 0 ***
*** Aborted at 1789350894 (unix time) try "date -d @1789350894" if you are using GNU date ***
*** Current BE git commitID: 97b77e6cda ***
*** SIGSEGV address not mapped to object (@0x55fcc107a928) received by PID 147000 (TID 150102 OR 0x7feaf970b700) from PID 18446744072653089064; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:421
1# os::Linux::chained_handler(int, siginfo*, void*) in /opt/jdk8u504-b01/jre/lib/amd64/server/libjvm.so
2# JVM_handle_linux_signal in /opt/jdk8u504-b01/jre/lib/amd64/server/libjvm.so
3# signalHandler(int, siginfo*, void*) in /opt/jdk8u504-b01/jre/lib/amd64/server/libjvm.so
4# 0x00007FF0DC386630 in /usr/lib64/libc.so.6
5# memcpy at /home/zcp/repo_center/doris_release/doris/be/src/glibc-compatibility/memcpy/memcpy_x86_64.cpp:219
6# auto doris::vectorized::ColumnStr::insert_indices_from(doris::vectorized::IColumn const&, unsigned int const*, unsigned int const*)::{lambda(auto:1 const&)#1}::operator()<doris::vectorized::ColumnStr >(doris::vectorized::ColumnStr const&) const at /home/zcp/repo_center/doris_release/doris/be/src/vec/columns/column_string.cpp:231
7# doris::vectorized::ColumnStr::insert_indices_from(doris::vectorized::IColumn const&, unsigned int const*, unsigned int const*) at /home/zcp/repo_center/doris_release/doris/be/src/vec/columns/column_string.cpp:242
8# doris::vectorized::ColumnNullable::insert_indices_from(doris::vectorized::IColumn const&, unsigned int const*, unsigned int const*) at /home/zcp/repo_center/doris_release/doris/be/src/vec/columns/column_nullable.cpp:309
9# doris::vectorized::ProcessHashTableProbe<7, doris::pipeline::HashJoinProbeLocalState>::build_side_output_column(std::vector<COW::mutable_ptr, std::allocator<COW::mutable_ptr > >&, std::vector<bool, std::allocator > const&, int, bool, bool) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/join/process_hash_table_probe_impl.h:119
10# doris::Status doris::vectorized::ProcessHashTableProbe<7, doris::pipeline::HashJoinProbeLocalState>::do_process<true, true, doris::vectorized::MethodSerialized<doris::JoinHashTable<doris::StringRef, DefaultHash<doris::StringRef, void> > >, false, true>(doris::vectorized::MethodSerialized<doris::JoinHashTable<doris::StringRef, DefaultHash<doris::StringRef, void> > >&, doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 16ul, 16ul> const*, doris::vectorized::MutableBlock&, doris::vectorized::Block*, unsigned long) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/join/process_hash_table_probe_impl.h:268
11# auto doris::vectorized::ProcessHashTableProbe<7, doris::pipeline::HashJoinProbeLocalState>::process<true, true, doris::vectorized::MethodSerialized<doris::JoinHashTable<doris::StringRef, DefaultHash<doris::StringRef, void> > > >(doris::vectorized::MethodSerialized<doris::JoinHashTable<doris::StringRef, DefaultHash<doris::StringRef, void> > >&, doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 16ul, 16ul> const*, doris::vectorized::MutableBlock&, doris::vectorized::Block*, unsigned long, bool, bool)::{lambda(auto:1, auto:2)#1}::operator()<std::integral_constant<bool, true>, std::integral_constant<bool, false> >(std::integral_constant<bool, true>, std::integral_constant<bool, false>) const at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/join/process_hash_table_probe_impl.h:735
12# doris::Status doris::vectorized::ProcessHashTableProbe<7, doris::pipeline::HashJoinProbeLocalState>::process<true, true, doris::vectorized::MethodSerialized<doris::JoinHashTable<doris::StringRef, DefaultHash<doris::StringRef, void> > > >(doris::vectorized::MethodSerialized<doris::JoinHashTable<doris::StringRef, DefaultHash<doris::StringRef, void> > >&, doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 16ul, 16ul> const*, doris::vectorized::MutableBlock&, doris::vectorized::Block*, unsigned long, bool, bool) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/join/process_hash_table_probe_impl.h:741
13# _ZNSt8__detail9__variant17__gen_vtable_implINS0_12_Multi_arrayIPFNS0_21__deduce_visit_resultIvEEOZNK5doris8pipeline22HashJoinProbeOperatorX4pullEPNS5_12RuntimeStateEPNS5_10vectorized5BlockEPbE3$0RSt7variantIJSt9monostateNSA_16MethodSerializedINS5_13JoinHashTableINS5_9StringRefE11DefaultHashISK_vEEEEENSA_15MethodOneNumberIhNSJ_Ih9HashCRC32IhEEEEENSP_ItNSJ_ItSQ_ItEEEEENSP_IjNSJ_IjSQ_IjEEEEENSP_ImNSJ_ImSQ_ImEEEEENSP_INSA_7UInt128ENSJ_IS13_SQ_IS13_EEEEENSP_INSA_7UInt256ENSJ_IS17_SQ_IS17_EEEEENSA_15MethodKeysFixedIS11_Lb1EEENS1B_IS11_Lb0EEENS1B_IS15_Lb1EEENS1B_IS15_Lb0EEENS1B_IS19_Lb1EEENS1B_IS19_Lb0EEENS1B_INSJ_INSA_7UInt136ESQ_IS1I_EEELb1EEENS1B_IS1K_Lb0EEEEERSG_IJSH_NSA_21ProcessHashTableProbeILi0ENS6_23HashJoinProbeLocalStateEEENS1P_ILi2ES1Q_EENS1P_ILi8ES1Q_EENS1P_ILi1ES1Q_EENS1P_ILi4ES1Q_EENS1P_ILi3ES1Q_EENS1P_ILi5ES1Q_EENS1P_ILi7ES1Q_EENS1P_ILi9ES1Q_EENS1P_ILi10ES1Q_EENS1P_ILi11ES1Q_EEEEOSG_IJSt17integral_constantIbLb0EES24_IbLb1EEEES28_EJEEESt16integer_sequenceImJLm1ELm8ELm1ELm1EEEE14__visit_invokeESF_S1O_S23_S28_S28 at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/variant:1013
14# doris::pipeline::HashJoinProbeOperatorX::pull(doris::RuntimeState*, doris::vectorized::Block*, bool*) const at /home/zcp/repo_center/doris_release/doris/be/src/pipeline/exec/hashjoin_probe_operator.cpp:313
15# doris::pipeline::StatefulOperatorX::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/pipeline/pipeline_x/operator.cpp:571
16# doris::pipeline::OperatorXBase::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/pipeline/pipeline_x/operator.cpp:288
17# doris::pipeline::PipelineXTask::execute(bool*) at /home/zcp/repo_center/doris_release/doris/be/src/pipeline/pipeline_x/pipeline_x_task.cpp:353
18# doris::pipeline::TaskScheduler::_do_work(unsigned long) at /home/zcp/repo_center/doris_release/doris/be/src/pipeline/task_scheduler.cpp:347
19# doris::ThreadPool::dispatch_thread() in /ssd/doris216/doris/be/lib/doris_be
20# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_release/doris/be/src/util/thread.cpp:499
21# 0x00007FF0DC190F2B in /usr/lib64/libpthread.so.0
22# __clone in /usr/lib64/libc.so.6

StdoutLogger 2026-09-14 09:55:15,252 Start time: Mon Sep 14 09:55:15 CST 2026
INFO: java_cmd /opt/jdk8u504-b01/bin/java
INFO: jdk_version 8
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/ssd/doris216/doris/be/lib/java_extensions/preload-extensions/preload-extensions-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/ssd/doris216/doris/be/lib/java_extensions/java-udf/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/ssd/doris216/doris/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
*** Query id: 7b14e4b78a3f4cc8-86e85090d27c0d40 ***
*** is nereids: 1 ***
*** tablet id: 0 ***
*** Aborted at 1789351034 (unix time) try "date -d @1789351034" if you are using GNU date ***
*** Current BE git commitID: 97b77e6cda ***
*** SIGSEGV invalid permissions for mapped object (@0x7efd40075000) received by PID 153724 (TID 157036 OR 0x7ef77494a700) from PID 1074221056; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:421
1# os::Linux::chained_handler(int, siginfo*, void*) in /opt/jdk8u504-b01/jre/lib/amd64/server/libjvm.so
2# JVM_handle_linux_signal in /opt/jdk8u504-b01/jre/lib/amd64/server/libjvm.so
3# signalHandler(int, siginfo*, void*) in /opt/jdk8u504-b01/jre/lib/amd64/server/libjvm.so
4# 0x00007EFD73E40630 in /usr/lib64/libc.so.6
5# memcpy at /home/zcp/repo_center/doris_release/doris/be/src/glibc-compatibility/memcpy/memcpy_x86_64.cpp:219
6# auto doris::vectorized::ColumnStr::insert_indices_from(doris::vectorized::IColumn const&, unsigned int const*, unsigned int const*)::{lambda(auto:1 const&)#1}::operator()<doris::vectorized::ColumnStr >(doris::vectorized::ColumnStr const&) const at /home/zcp/repo_center/doris_release/doris/be/src/vec/columns/column_string.cpp:231
7# doris::vectorized::ColumnStr::insert_indices_from(doris::vectorized::IColumn const&, unsigned int const*, unsigned int const*) at /home/zcp/repo_center/doris_release/doris/be/src/vec/columns/column_string.cpp:242
8# doris::vectorized::ColumnNullable::insert_indices_from(doris::vectorized::IColumn const&, unsigned int const*, unsigned int const*) at /home/zcp/repo_center/doris_release/doris/be/src/vec/columns/column_nullable.cpp:309
9# doris::vectorized::ProcessHashTableProbe<7, doris::pipeline::HashJoinProbeLocalState>::build_side_output_column(std::vector<COW::mutable_ptr, std::allocator<COW::mutable_ptr > >&, std::vector<bool, std::allocator > const&, int, bool, bool) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/join/process_hash_table_probe_impl.h:119
10# doris::Status doris::vectorized::ProcessHashTableProbe<7, doris::pipeline::HashJoinProbeLocalState>::do_process<true, true, doris::vectorized::MethodSerialized<doris::JoinHashTable<doris::StringRef, DefaultHash<doris::StringRef, void> > >, false, true>(doris::vectorized::MethodSerialized<doris::JoinHashTable<doris::StringRef, DefaultHash<doris::StringRef, void> > >&, doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 16ul, 16ul> const*, doris::vectorized::MutableBlock&, doris::vectorized::Block*, unsigned long) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/join/process_hash_table_probe_impl.h:268
11# auto doris::vectorized::ProcessHashTableProbe<7, doris::pipeline::HashJoinProbeLocalState>::process<true, true, doris::vectorized::MethodSerialized<doris::JoinHashTable<doris::StringRef, DefaultHash<doris::StringRef, void> > > >(doris::vectorized::MethodSerialized<doris::JoinHashTable<doris::StringRef, DefaultHash<doris::StringRef, void> > >&, doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 16ul, 16ul> const*, doris::vectorized::MutableBlock&, doris::vectorized::Block*, unsigned long, bool, bool)::{lambda(auto:1, auto:2)#1}::operator()<std::integral_constant<bool, true>, std::integral_constant<bool, false> >(std::integral_constant<bool, true>, std::integral_constant<bool, false>) const at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/join/process_hash_table_probe_impl.h:735
12# doris::Status doris::vectorized::ProcessHashTableProbe<7, doris::pipeline::HashJoinProbeLocalState>::process<true, true, doris::vectorized::MethodSerialized<doris::JoinHashTable<doris::StringRef, DefaultHash<doris::StringRef, void> > > >(doris::vectorized::MethodSerialized<doris::JoinHashTable<doris::StringRef, DefaultHash<doris::StringRef, void> > >&, doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 16ul, 16ul> const*, doris::vectorized::MutableBlock&, doris::vectorized::Block*, unsigned long, bool, bool) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/join/process_hash_table_probe_impl.h:741
13# _ZNSt8__detail9__variant17__gen_vtable_implINS0_12_Multi_arrayIPFNS0_21__deduce_visit_resultIvEEOZNK5doris8pipeline22HashJoinProbeOperatorX4pullEPNS5_12RuntimeStateEPNS5_10vectorized5BlockEPbE3$0RSt7variantIJSt9monostateNSA_16MethodSerializedINS5_13JoinHashTableINS5_9StringRefE11DefaultHashISK_vEEEEENSA_15MethodOneNumberIhNSJ_Ih9HashCRC32IhEEEEENSP_ItNSJ_ItSQ_ItEEEEENSP_IjNSJ_IjSQ_IjEEEEENSP_ImNSJ_ImSQ_ImEEEEENSP_INSA_7UInt128ENSJ_IS13_SQ_IS13_EEEEENSP_INSA_7UInt256ENSJ_IS17_SQ_IS17_EEEEENSA_15MethodKeysFixedIS11_Lb1EEENS1B_IS11_Lb0EEENS1B_IS15_Lb1EEENS1B_IS15_Lb0EEENS1B_IS19_Lb1EEENS1B_IS19_Lb0EEENS1B_INSJ_INSA_7UInt136ESQ_IS1I_EEELb1EEENS1B_IS1K_Lb0EEEEERSG_IJSH_NSA_21ProcessHashTableProbeILi0ENS6_23HashJoinProbeLocalStateEEENS1P_ILi2ES1Q_EENS1P_ILi8ES1Q_EENS1P_ILi1ES1Q_EENS1P_ILi4ES1Q_EENS1P_ILi3ES1Q_EENS1P_ILi5ES1Q_EENS1P_ILi7ES1Q_EENS1P_ILi9ES1Q_EENS1P_ILi10ES1Q_EENS1P_ILi11ES1Q_EEEEOSG_IJSt17integral_constantIbLb0EES24_IbLb1EEEES28_EJEEESt16integer_sequenceImJLm1ELm8ELm1ELm1EEEE14__visit_invokeESF_S1O_S23_S28_S28 at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/variant:1013
14# doris::pipeline::HashJoinProbeOperatorX::pull(doris::RuntimeState*, doris::vectorized::Block*, bool*) const at /home/zcp/repo_center/doris_release/doris/be/src/pipeline/exec/hashjoin_probe_operator.cpp:313
15# doris::pipeline::StatefulOperatorX::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/pipeline/pipeline_x/operator.cpp:571
16# doris::pipeline::OperatorXBase::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/pipeline/pipeline_x/operator.cpp:288
17# doris::pipeline::PipelineXTask::execute(bool*) at /home/zcp/repo_center/doris_release/doris/be/src/pipeline/pipeline_x/pipeline_x_task.cpp:353
18# doris::pipeline::TaskScheduler::_do_work(unsigned long) at /home/zcp/repo_center/doris_release/doris/be/src/pipeline/task_scheduler.cpp:347
19# doris::ThreadPool::dispatch_thread() in /ssd/doris216/doris/be/lib/doris_be
20# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_release/doris/be/src/util/thread.cpp:499
21# 0x00007EFD73C4AF2B in /usr/lib64/libpthread.so.0
22# __clone in /usr/lib64/libc.so.6

StdoutLogger 2026-09-14 09:57:30,264 Start time: Mon Sep 14 09:57:30 CST 2026
INFO: java_cmd /opt/jdk8u504-b01/bin/java
INFO: jdk_version 8
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/ssd/doris216/doris/be/lib/java_extensions/preload-extensions/preload-extensions-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/ssd/doris216/doris/be/lib/java_extensions/java-udf/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/ssd/doris216/doris/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
StdoutLogger 2026-09-14 10:00:34,274 Start time: Mon Sep 14 10:00:34 CST 2026
INFO: java_cmd /opt/jdk8u504-b01/bin/java
INFO: jdk_version 8
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/ssd/doris216/doris/be/lib/java_extensions/preload-extensions/preload-extensions-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/ssd/doris216/doris/be/lib/java_extensions/java-udf/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/ssd/doris216/doris/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
*** Query id: cf76726131a14c5a-8359a431d34299d6 ***
*** is nereids: 1 ***
*** tablet id: 0 ***
*** Aborted at 1789351346 (unix time) try "date -d @1789351346" if you are using GNU date ***
*** Current BE git commitID: 97b77e6cda ***
*** SIGSEGV address not mapped to object (@0x5600e9c96000) received by PID 166316 (TID 169464 OR 0x7f34378c3700) from PID 18446744073336872960; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:421
1# os::Linux::chained_handler(int, siginfo*, void*) in /opt/jdk8u504-b01/jre/lib/amd64/server/libjvm.so
2# JVM_handle_linux_signal in /opt/jdk8u504-b01/jre/lib/amd64/server/libjvm.so
3# signalHandler(int, siginfo*, void*) in /opt/jdk8u504-b01/jre/lib/amd64/server/libjvm.so
4# 0x00007F3A41A65630 in /usr/lib64/libc.so.6
5# memcpy at /home/zcp/repo_center/doris_release/doris/be/src/glibc-compatibility/memcpy/memcpy_x86_64.cpp:219
6# auto doris::vectorized::ColumnStr::insert_indices_from(doris::vectorized::IColumn const&, unsigned int const*, unsigned int const*)::{lambda(auto:1 const&)#1}::operator()<doris::vectorized::ColumnStr >(doris::vectorized::ColumnStr const&) const at /home/zcp/repo_center/doris_release/doris/be/src/vec/columns/column_string.cpp:231
7# doris::vectorized::ColumnStr::insert_indices_from(doris::vectorized::IColumn const&, unsigned int const*, unsigned int const*) at /home/zcp/repo_center/doris_release/doris/be/src/vec/columns/column_string.cpp:242
8# doris::vectorized::ColumnNullable::insert_indices_from(doris::vectorized::IColumn const&, unsigned int const*, unsigned int const*) at /home/zcp/repo_center/doris_release/doris/be/src/vec/columns/column_nullable.cpp:309
9# doris::vectorized::ProcessHashTableProbe<7, doris::pipeline::HashJoinProbeLocalState>::build_side_output_column(std::vector<COW::mutable_ptr, std::allocator<COW::mutable_ptr > >&, std::vector<bool, std::allocator > const&, int, bool, bool) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/join/process_hash_table_probe_impl.h:119
10# doris::Status doris::vectorized::ProcessHashTableProbe<7, doris::pipeline::HashJoinProbeLocalState>::do_process<true, true, doris::vectorized::MethodSerialized<doris::JoinHashTable<doris::StringRef, DefaultHash<doris::StringRef, void> > >, false, true>(doris::vectorized::MethodSerialized<doris::JoinHashTable<doris::StringRef, DefaultHash<doris::StringRef, void> > >&, doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 16ul, 16ul> const*, doris::vectorized::MutableBlock&, doris::vectorized::Block*, unsigned long) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/join/process_hash_table_probe_impl.h:268
11# auto doris::vectorized::ProcessHashTableProbe<7, doris::pipeline::HashJoinProbeLocalState>::process<true, true, doris::vectorized::MethodSerialized<doris::JoinHashTable<doris::StringRef, DefaultHash<doris::StringRef, void> > > >(doris::vectorized::MethodSerialized<doris::JoinHashTable<doris::StringRef, DefaultHash<doris::StringRef, void> > >&, doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 16ul, 16ul> const*, doris::vectorized::MutableBlock&, doris::vectorized::Block*, unsigned long, bool, bool)::{lambda(auto:1, auto:2)#1}::operator()<std::integral_constant<bool, true>, std::integral_constant<bool, false> >(std::integral_constant<bool, true>, std::integral_constant<bool, false>) const at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/join/process_hash_table_probe_impl.h:735
12# doris::Status doris::vectorized::ProcessHashTableProbe<7, doris::pipeline::HashJoinProbeLocalState>::process<true, true, doris::vectorized::MethodSerialized<doris::JoinHashTable<doris::StringRef, DefaultHash<doris::StringRef, void> > > >(doris::vectorized::MethodSerialized<doris::JoinHashTable<doris::StringRef, DefaultHash<doris::StringRef, void> > >&, doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 16ul, 16ul> const*, doris::vectorized::MutableBlock&, doris::vectorized::Block*, unsigned long, bool, bool) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/join/process_hash_table_probe_impl.h:741
13# _ZNSt8__detail9__variant17__gen_vtable_implINS0_12_Multi_arrayIPFNS0_21__deduce_visit_resultIvEEOZNK5doris8pipeline22HashJoinProbeOperatorX4pullEPNS5_12RuntimeStateEPNS5_10vectorized5BlockEPbE3$0RSt7variantIJSt9monostateNSA_16MethodSerializedINS5_13JoinHashTableINS5_9StringRefE11DefaultHashISK_vEEEEENSA_15MethodOneNumberIhNSJ_Ih9HashCRC32IhEEEEENSP_ItNSJ_ItSQ_ItEEEEENSP_IjNSJ_IjSQ_IjEEEEENSP_ImNSJ_ImSQ_ImEEEEENSP_INSA_7UInt128ENSJ_IS13_SQ_IS13_EEEEENSP_INSA_7UInt256ENSJ_IS17_SQ_IS17_EEEEENSA_15MethodKeysFixedIS11_Lb1EEENS1B_IS11_Lb0EEENS1B_IS15_Lb1EEENS1B_IS15_Lb0EEENS1B_IS19_Lb1EEENS1B_IS19_Lb0EEENS1B_INSJ_INSA_7UInt136ESQ_IS1I_EEELb1EEENS1B_IS1K_Lb0EEEEERSG_IJSH_NSA_21ProcessHashTableProbeILi0ENS6_23HashJoinProbeLocalStateEEENS1P_ILi2ES1Q_EENS1P_ILi8ES1Q_EENS1P_ILi1ES1Q_EENS1P_ILi4ES1Q_EENS1P_ILi3ES1Q_EENS1P_ILi5ES1Q_EENS1P_ILi7ES1Q_EENS1P_ILi9ES1Q_EENS1P_ILi10ES1Q_EENS1P_ILi11ES1Q_EEEEOSG_IJSt17integral_constantIbLb0EES24_IbLb1EEEES28_EJEEESt16integer_sequenceImJLm1ELm8ELm1ELm1EEEE14__visit_invokeESF_S1O_S23_S28_S28 at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/variant:1013
14# doris::pipeline::HashJoinProbeOperatorX::pull(doris::RuntimeState*, doris::vectorized::Block*, bool*) const at /home/zcp/repo_center/doris_release/doris/be/src/pipeline/exec/hashjoin_probe_operator.cpp:313
15# doris::pipeline::StatefulOperatorX::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/pipeline/pipeline_x/operator.cpp:571
16# doris::pipeline::OperatorXBase::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/pipeline/pipeline_x/operator.cpp:288
17# doris::pipeline::PipelineXTask::execute(bool*) at /home/zcp/repo_center/doris_release/doris/be/src/pipeline/pipeline_x/pipeline_x_task.cpp:353
18# doris::pipeline::TaskScheduler::_do_work(unsigned long) at /home/zcp/repo_center/doris_release/doris/be/src/pipeline/task_scheduler.cpp:347
19# doris::ThreadPool::dispatch_thread() in /ssd/doris216/doris/be/lib/doris_be
20# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_release/doris/be/src/util/thread.cpp:499
21# 0x00007F3A4186FF2B in /usr/lib64/libpthread.so.0
22# __clone in /usr/lib64/libc.so.6

以下是gc日志信息

OpenJDK 64-Bit Server VM (25.504-b01) for linux-amd64 JRE (1.8.0_504-b01), built on Aug 19 2026 15:02:07 by "jenkins" with gcc 7.5.0
Memory: 4k page, physical 262700152k(193577508k free), swap 0k(0k free)
CommandLine flags: -XX:CompressedClassSpaceSize=1065353216 -XX:-CriticalJNINatives -XX:InitialHeapSize=4294967296 -XX:MaxGCPauseMillis=200 -XX:MaxHeapSize=8589934592 -XX:MaxMetaspaceSize=1073741824 -XX:MetaspaceSize=536870912 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC
2026-09-14T11:08:01.286+0800: 1.690: [GC pause (G1 Evacuation Pause) (young), 0.1095943 secs]
[Parallel Time: 69.5 ms, GC Workers: 53]
[GC Worker Start (ms): Min: 1690.1, Avg: 1691.7, Max: 1693.1, Diff: 2.9]
[Ext Root Scanning (ms): Min: 0.0, Avg: 0.4, Max: 11.1, Diff: 11.1, Sum: 19.2]
[Update RS (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
[Processed Buffers: Min: 0, Avg: 0.0, Max: 0, Diff: 0, Sum: 0]
[Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.5]
[Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.9, Diff: 0.9, Sum: 2.2]
[Object Copy (ms): Min: 29.2, Avg: 38.6, Max: 44.2, Diff: 15.0, Sum: 2045.6]
[Termination (ms): Min: 23.8, Avg: 26.4, Max: 29.0, Diff: 5.2, Sum: 1398.0]
[Termination Attempts: Min: 155, Avg: 191.9, Max: 254, Diff: 99, Sum: 10171]
[GC Worker Other (ms): Min: 0.1, Avg: 2.1, Max: 10.5, Diff: 10.5, Sum: 112.9]
[GC Worker Total (ms): Min: 66.1, Avg: 67.5, Max: 69.2, Diff: 3.1, Sum: 3578.6]
[GC Worker End (ms): Min: 1759.1, Avg: 1759.2, Max: 1759.5, Diff: 0.4]
[Code Root Fixup: 0.1 ms]
[Code Root Purge: 0.0 ms]
[Clear CT: 1.3 ms]
[Other: 38.7 ms]
[Choose CSet: 0.0 ms]
[Ref Proc: 36.2 ms]
[Ref Enq: 0.1 ms]
[Redirty Cards: 1.1 ms]
[Humongous Register: 0.1 ms]
[Humongous Reclaim: 0.0 ms]
[Free CSet: 0.6 ms]
[Eden: 204.0M(204.0M)->0.0B(184.0M) Survivors: 0.0B->20480.0K Heap: 204.0M(4096.0M)->19226.3K(4096.0M)]
[Times: user=3.30 sys=0.12, real=0.11 secs]
2026-09-14T11:08:01.716+0800: 2.120: [GC pause (G1 Evacuation Pause) (young), 0.0331705 secs]
[Parallel Time: 27.7 ms, GC Workers: 53]
[GC Worker Start (ms): Min: 2120.3, Avg: 2120.7, Max: 2121.2, Diff: 0.9]
[Ext Root Scanning (ms): Min: 2.3, Avg: 3.2, Max: 26.7, Diff: 24.4, Sum: 169.1]
[Update RS (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.2]
[Processed Buffers: Min: 0, Avg: 0.0, Max: 0, Diff: 0, Sum: 0]
[Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.2, Diff: 0.2, Sum: 1.9]
[Code Root Scanning (ms): Min: 0.0, Avg: 0.1, Max: 6.2, Diff: 6.2, Sum: 6.2]
[Object Copy (ms): Min: 0.1, Avg: 15.9, Max: 16.9, Diff: 16.8, Sum: 845.0]
[Termination (ms): Min: 0.1, Avg: 7.2, Max: 7.8, Diff: 7.7, Sum: 382.4]
[Termination Attempts: Min: 1, Avg: 21.2, Max: 30, Diff: 29, Sum: 1122]
[GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.3, Sum: 6.1]
[GC Worker Total (ms): Min: 26.1, Avg: 26.6, Max: 27.1, Diff: 1.0, Sum: 1410.9]
[GC Worker End (ms): Min: 2147.3, Avg: 2147.4, Max: 2147.5, Diff: 0.3]
[Code Root Fixup: 0.0 ms]
[Code Root Purge: 0.1 ms]
[Clear CT: 0.9 ms]
[Other: 4.5 ms]
[Choose CSet: 0.0 ms]
[Ref Proc: 1.3 ms]
[Ref Enq: 0.0 ms]
[Redirty Cards: 0.9 ms]
[Humongous Register: 0.6 ms]
[Humongous Reclaim: 0.0 ms]
[Free CSet: 1.4 ms]
[Eden: 184.0M(184.0M)->0.0B(248.0M) Survivors: 20480.0K->10240.0K Heap: 202.8M(4096.0M)->9460.8K(4096.0M)]
[Times: user=1.13 sys=0.25, real=0.03 secs]
2026-09-14T11:08:02.068+0800: 2.472: [GC pause (G1 Evacuation Pause) (young), 0.0330332 secs]
[Parallel Time: 28.6 ms, GC Workers: 53]
[GC Worker Start (ms): Min: 2471.7, Avg: 2472.1, Max: 2472.6, Diff: 0.9]
[Ext Root Scanning (ms): Min: 0.0, Avg: 0.4, Max: 8.8, Diff: 8.8, Sum: 22.5]
[Update RS (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
[Processed Buffers: Min: 0, Avg: 0.0, Max: 0, Diff: 0, Sum: 0]
[Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.2, Diff: 0.2, Sum: 1.3]
[Code Root Scanning (ms): Min: 0.0, Avg: 0.1, Max: 1.3, Diff: 1.3, Sum: 5.0]
[Object Copy (ms): Min: 19.1, Avg: 26.0, Max: 27.0, Diff: 7.9, Sum: 1379.3]
[Termination (ms): Min: 0.1, Avg: 0.3, Max: 0.4, Diff: 0.3, Sum: 17.1]
[Termination Attempts: Min: 1, Avg: 44.5, Max: 60, Diff: 59, Sum: 2358]
[GC Worker Other (ms): Min: 0.0, Avg: 0.9, Max: 4.5, Diff: 4.5, Sum: 47.7]
[GC Worker Total (ms): Min: 27.3, Avg: 27.8, Max: 28.3, Diff: 1.0, Sum: 1473.1]
[GC Worker End (ms): Min: 2499.8, Avg: 2499.9, Max: 2500.1, Diff: 0.3]
[Code Root Fixup: 0.0 ms]
[Code Root Purge: 0.0 ms]
[Clear CT: 0.9 ms]
[Other: 3.5 ms]
[Choose CSet: 0.0 ms]
[Ref Proc: 1.6 ms]
[Ref Enq: 0.0 ms]
[Redirty Cards: 1.1 ms]
[Humongous Register: 0.1 ms]
[Humongous Reclaim: 0.0 ms]
[Free CSet: 0.5 ms]
[Eden: 248.0M(248.0M)->0.0B(2450.0M) Survivors: 10240.0K->6144.0K Heap: 257.2M(4096.0M)->5640.9K(4096.0M)]
[Times: user=1.40 sys=0.04, real=0.04 secs]

1 Answers

从栈来分析，是一个fix的问题，是RIGHT_SEMI mark join的bug ，这个问题在 4 版本有修复。

  1. 可以通过query_id在fe.audit.log 中先定位一下是哪个SQL，然后找到这个SQL后直接执行看看BE是否还core。如果是这个SQL，你尝试先执行：
    set enable_fallback_to_original_planner = true;
    set enable_nereids_planner = false;

然后再执行这个SQL，看看是否还core，如果不core了，那就给这个SQL单独 hint 一下这个两个变量。如果还core 那就只能升级了，4版本这块彻底fix了。

阿渊@SelectDB (没回帖直接加我主页微信)10485
Related Questions
Doris2.1.6升级为Doris2.1.11后全部be节点进程频繁停止运行
1 answers
Huawei MRS使用doris，tpc-ds测试运行load data至analyse时报错
doris2.1.11版本漏洞
1 answers
with as加insert into报错，解析器解析不到insert into后面的内容
1 answers
FE的JVM每天以1%的频率增长，原因是什么：Doris2.1.6
1 answers
Doris 2.1.6版本升级至4.1.3版本详细升级路径疑问。
1 answers