Doris2.1.4 be节点宕机,jemalloc相关问题

Viewed 3

doris 2.1.4版本集群,be 71个节点,64核 256G内存,Arm架构,最近出现be节点突然宕机,当时的query很多,且be.out没有有效的queryId信息,coredump文件未记录,以下是be.out,请各位大佬帮忙看下。

*** Query id: 0-0 ***
*** is nereids: 0 ***
*** tablet id: 0 ***
*** Aborted at 177673490 (unix time) try "date -d @177673490" if you are using GNU date ***
*** Current BE git commitID: 6ff0573991 ***
***
*** SIGSEGV address not mapped to object (@0x100036615ba78) received by PID 397266 (TID 397785 OR 0xffffb605d9890) from PID 1712700024; stack trace
: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_enterprise/doris/be/src/
common/signal_handler.h:421
1# os::Linux::chained_handler(int, siginfo_t*, void*) in /app/doris/be/java8/jre/lib/aarch64/server/libjvm.so
2# JVM_handle_linux_signal in /app/doris/be/java8/jre/lib/aarch64/server/libjvm.so
3# signalHandler(int, siginfo_t*, void*) in /app/doris/be/java8/jre/lib/aarch64/server/libjvm.so
4# 0x0000FFFCDA0207C0 in linux-vdso.so.1
5# 0x0000FFFCD9E777D0 in /usr/lib64/libc.so.6
6# je_tcache_bin_flush_small at ../src/tcache.c:529
7# tcache_gc_small at ../src/tcache.c:155
8# je_tcache_gc_dalloc_event_handler at ../src/tcache.c:173
9# je_event_trigger at ../src/thread_event.c:299
10# je_free_default at ../src/jemalloc.c:3026
11# doris::vectorized::ColumnDecimal<doris::vectorized::Decimal128V3>::~ColumnDecimal() at /home/zcp/repo_center/doris_enterprise/doris/be/src/vec
/columns/column_decimal.h:87
12# doris::vectorized::ColumnNullable::~ColumnNullable() at /home/zcp/repo_center/doris_enterprise/doris/be/src/vec/columns/column_nullable.h:62
13# std::vector<doris::vectorized::ColumnWithTypeAndName, std::allocator<doris::vectorized::ColumnWithTypeAndName> >::~vector() at /usr/local/bin/
ldb-toolchain/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:680
14# doris::pipeline::PipelineTask::~PipelineTask() at /home/zcp/repo_center/doris_enterprise/doris/be/src/pipeline/pipeline_task.h:128
15# doris::pipeline::PipelineXTask::~PipelineXTask() at /home/zcp/repo_center/doris_enterprise/doris/be/src/pipeline/pipeline_x/pipeline_x_task.h:
52
16# doris::pipeline::PipelineXFragmentContext::~PipelineXFragmentContext() at /home/zcp/repo_center/doris_enterprise/doris/be/src/pipeline/
pipeline_x/pipeline_x_fragment_context.cpp:117
17# std::_Function_handler<void (), doris::FragmentMgr::trigger_pipeline_context_report(doris::ReportStatusRequest, std::shared_ptr<doris::
pipeline::PipelineFragmentContext>&&)::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) at /usr/local/bin/ldb-
toolchain/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:283
18# doris::FunctionRunnable::~FunctionRunnable() at /home/zcp/repo_center/doris_enterprise/doris/be/src/util/threadpool.cpp:44
19# doris::ThreadPool::dispatch_thread() at /home/zcp/repo_center/doris_enterprise/doris/be/src/util/threadpool.cpp:551
20# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_enterprise/doris/be/src/util/thread.cpp:499
21# 0x0000FFFCD9C787AC in /usr/lib64/libpthread.so.0

0 Answers