doris_be崩溃退出

Viewed 16

doris.out内容:
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
*** Query id: 5db1db454aa340ff-89310adb1c313552 ***
*** is nereids: 1 ***
*** tablet id: 0 ***
*** Aborted at 1749520572 (unix time) try "date -d @1749520572" if you are using GNU date ***
*** Current BE git commitID: 443e87e203 ***
*** SIGABRT unknown detail explain (@0x1fe00008f92) received by PID 36754 (TID 39627 OR 0x7fb52c6bd700) from PID 36754; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:421
1# 0x00007FBBBD8B4400 in /lib64/libc.so.6
2# gsignal in /lib64/libc.so.6
3# abort in /lib64/libc.so.6
4# __gnu_cxx::__verbose_terminate_handler() [clone .cold] at ../../../../libstdc++-v3/libsupc++/vterminate.cc:75
5# __cxxabiv1::__terminate(void ()()) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48
6# 0x0000558AE44869C1 in /data/public/apache-doris-2.1.7-bin-x64/be/lib/doris_be
7# 0x0000558AE4486B14 in /data/public/apache-doris-2.1.7-bin-x64/be/lib/doris_be
8# 0x0000558AE4486F06 in /data/public/apache-doris-2.1.7-bin-x64/be/lib/doris_be
9# phmap::priv::raw_hash_set<phmap::priv::FlatHashSetPolicy, phmap::Hash, phmap::EqualTo, std::allocator >::resize(unsigned long) at /home/zcp/repo_center/doris_release/doris/thirdparty/installed/include/parallel_hashmap/phmap.h:2014
10# phmap::priv::raw_hash_set<phmap::priv::FlatHashSetPolicy, phmap::Hash, phmap::EqualTo, std::allocator >::raw_hash_set(phmap::priv::raw_hash_set<phmap::priv::FlatHashSetPolicy, phmap::Hash, phmap::EqualTo, std::allocator > const&, std::allocator const&) at /home/zcp/repo_center/doris_release/doris/thirdparty/installed/include/parallel_hashmap/phmap.h:1167
11# doris::BitmapValue::operator=(doris::BitmapValue const&) at /home/zcp/repo_center/doris_release/doris/be/src/util/bitmap_value.h:902
12# doris::vectorized::AggregateFunctionBitmapSerializationHelper<doris::vectorized::AggregateFunctionBitmapData, doris::vectorized::AggregateFunctionBitmapCount<false, doris::vectorized::ColumnComplexType > >::deserialize_and_merge_from_column_range(char
, doris::vectorized::IColumn const&, unsigned long, unsigned long, doris::vectorized::Arena*) const at /home/zcp/repo_center/doris_release/doris/be/src/vec/aggregate_functions/aggregate_function_bitmap.h:217
13# doris::vectorized::AggregateStateUnion::add_batch(unsigned long, char**, unsigned long, doris::vectorized::IColumn const**, doris::vectorized::Arena*, bool) const in /data/public/apache-doris-2.1.7-bin-x64/be/lib/doris_be
14# doris::vectorized::AggFnEvaluator::execute_batch_add(doris::vectorized::Block*, unsigned long, char**, doris::vectorized::Arena*, bool) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exprs/vectorized_agg_fn.cpp:242
15# doris::pipeline::StreamingAggLocalState::_pre_agg_with_serialized_key(doris::vectorized::Block*, doris::vectorized::Block*) at /home/zcp/repo_center/doris_release/doris/be/src/pipeline/exec/streaming_aggregation_operator.cpp:751
16# doris::pipeline::StreamingAggLocalState::do_pre_agg(doris::vectorized::Block*, doris::vectorized::Block*) at /home/zcp/repo_center/doris_release/doris/be/src/pipeline/exec/streaming_aggregation_operator.cpp:553
17# doris::pipeline::StreamingAggOperatorX::push(doris::RuntimeState*, doris::vectorized::Block*, bool) const at /home/zcp/repo_center/doris_release/doris/be/src/pipeline/exec/streaming_aggregation_operator.cpp:1315
18# doris::pipeline::StatefulOperatorX::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/pipeline/pipeline_x/operator.cpp:564
19# doris::pipeline::OperatorXBase::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/pipeline/pipeline_x/operator.cpp:293
20# doris::pipeline::PipelineXTask::execute(bool*) at /home/zcp/repo_center/doris_release/doris/be/src/pipeline/pipeline_x/pipeline_x_task.cpp:346
21# doris::pipeline::TaskScheduler::_do_work(unsigned long) at /home/zcp/repo_center/doris_release/doris/be/src/pipeline/task_scheduler.cpp:347
22# doris::ThreadPool::dispatch_thread() in /data/public/apache-doris-2.1.7-bin-x64/be/lib/doris_be
23# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_release/doris/be/src/util/thread.cpp:499
24# start_thread in /lib64/libpthread.so.0
25# clone in /lib64/libc.so.6

/var/log/message内容:
Jun 10 09:56:14 localhost abrt-hook-ccpp: Process 36754 (doris_be) of user 510 killed by SIGABRT - dumping core
Jun 10 09:56:19 localhost systemd-logind: Removed session 63493.
Jun 10 09:56:19 localhost abrt-server: Executable '/data/public/apache-doris-2.1.7-bin-x64/be/lib/doris_be' doesn't belong to any package and ProcessUnpackaged is set to 'no'
Jun 10 09:56:19 localhost abrt-server: 'post-create' on '/var/spool/abrt/ccpp-2025-06-10-09:56:14-36754' exited with 1
Jun 10 09:56:19 localhost abrt-server: Deleting problem directory '/var/spool/abrt/ccpp-2025-06-10-09:56:14-36754'

doris INFO 日志:
I20250610 09:56:12.400739 38814 pipeline_x_fragment_context.cpp:207] PipelineXFragmentContext::prepare|query_id=8f8ac0ebc49c4b66-8f291fcc5a35fc68|fragment_id=0|pthread_id=140427187910400
I20250610 09:56:12.400794 38815 fragment_mgr.cpp:778] query_id: 63da8596bada41f3-9843d7de2a413650, coord_addr: TNetworkAddress(hostname=10.103.251.122, port=9020), total fragment num on current host: 1, fe process uuid: 1749207689428, query type: SELECT, report audit fe:TNetworkAddress(hostname=10.103.251.122, port=9020)
I20250610 09:56:12.400894 38815 fragment_mgr.cpp:819] Query/load id: 63da8596bada41f3-9843d7de2a413650, use workload group: TG[id = 1, name = normal, cpu_share = 1024, memory_limit = 135.91 GB, enable_memory_overcommit = true, version = 0, cpu_hard_limit = -1, scan_thread_num = 192, max_remote_scan_thread_num = 960, min_remote_scan_thread_num = 8, spill_low_watermark=50, spill_high_watermark=80, is_shutdown=false, query_num=6, read_bytes_per_second=-1, remote_read_bytes_per_second=-1], is pipeline: 1
I20250610 09:56:12.400918 38815 fragment_mgr.cpp:830] Register query/load memory tracker, query/load id: 63da8596bada41f3-9843d7de2a413650 limit: 0
I20250610 09:56:12.400931 38815 pipeline_x_fragment_context.cpp:207] PipelineXFragmentContext::prepare|query_id=63da8596bada41f3-9843d7de2a413650|fragment_id=0|pthread_id=140427179517696
I20250610 09:56:12.404233 39625 fragment_mgr.cpp:730] Removing query 92058bbeea1443fa-a2ccb5f67afe9ec7 instance 92058bbeea1443fa-a2ccb5f67afe9ec9, all done? false
I20250610 09:56:12.404428 39638 fragment_mgr.cpp:730] Removing query 63da8596bada41f3-9843d7de2a413650 instance 63da8596bada41f3-9843d7de2a413652, all done? true
I20250610 09:56:12.404465 39638 fragment_mgr.cpp:736] Query 63da8596bada41f3-9843d7de2a413650 finished
I20250610 09:56:12.404481 39640 fragment_mgr.cpp:730] Removing query 8f8ac0ebc49c4b66-8f291fcc5a35fc68 instance 8f8ac0ebc49c4b66-8f291fcc5a35fc6a, all done? true
I20250610 09:56:12.404506 39640 fragment_mgr.cpp:736] Query 8f8ac0ebc49c4b66-8f291fcc5a35fc68 finished
I20250610 09:56:12.404610 39638 query_context.cpp:156] Query 63da8596bada41f3-9843d7de2a413650 deconstructed, , deregister query/load memory tracker, queryId=63da8596bada41f3-9843d7de2a413650, Limit=2.00 GB, CurrUsed=428.44 KB, PeakUsed=796.44 KB
I20250610 09:56:12.404633 39638 query_context.cpp:188] Query 63da8596bada41f3-9843d7de2a413650 deconstructed, , deregister query/load memory tracker, queryId=63da8596bada41f3-9843d7de2a413650, Limit=2.00 GB, CurrUsed=428.44 KB, PeakUsed=796.44 KB
I20250610 09:56:12.404677 39640 query_context.cpp:156] Query 8f8ac0ebc49c4b66-8f291fcc5a35fc68 deconstructed, , deregister query/load memory tracker, queryId=8f8ac0ebc49c4b66-8f291fcc5a35fc68, Limit=2.00 GB, CurrUsed=428.44 KB, PeakUsed=796.44 KB
I20250610 09:56:12.404735 39640 query_context.cpp:188] Query 8f8ac0ebc49c4b66-8f291fcc5a35fc68 deconstructed, , deregister query/load memory tracker, queryId=8f8ac0ebc49c4b66-8f291fcc5a35fc68, Limit=2.00 GB, CurrUsed=428.44 KB, PeakUsed=796.44 KB
I20250610 09:56:12.404830 39644 fragment_mgr.cpp:730] Removing query efd439305a0c4dd1-93f57584548b386b instance efd439305a0c4dd1-93f57584548b386d, all done? true
I20250610 09:56:12.404848 39644 fragment_mgr.cpp:736] Query efd439305a0c4dd1-93f57584548b386b finished
I20250610 09:56:12.405032 39644 query_context.cpp:156] Query efd439305a0c4dd1-93f57584548b386b deconstructed, , deregister query/load memory tracker, queryId=efd439305a0c4dd1-93f57584548b386b, Limit=2.00 GB, CurrUsed=440.50 KB, PeakUsed=808.50 KB
I20250610 09:56:12.405056 39644 query_context.cpp:188] Query efd439305a0c4dd1-93f57584548b386b deconstructed, , deregister query/load memory tracker, queryId=efd439305a0c4dd1-93f57584548b386b, Limit=2.00 GB, CurrUsed=440.50 KB, PeakUsed=808.50 KB
I20250610 09:56:12.405181 39645 result_sink_operator.cpp:202] Query 92058bbeea1443fa-a2ccb5f67afe9ec7 result sink closed with status [OK] and has written 0 rows
I20250610 09:56:12.405272 39645 fragment_mgr.cpp:730] Removing query 92058bbeea1443fa-a2ccb5f67afe9ec7 instance 92058bbeea1443fa-a2ccb5f67afe9ecb, all done? true
I20250610 09:56:12.405282 39645 fragment_mgr.cpp:736] Query 92058bbeea1443fa-a2ccb5f67afe9ec7 finished
I20250610 09:56:12.405346 39645 query_context.cpp:156] Query 92058bbeea1443fa-a2ccb5f67afe9ec7 deconstructed, , deregister query/load memory tracker, queryId=92058bbeea1443fa-a2ccb5f67afe9ec7, Limit=2.00 GB, CurrUsed=4.00 KB, PeakUsed=796.50 KB
I20250610 09:56:12.405373 39645 query_context.cpp:188] Query 92058bbeea1443fa-a2ccb5f67afe9ec7 deconstructed, , deregister query/load memory tracker, queryId=92058bbeea1443fa-a2ccb5f67afe9ec7, Limit=2.00 GB, CurrUsed=4.00 KB, PeakUsed=796.50 KB
E20250610 09:56:12.411161 39633 bitmap_value.h:1963] BitmapTypeCode invalid, should between: 0 and 4 actual is 110
E20250610 09:56:12.411231 39633 bitmap_value.h:1963] BitmapTypeCode invalid, should between: 0 and 4 actual is -48
E20250610 09:56:12.412282 39624 bitmap_value.h:1963] BitmapTypeCode invalid, should between: 0 and 4 actual is 27
E20250610 09:56:12.413187 39567 bitmap_value.h:1963] BitmapTypeCode invalid, should between: 0 and 4 actual is 110
E20250610 09:56:12.413887 39609 bitmap_value.h:1963] BitmapTypeCode invalid, should between: 0 and 4 actual is 90
E20250610 09:56:12.414870 39581 bitmap_value.h:1963] BitmapTypeCode invalid, should between: 0 and 4 actual is 52
I20250610 09:56:12.813264 39383 workload_group_manager.cpp:192]
Process Memory Summary: process memory used 3.01 GB(= 3.66 GB[vm/rss] - 666.14 MB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), sys available memory 474.24 GB(= 474.24 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), all workload groups memory usage: 52.25 MB, weighted_memory_limit_ratio: 0.9934731936356963
I20250610 09:56:12.912709 38823 fragment_mgr.cpp:778] query_id: 227a4fab6f984bdd-9dcb8788e798f07a, coord_addr: TNetworkAddress(hostname=10.103.251.122, port=9020), total fragment num on current host: 0, fe process uuid: 0, query type: LOAD, report audit fe:TNetworkAddress(hostname=10.103.251.122, port=9020)
I20250610 09:56:12.912801 38823 fragment_mgr.cpp:830] Register query/load memory tracker, query/load id: 227a4fab6f984bdd-9dcb8788e798f07a limit: 0
I20250610 09:56:12.912814 38823 plan_fragment_executor.cpp:121] PlanFragmentExecutor::prepare|query_id=227a4fab6f984bdd-9dcb8788e798f07a|instance_id=227a4fab6f984bdd-9dcb8788e798f07b|backend_num=0|pthread_id=140427112376064
I20250610 09:56:12.915344 37321 plan_fragment_executor.cpp:255] PlanFragmentExecutor::open 227a4fab6f984bdd-9dcb8788e798f07a|227a4fab6f984bdd-9dcb8788e798f07b, mem_limit 2.00 GB
I20250610 09:56:12.920535 37321 exec_node.cpp:247] query= 227a4fab6f984bdd-9dcb8788e798f07a, fragment_instance_id=227a4fab6f984bdd-9dcb8788e798f07b, id=0 type=FILE_SCAN_NODE closed
I20250610 09:56:12.920558 37321 fragment_mgr.cpp:612] Instance 227a4fab6f984bdd-9dcb8788e798f07b finished
I20250610 09:56:12.920567 37321 fragment_mgr.cpp:616] Query 227a4fab6f984bdd-9dcb8788e798f07a finished
I20250610 09:56:12.920701 37321 query_context.cpp:156] Query 227a4fab6f984bdd-9dcb8788e798f07a deconstructed, , deregister query/load memory tracker, queryId=227a4fab6f984bdd-9dcb8788e798f07a, Limit=2.00 GB, CurrUsed=2.03 MB, PeakUsed=4.42 MB
I20250610 09:56:12.920734 37321 query_context.cpp:188] Query 227a4fab6f984bdd-9dcb8788e798f07a deconstructed, , deregister query/load memory tracker, queryId=227a4fab6f984bdd-9dcb8788e798f07a, Limit=2.00 GB, CurrUsed=2.03 MB, PeakUsed=4.42 MB
I20250610 09:56:13.508411 38822 fragment_mgr.cpp:778] query_id: 4589759cfab04f27-93eb989339b6deff, coord_addr: TNetworkAddress(hostname=10.103.251.122, port=9020), total fragment num on current host: 0, fe process uuid: 0, query type: LOAD, report audit fe:TNetworkAddress(hostname=10.103.251.122, port=9020)
I20250610 09:56:13.508483 38822 fragment_mgr.cpp:830] Register query/load memory tracker, query/load id: 4589759cfab04f27-93eb989339b6deff limit: 0
I20250610 09:56:13.508495 38822 plan_fragment_executor.cpp:121] PlanFragmentExecutor::prepare|query_id=4589759cfab04f27-93eb989339b6deff|instance_id=4589759cfab04f27-93eb989339b6df00|backend_num=0|pthread_id=140427120768768
I20250610 09:56:13.509407 37308 plan_fragment_executor.cpp:255] PlanFragmentExecutor::open 4589759cfab04f27-93eb989339b6deff|4589759cfab04f27-93eb989339b6df00, mem_limit 2.00 GB
I20250610 09:56:13.538205 37308 exec_node.cpp:247] query= 4589759cfab04f27-93eb989339b6deff, fragment_instance_id=4589759cfab04f27-93eb989339b6df00, id=0 type=FILE_SCAN_NODE closed
I20250610 09:56:13.538241 37308 fragment_mgr.cpp:612] Instance 4589759cfab04f27-93eb989339b6df00 finished
I20250610 09:56:13.538249 37308 fragment_mgr.cpp:616] Query 4589759cfab04f27-93eb989339b6deff finished
I20250610 09:56:13.538372 37308 query_context.cpp:156] Query 4589759cfab04f27-93eb989339b6deff deconstructed, , deregister query/load memory tracker, queryId=4589759cfab04f27-93eb989339b6deff, Limit=2.00 GB, CurrUsed=8.03 MB, PeakUsed=16.42 MB
I20250610 09:56:13.538395 37308 query_context.cpp:188] Query 4589759cfab04f27-93eb989339b6deff deconstructed, , deregister query/load memory tracker, queryId=4589759cfab04f27-93eb989339b6deff, Limit=2.00 GB, CurrUsed=8.03 MB, PeakUsed=16.42 MB
I20250610 09:56:13.547281 38821 fragment_mgr.cpp:778] query_id: 30bf410c1678482c-a192ab82fc7c00d9, coord_addr: TNetworkAddress(hostname=10.103.251.122, port=9020), total fragment num on current host: 0, fe process uuid: 0, query type: LOAD, report audit fe:TNetworkAddress(hostname=10.103.251.122, port=9020)
I20250610 09:56:13.547374 38821 fragment_mgr.cpp:830] Register query/load memory tracker, query/load id: 30bf410c1678482c-a192ab82fc7c00d9 limit: 0
I20250610 09:56:13.547386 38821 plan_fragment_executor.cpp:121] PlanFragmentExecutor::prepare|query_id=30bf410c1678482c-a192ab82fc7c00d9|instance_id=30bf410c1678482c-a192ab82fc7c00da|backend_num=0|pthread_id=140427129161472
I20250610 09:56:13.548539 37288 plan_fragment_executor.cpp:255] PlanFragmentExecutor::open 30bf410c1678482c-a192ab82fc7c00d9|30bf410c1678482c-a192ab82fc7c00da, mem_limit 2.00 GB
I20250610 09:56:13.549615 37288 exec_node.cpp:247] query= 30bf410c1678482c-a192ab82fc7c00d9, fragment_instance_id=30bf410c1678482c-a192ab82fc7c00da, id=0 type=FILE_SCAN_NODE closed
I20250610 09:56:13.549638 37288 fragment_mgr.cpp:612] Instance 30bf410c1678482c-a192ab82fc7c00da finished
I20250610 09:56:13.549644 37288 fragment_mgr.cpp:616] Query 30bf410c1678482c-a192ab82fc7c00d9 finished
I20250610 09:56:13.549726 37288 query_context.cpp:156] Query 30bf410c1678482c-a192ab82fc7c00d9 deconstructed, , deregister query/load memory tracker, queryId=30bf410c1678482c-a192ab82fc7c00d9, Limit=2.00 GB, CurrUsed=23.44 KB, PeakUsed=683.25 KB
I20250610 09:56:13.549752 37288 query_context.cpp:188] Query 30bf410c1678482c-a192ab82fc7c00d9 deconstructed, , deregister query/load memory tracker, queryId=30bf410c1678482c-a192ab82fc7c00d9, Limit=2.00 GB, CurrUsed=23.44 KB, PeakUsed=683.25 KB
I20250610 09:56:13.678644 38825 fragment_mgr.cpp:778] query_id: 3c977276c2384883-abb33d1b8a259fe3, coord_addr: TNetworkAddress(hostname=10.103.251.122, port=9020), total fragment num on current host: 0, fe process uuid: 0, query type: LOAD, report audit fe:TNetworkAddress(hostname=10.103.251.122, port=9020)
I20250610 09:56:13.678714 38825 fragment_mgr.cpp:830] Register query/load memory tracker, query/load id: 3c977276c2384883-abb33d1b8a259fe3 limit: 0
I20250610 09:56:13.678725 38825 plan_fragment_executor.cpp:121] PlanFragmentExecutor::prepare|query_id=3c977276c2384883-abb33d1b8a259fe3|instance_id=3c977276c2384883-abb33d1b8a259fe4|backend_num=0|pthread_id=140427095590656
I20250610 09:56:13.679508 37281 plan_fragment_executor.cpp:255] PlanFragmentExecutor::open 3c977276c2384883-abb33d1b8a259fe3|3c977276c2384883-abb33d1b8a259fe4, mem_limit 2.00 GB
I20250610 09:56:13.680763 37281 exec_node.cpp:247] query= 3c977276c2384883-abb33d1b8a259fe3, fragment_instance_id=3c977276c2384883-abb33d1b8a259fe4, id=0 type=FILE_SCAN_NODE closed
I20250610 09:56:13.680789 37281 fragment_mgr.cpp:612] Instance 3c977276c2384883-abb33d1b8a259fe4 finished
I20250610 09:56:13.680804 37281 fragment_mgr.cpp:616] Query 3c977276c2384883-abb33d1b8a259fe3 finished
I20250610 09:56:13.680897 37281 query_context.cpp:156] Query 3c977276c2384883-abb33d1b8a259fe3 deconstructed, , deregister query/load memory tracker, queryId=3c977276c2384883-abb33d1b8a259fe3, Limit=2.00 GB, CurrUsed=28.88 KB, PeakUsed=687.44 KB
I20250610 09:56:13.680922 37281 query_context.cpp:188] Query 3c977276c2384883-abb33d1b8a259fe3 deconstructed, , deregister query/load memory tracker, queryId=3c977276c2384883-abb33d1b8a259fe3, Limit=2.00 GB, CurrUsed=28.88 KB, PeakUsed=687.44 KB
I20250610 09:56:13.701715 39376 daemon.cpp:221] os physical memory 503.38 GB. process memory used 3.23 GB(= 3.95 GB[vm/rss] - 733.92 MB[tc/jemalloc_cache] + 0[reserved]

  • 0B[waiting_refresh]), limit 453.04 GB, soft limit 407.74 GB. sys available memory 474.05 GB(= 474.05 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), low water mark 6.40 GB, warning water mark 12.80 GB.
    I20250610 09:56:14.096771 38824 fragment_mgr.cpp:778] query_id: 4c4c0a4fd0494ca2-9c7d8a4a31b35af3, coord_addr: TNetworkAddress(hostname=10.103.251.122, port=9020), total fragment num on current host: 0, fe process uuid: 0, query type: LOAD, report audit fe:TNetworkAddress(hostname=10.103.251.122, port=9020)
    I20250610 09:56:14.096899 38824 fragment_mgr.cpp:830] Register query/load memory tracker, query/load id: 4c4c0a4fd0494ca2-9c7d8a4a31b35af3 limit: 0
    I20250610 09:56:14.096937 38824 plan_fragment_executor.cpp:121] PlanFragmentExecutor::prepare|query_id=4c4c0a4fd0494ca2-9c7d8a4a31b35af3|instance_id=4c4c0a4fd0494ca2-9c7d8a4a31b35af4|backend_num=0|pthread_id=140427103983360
    I20250610 09:56:14.098327 37289 plan_fragment_executor.cpp:255] PlanFragmentExecutor::open 4c4c0a4fd0494ca2-9c7d8a4a31b35af3|4c4c0a4fd0494ca2-9c7d8a4a31b35af4, mem_limit 2.00 GB
    I20250610 09:56:14.099458 37289 exec_node.cpp:247] query= 4c4c0a4fd0494ca2-9c7d8a4a31b35af3, fragment_instance_id=4c4c0a4fd0494ca2-9c7d8a4a31b35af4, id=0 type=FILE_SCAN_NODE closed
    I20250610 09:56:14.099480 37289 fragment_mgr.cpp:612] Instance 4c4c0a4fd0494ca2-9c7d8a4a31b35af4 finished
    I20250610 09:56:14.099488 37289 fragment_mgr.cpp:616] Query 4c4c0a4fd0494ca2-9c7d8a4a31b35af3 finished
    I20250610 09:56:14.099587 37289 query_context.cpp:156] Query 4c4c0a4fd0494ca2-9c7d8a4a31b35af3 deconstructed, , deregister query/load memory tracker, queryId=4c4c0a4fd0494ca2-9c7d8a4a31b35af3, Limit=2.00 GB, CurrUsed=26.69 KB, PeakUsed=686.25 KB
    I20250610 09:56:14.099611 37289 query_context.cpp:188] Query 4c4c0a4fd0494ca2-9c7d8a4a31b35af3 deconstructed, , deregister query/load memory tracker, queryId=4c4c0a4fd0494ca2-9c7d8a4a31b35af3, Limit=2.00 GB, CurrUsed=26.69 KB, PeakUsed=686.25 KB
    I20250610 09:56:14.211488 38828 fragment_mgr.cpp:778] query_id: 53a632dfb861410e-95e5cd90f5556828, coord_addr: TNetworkAddress(hostname=10.103.251.122, port=9020), total fragment num on current host: 0, fe process uuid: 0, query type: LOAD, report audit fe:TNetworkAddress(hostname=10.103.251.122, port=9020)
    I20250610 09:56:14.211566 38828 fragment_mgr.cpp:830] Register query/load memory tracker, query/load id: 53a632dfb861410e-95e5cd90f5556828 limit: 0
    I20250610 09:56:14.211578 38828 plan_fragment_executor.cpp:121] PlanFragmentExecutor::prepare|query_id=53a632dfb861410e-95e5cd90f5556828|instance_id=53a632dfb861410e-95e5cd90f5556829|backend_num=0|pthread_id=140427070412544
    I20250610 09:56:14.212698 37303 plan_fragment_executor.cpp:255] PlanFragmentExecutor::open 53a632dfb861410e-95e5cd90f5556828|53a632dfb861410e-95e5cd90f5556829, mem_limit 2.00 GB
    I20250610 09:56:14.213987 37303 exec_node.cpp:247] query= 53a632dfb861410e-95e5cd90f5556828, fragment_instance_id=53a632dfb861410e-95e5cd90f5556829, id=0 type=FILE_SCAN_NODE closed
    I20250610 09:56:14.214007 37303 fragment_mgr.cpp:612] Instance 53a632dfb861410e-95e5cd90f5556829 finished
    I20250610 09:56:14.214013 37303 fragment_mgr.cpp:616] Query 53a632dfb861410e-95e5cd90f5556828 finished
    I20250610 09:56:14.214107 37303 query_context.cpp:156] Query 53a632dfb861410e-95e5cd90f5556828 deconstructed, , deregister query/load memory tracker, queryId=53a632dfb861410e-95e5cd90f5556828, Limit=2.00 GB, CurrUsed=154.75 KB, PeakUsed=813.44 KB
    I20250610 09:56:14.214133 37303 query_context.cpp:188] Query 53a632dfb861410e-95e5cd90f5556828 deconstructed, , deregister query/load memory tracker, queryId=53a632dfb861410e-95e5cd90f5556828, Limit=2.00 GB, CurrUsed=154.75 KB, PeakUsed=813.44 KB
    I20250610 09:56:14.305810 38827 fragment_mgr.cpp:778] query_id: a037ff175b9047ba-b570fbd34d5d68fc, coord_addr: TNetworkAddress(hostname=10.103.251.122, port=9020), total fragment num on current host: 0, fe process uuid: 0, query type: LOAD, report audit fe:TNetworkAddress(hostname=10.103.251.122, port=9020)
    I20250610 09:56:14.305877 38827 fragment_mgr.cpp:830] Register query/load memory tracker, query/load id: a037ff175b9047ba-b570fbd34d5d68fc limit: 0
    I20250610 09:56:14.305888 38827 plan_fragment_executor.cpp:121] PlanFragmentExecutor::prepare|query_id=a037ff175b9047ba-b570fbd34d5d68fc|instance_id=a037ff175b9047ba-b570fbd34d5d68fd|backend_num=0|pthread_id=140427078805248

I20250616 09:53:23.980819 4454 doris_main.cpp:382] version doris-2.1.7-rc03(AVX2) RELEASE (build git://vm-36@443e87e20327eaa5577cc10f08a63ec1694de358)
Built on Wed, 06 Nov 2024 15:34:46 CST by vm-36


doris warning日志:
50737, tablet_state=TABLET_NOTREADY, version=1437
W20250610 09:54:51.766556 38209 engine_publish_version_task.cpp:330] publish version failed on transaction, tablet version not exists. transaction_id=564610, tablet_id=750741, tablet_state=TABLET_NOTREADY, version=1437
W20250610 09:54:51.766562 38209 engine_publish_version_task.cpp:330] publish version failed on transaction, tablet version not exists. transaction_id=564610, tablet_id=750745, tablet_state=TABLET_NOTREADY, version=1437
W20250610 09:54:51.766568 38209 engine_publish_version_task.cpp:330] publish version failed on transaction, tablet version not exists. transaction_id=564610, tablet_id=750749, tablet_state=TABLET_NOTREADY, version=1437
W20250610 09:54:51.766577 38209 engine_publish_version_task.cpp:330] publish version failed on transaction, tablet version not exists. transaction_id=564610, tablet_id=750753, tablet_state=TABLET_NOTREADY, version=1437
W20250610 09:54:51.766582 38209 engine_publish_version_task.cpp:330] publish version failed on transaction, tablet version not exists. transaction_id=564610, tablet_id=750757, tablet_state=TABLET_NOTREADY, version=1437
W20250610 09:54:51.766589 38209 engine_publish_version_task.cpp:330] publish version failed on transaction, tablet version not exists. transaction_id=564610, tablet_id=750761, tablet_state=TABLET_NOTREADY, version=1437
W20250610 09:54:51.766595 38209 engine_publish_version_task.cpp:330] publish version failed on transaction, tablet version not exists. transaction_id=564610, tablet_id=750765, tablet_state=TABLET_NOTREADY, version=1437
W20250610 09:54:51.766602 38209 engine_publish_version_task.cpp:330] publish version failed on transaction, tablet version not exists. transaction_id=564610, tablet_id=750769, tablet_state=TABLET_NOTREADY, version=1437
W20250610 09:54:51.766608 38209 engine_publish_version_task.cpp:330] publish version failed on transaction, tablet version not exists. transaction_id=564610, tablet_id=750773, tablet_state=TABLET_NOTREADY, version=1437
E20250610 09:56:12.411161 39633 bitmap_value.h:1963] BitmapTypeCode invalid, should between: 0 and 4 actual is 110
E20250610 09:56:12.411231 39633 bitmap_value.h:1963] BitmapTypeCode invalid, should between: 0 and 4 actual is -48
E20250610 09:56:12.412282 39624 bitmap_value.h:1963] BitmapTypeCode invalid, should between: 0 and 4 actual is 27
E20250610 09:56:12.413187 39567 bitmap_value.h:1963] BitmapTypeCode invalid, should between: 0 and 4 actual is 110
E20250610 09:56:12.413887 39609 bitmap_value.h:1963] BitmapTypeCode invalid, should between: 0 and 4 actual is 90
E20250610 09:56:12.414870 39581 bitmap_value.h:1963] BitmapTypeCode invalid, should between: 0 and 4 actual is 52
W20250616 09:53:25.714778 4454 timezone_utils.cpp:94] Meet illegal tzdata file: iso3166.tab. skipped
W20250616 09:53:25.715139 4454 timezone_utils.cpp:94] Meet illegal tzdata file: leapseconds. skipped

doris fe audit_log内容:
[ifly@observe-storage01 log]$ cat fe.audit.log.20250610-1 | grep 5db1db454aa340ff-89310adb1c313552
2025-06-10 09:56:36,997 [query] |Client=127.0.0.1:47762|User=admin|Ctl=internal|Db=monitor_analysis|State=ERR|ErrorCode=1105|ErrorMessage=errCode = 2, detailMessage = Backend Backend [id=10170, host=10.103.251.126, heartbeatPort=9050, alive=false, lastStartTime=2025-06-06 19:53:42, process epoch=1749210822402, tags: {location=default}] not exists or dead, query 5db1db454aa340ff-89310adb1c313552 should be cancelled|Time(ms)=18298|ScanBytes=6585280|ScanRows=427|ReturnRows=0|StmtId=1305536|QueryId=5db1db454aa340ff-89310adb1c313552|IsQuery=true|IsNereids=true|FeIp=10.103.251.122|Stmt=select \n Date(uploadTime) as dt, \n projectName, \n bitmap_union_count(to_bitmap(userId)) \nfrom \n frontend_monitor_analysis \ngroup by \n Date(uploadTime), projectName;|CpuTimeMS=948|ShuffleSendBytes=20710|ShuffleSendRows=233|SqlHash=9d193d77130b1845b044cf465bbc3879|PeakMemoryBytes=78027112|SqlDigest=|WorkloadGroup=normal|FuzzyVariables=
2025-06-10 09:56:36,997 [slow_query] |Client=127.0.0.1:47762|User=admin|Ctl=internal|Db=monitor_analysis|State=ERR|ErrorCode=1105|ErrorMessage=errCode = 2, detailMessage = Backend Backend [id=10170, host=10.103.251.126, heartbeatPort=9050, alive=false, lastStartTime=2025-06-06 19:53:42, process epoch=1749210822402, tags: {location=default}] not exists or dead, query 5db1db454aa340ff-89310adb1c313552 should be cancelled|Time(ms)=18298|ScanBytes=6585280|ScanRows=427|ReturnRows=0|StmtId=1305536|QueryId=5db1db454aa340ff-89310adb1c313552|IsQuery=true|IsNereids=true|FeIp=10.103.251.122|Stmt=select \n Date(uploadTime) as dt, \n projectName, \n bitmap_union_count(to_bitmap(userId)) \nfrom \n frontend_monitor_analysis \ngroup by \n Date(uploadTime), projectName;|CpuTimeMS=948|ShuffleSendBytes=20710|ShuffleSendRows=233|SqlHash=9d193d77130b1845b044cf465bbc3879|PeakMemoryBytes=78027112|SqlDigest=|WorkloadGroup=normal|FuzzyVariables=

烦请大佬帮忙看下是什么原因导致的be异常退出,doris版本2.1.7。这个sql我看在后面还有查询,感觉不像是这个SQL的问题

1 Answers

cat /proc/sys/vm/overcommit_memory看下是否是 0

需要将所有的机器改成1
echo 1 | sudo tee /proc/sys/vm/overcommit_memory