版本：2.1.7 机器3台8C16G 作为BE
问题1.为什么这些加起来与VMRSS差的有点多
问题2.：目前BE宕机日志及相关信息
参数mem_limit=70%
日志：
I20250524 14:41:36.323154 6814 memory_reclamation.cpp:93] [MemoryGC] end full GC, free Memory 0. cost(us): 34177, details: :
FreeTopMemoryLoad:
- CancelCostTime: 67.000ns
- CancelTasksNum: 0
- FindCostTime: 893.593us
- FreedMemory: 0
- PreviouslyCancelingTasksNum: 1.03K
- SeekTasksNum: 1.03K
FreeTopMemoryOvercommitLoad:
FreeTopMemoryQuery:
- CancelCostTime: 48.000ns
- CancelTasksNum: 0
- FindCostTime: 20.669ms
- FreedMemory: 0
- PreviouslyCancelingTasksNum: 0
- SeekTasksNum: 0
E20250524 14:41:36.400938 6395 vexpr.cpp:397] Could not construct expr tree.
[MEM_ALLOC_FAILED]Create Expr failed because [E11] Allocator sys memory check failed: Cannot alloc:64, consuming tracker:<Load#Id=a06ff91265de43a0-a58f2cab844b0f72>, peak used 4096, current used 4096, exec node:<>, process memory used 10.89 GB exceed limit 10.85 GB or sys available memory 2.64 GB less than low water mark 793.80 MB.
Process Memory Summary:
os physical memory 15.50 GB. process memory used 10.91 GB(= 11.23 GB[vm/rss] - 329.25 MB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), limit 10.85 GB, soft limit 9.77 GB. sys available memory 2.63 GB(= 2.63 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), low water mark 793.80 MB, warning water mark 1.55 GB.
Memory Tracker Summary:
MemTrackerLimiter Label=other, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=4.61 KB(4720 B)
MemTrackerLimiter Label=schema_change, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=compaction, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=44.68 MB(46851250 B)
MemTrackerLimiter Label=load, Type=overview, Limit=-1.00 B(-1 B), Used=182.88 MB(191762432 B), Peak=186.12 MB(195158272 B)
MemTrackerLimiter Label=query, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=global, Type=overview, Limit=-1.00 B(-1 B), Used=14.47 MB(15169013 B), Peak=14.47 MB(15169013 B)
MemTrackerLimiter Label=tc/jemalloc_cache, Type=overview, Limit=-1.00 B(-1 B), Used=329.25 MB(345239424 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=tc/jemalloc_metadata, Type=overview, Limit=-1.00 B(-1 B), Used=372.14 MB(390214240 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=reserved_memory, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=sum_of_all_trackers, Type=overview, Limit=-1.00 B(-1 B), Used=898.73 MB(942385109 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=VmRSS(process resident memory), Type=overview, Limit=-1.00 B(-1 B), Used=11.23 GB(12054560768 B), Peak=13.24 GB(14212018176 B)
MemTrackerLimiter Label=VmSize(process virtual memory), Type=overview, Limit=-1.00 B(-1 B), Used=70.64 GB(75851808768 B), Peak=70.64 GB(75852075008 B)
MemTrackerLimiter Label=Orphan, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=DetailsTrackerSet, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTracker Label=PageNoCache, Parent Label=DetailsTrackerSet, Used=0(0 B), Peak=10.24 MB(10734217 B)
MemTracker Label=IOBufBlockMemory, Parent Label=DetailsTrackerSet, Used=21.49 MB(22536192 B), Peak=26.43 MB(27713536 B)
MemTracker Label=SegmentCache[size], Parent Label=DetailsTrackerSet, Used=3.11 MB(3262482 B), Peak=74.25 MB(77853151 B)
MemTracker Label=SchemaCache[number], Parent Label=DetailsTrackerSet, Used=1.38 KB(1414 B), Peak=1.38 KB(1414 B)
MemTracker Label=AllMemTableMemory, Parent Label=DetailsTrackerSet, Used=0(0 B), Peak=11.24 MB(11782349 B)
MemTracker Label=TabletSchemaCache[number], Parent Label=DetailsTrackerSet, Used=70.39 KB(72075 B), Peak=74.21 KB(75996 B)
MemTracker Label=TabletMeta(experimental), Parent Label=DetailsTrackerSet, Used=5.83 MB(6114688 B), Peak=5.87 MB(6158848 B)
MemTracker Label=MowTabletVersionCache[number], Parent Label=DetailsTrackerSet, Used=11.35 MB(11900000 B), Peak=11.35 MB(11900238 B)
MemTracker Label=MowDeleteBitmapAggCache[size], Parent Label=DetailsTrackerSet, Used=9.67 KB(9903 B), Peak=9.67 KB(9903 B)
MemTrackerLimiter Label=SegCompaction, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=PointQueryExecutor, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=BlockCompression, Type=global, Limit=-1.00 B(-1 B), Used=3.52 MB(3688501 B), Peak=4.27 MB(4474545 B)
MemTrackerLimiter Label=RowIdStorageReader, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=SubcolumnsTree, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=S3FileBuffer, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=DataPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=1.70 MB(1786910 B), Peak=1.70 MB(1786910 B)
MemTrackerLimiter Label=IndexPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=6.52 MB(6834582 B), Peak=6.58 MB(6903333 B)
MemTrackerLimiter Label=PKIndexPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=2.73 MB(2859020 B), Peak=2.73 MB(2859020 B)
MemTrackerLimiter Label=Load#Id=9240a03109654e17-906c94a1da752503, Type=load, Limit=2.00 GB(2147483648 B), Used=4.00 KB(4096 B), Peak=4.00 KB(4096 B)
MemTrackerLimiter Label=Load#Id=d2a3f57ed13c4568-a7649685cff68d90, Type=load, Limit=2.00 GB(2147483648 B), Used=4.00 KB(4096 B), Peak=4.00 KB(4096 B)
MemTrackerLimiter Label=Load#Id=dcbe99fe1d794dff-bcb84b9200befde5, Type=load, Limit=2.00 GB(2147483648 B), Used=4.00 KB(4096 B), Peak=4.00 KB(4096 B)
MemTrackerLimiter Label=Load#Id=ba048020f01c45c1-82370df2028ac985, Type=load, Limit=2.00 GB(2147483648 B), Used=4.00 KB(4096 B), Peak=4.00 KB(4096 B)
MemTrackerLimiter Label=Load#Id=db454e793e714a6c-967169aa70ec12e6, Type=load, Limit=2.00 GB(2147483648 B), Used=4.00 KB(4096 B), Peak=4.00 KB(4096 B)
MemTrackerLimiter Label=Load#Id=ea1d154b64bb42b1-ba281b9c1297e9b0, Type=load, Limit=2.00 GB(2147483648 B), Used=4.00 KB(4096 B), Peak=4.00 KB(4096 B)
MemTrackerLimiter Label=Load#Id=760917e2c6f4465f-9096391452792549, Type=load, Limit=2.00 GB(2147483648 B), Used=4.00 KB(4096 B), Peak=4.00 KB(4096 B)
MemTrackerLimiter Label=Load#Id=cbb46d3ef83a4aa5-aefbd9fd61df124b, Type=load, Limit=2.00 GB(2147483648 B), Used=4.00 KB(4096 B), Peak=4.00 KB(4096 B)
MemTrackerLimiter Label=Load#Id=abacc772c3734d50-8fb5828797bb8c81, Type=load, Limit=2.00 GB(2147483648 B), Used=4.00 KB(4096 B), Peak=4.00 KB(4096 B)
MemTrackerLimiter Label=Load#Id=c6a817fa98d24684-826aff904e8d7bf0, Type=load, Limit=2.00 GB(2147483648 B), Used=4.00 KB(4096 B), Peak=4.00 KB(4096 B)
MemTrackerLimiter Label=Load#Id=99bf12d988954cd1-b186cbf4446d6a5a, Type=load, Limit=2.00 GB(2147483648 B), Used=4.00 KB(4096 B), Peak=4.00 KB(4096 B)
MemTrackerLimiter Label=Load#Id=a67424b3434439-a5f3768ec2d5158a, Type=load, Limit=2.00 GB(2147483648 B), Used=4.00 KB(4096 B), Peak=4.00 KB(4096 B)
MemTrackerLimiter Label=Load#Id=b7cc718b4f3441fc-bb0f8cddcdce08c8, Type=load, Limit=2.00 GB(2147483648 B), Used=4.00 KB(4096 B), Peak=4.00 KB(4096 B)
MemTrackerLimiter Label=Load#Id=25623d26a1fd402e-bb37605057322a93, Type=load, Limit=2.00 GB(2147483648 B), Used=4.00 KB(4096 B), Peak=4.00 KB(4096 B)
MemTrackerLimiter Label=Load#Id=b31dbc345b9b4fef-b08b7149994f9295, Type=load, Limit=2.00 GB(2147483648 B), Used=4.00 KB(4096 B), Peak=4.00 KB(4096 B)
MemTracker Label=AllMemTableMemory, Parent Label=DetailsTrackerSet, Used=0(0 B), Peak=11.24 MB(11782349 B)
I20250524 14:40:50.259769 6247 allocator.cpp:125] Query:54698fb32a3d6ae-3d848783dd9dca9a throw exception, after waiting for memory 1000ms, Allocator sys memory check failed: Cannot alloc:128, consuming tracker:<(FromLoadChannel)Load#Id=054698fb32a3d6ae-3d848783dd9dca9a>, peak used 0, current used 0, exec node:<>, process memory used 11.13 GB exceed limit 10.85 GB or sys available memory 2.44 GB less than low water mark 793.80 MB..
W20250524 14:40:50.259955 6247 internal_service.cpp:472] tablet writer add block failed, message=[MEM_LIMIT_EXCEEDED]PreCatch error code:11, [E11] Allocator sys memory check failed: Cannot alloc:128, consuming tracker:<(FromLoadChannel)Load#Id=054698fb32a3d6ae-3d848783dd9dca9a>, peak used 0, current used 0, exec node:<>, process memory used 11.13 GB exceed limit 10.85 GB or sys available memory 2.44 GB less than low water mark 793.80 MB.
0# doris::Exception::Exception(int, std::basic_string_view<char, std::char_traits<char> > const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:173
1# Allocator<false, false, false, DefaultMemoryAllocator>::sys_memory_check(unsigned long) const at /home/zcp/repo_center/doris_release/doris/be/src/vec/common/allocator.cpp:129
2# Allocator<false, false, false, DefaultMemoryAllocator>::alloc_impl(unsigned long, unsigned long) at /home/zcp/repo_center/doris_release/doris/be/src/vec/common/allocator.cpp:187
3# doris::vectorized::DataTypeNullable::deserialize(char const*, doris::vectorized::IColumn*, int) const at /home/zcp/repo_center/doris_release/doris/be/src/vec/common/pod_array.h:143
4# doris::vectorized::Block::deserialize(doris::PBlock const&) at /home/zcp/repo_center/doris_release/doris/be/src/vec/core/block.cpp:0
5# doris::BaseTabletsChannel::add_batch(doris::PTabletWriterAddBlockRequest const&, doris::PTabletWriterAddBlockResult*) at /home/zcp/repo_center/doris_release/doris/be/src/runtime/tablets_channel.cpp:0
6# doris::LoadChannel::add_batch(doris::PTabletWriterAddBlockRequest const&, doris::PTabletWriterAddBlockResult*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
7# doris::LoadChannelMgr::add_batch(doris::PTabletWriterAddBlockRequest const&, doris::PTabletWriterAddBlockResult*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
8# std::_Function_handler<void (), doris::PInternalServiceImpl::tablet_writer_add_block(google::protobuf::RpcController*, doris::PTabletWriterAddBlockRequest const*, doris::PTabletWriterAddBlockResult*, google::protobuf::Closure*)::$_0>::_M_invoke(std::_Any_data const&) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
9# doris::WorkThreadPool<false>::work_thread(int) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:646
10# execute_native_thread_routine at /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/unique_ptr.h:85
11# start_thread
12# __clone
, FILE:/home/zcp/repo_center/doris_release/doris/be/src/vec/core/block.cpp, LINE:129, FUNCTION:Status doris::vectorized::Block::deserialize(const PBlock &), id=054698fb32a3d6ae-3d848783dd9dca9a, index_id=30298798, sender_id=0, backend id=10003
