Doris版本V3.0.5，目前遇到了BE内存持续上涨到不足的问题；其中UntrackedMemory占比达90%，并且minor GC无法回收内存，也没有触发full GC，请教下各位老师如何定位这种问题。下面是一段Memory Snapshot
W20260415 14:25:32.021510 1271504 memory_profile.cpp:339] Process Memory Summary: os physical memory 62.66 GB. process memory used 44.55 GB(= 45.06 GB[vm/rss] - 523.34 MB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), limit 56.39 GB, soft limit 50.75 GB. sys available memory 6.21 GB(= 6.21 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), low water mark 3.13 GB, warning water mark 6.27 GB.
W20260415 14:25:32.021541 1271504 memory_profile.cpp:340]
MemoryOverviewSnapshot:
- PhysicalMemory(VmRSS) Current: 44.89 GB (Peak: 45.50 GB)
- VirtualMemory(VmSize) Current: 183.94 GB (Peak: 185.36 GB)
UntrackedMemory:
- Memory Current: 92.61 GB (Peak: 95.00 GB)
TrackedMemory:
- Memory Current: -51236730571.00 B (Peak: 21.17 GB)
TasksMemory:
- Memory Current: 1.96 GB (Peak: 28.79 GB)
- ReservedMemory Current: 0 (Peak: 0)
Details:
- Compaction Current: 0 (Peak: 28.79 GB)
- Load Current: 0 (Peak: 2.49 GB)
- AllMemTablesMemory Current: 0 (Peak: 2.55 GB)
- Other Current: 0 (Peak: 60.56 KB)
- Query Current: 1.96 GB (Peak: 9.59 GB)
- SchemaChange Current: 0 (Peak: 1.09 MB)
GlobalMemory:
- Memory Current: 148.97 MB (Peak: 1.77 GB)
MetadataMemory:
- Memory Current: -56934229315.00 B (Peak: 991.45 MB)
CacheMemory:
- Memory Current: 1.98 GB (Peak: 12.74 GB)
JemallocMemory:
- Memory Current: 1.22 GB (Peak: 20.76 GB)
Details:
- Cache Current: 427.99 MB (Peak: 20.26 GB)
- Metadata Current: 818.89 MB (Peak: 819.93 MB)
W20260415 14:25:32.021625 1271504 memory_profile.cpp:341]
GlobalMemorySnapshot:
Orphan@global@id=294283006b1d7c2f-48475f8320949ea0:
- Memory Current: 0 (Peak: 0)
IOBufBlockMemory@global@id=374cb565fc9839dd-e63e3ca789d0dfa6:
- Memory Current: 40.11 MB (Peak: 508.68 MB)
PointQueryExecutor@global@id=af47f01d9d8abf35-c9ce91004b16bda1:
- Memory Current: 0 (Peak: 0)
BlockCompression@global@id=e54d45bef7a0dd5d-4dce259b46a330a2:
- Memory Current: 108.09 MB (Peak: 110.81 MB)
RowIdStorageReader@global@id=924f6c93169a02f9-6a4762a5dee7bdad:
- Memory Current: 0 (Peak: 313.24 MB)
SubcolumnsTree@global@id=594213f2ec7275c0-d2f517e996b61e9c:
- Memory Current: 792.00 KB (Peak: 33.12 MB)
S3FileBuffer@global@id=0c49273ff8e79aa1-5d32e17fbbf2468c:
- Memory Current: 0 (Peak: 1.79 GB)
W20260415 14:25:32.021654 1271504 memory_profile.cpp:342]
MetadataMemorySnapshot:
Tablets(not in SchemaCache, TabletSchemaCache)@metadata@id=1540ace0dafeb842-3687b8db6e51f1ad:
- Memory Current: -57704767696.00 B (Peak: 140.78 MB)
Segments(not in SegmentCache)@metadata@id=784227393705249c-e30eec975ed75e9b:
- Memory Current: 57.13 MB (Peak: 1.90 GB)
Rowsets@metadata@id=d84b593c1749588b-4ebc7eb25b26d8b0:
- Memory Current: 26.49 MB (Peak: 245.70 MB)
SegmentCache[size]@metadata@id=fe48b988295bb010-aaaedeffdf3c3eaa:
- Memory Current: 646.98 MB (Peak: 2.56 GB)
SchemaCache[number]@metadata@id=15441bcfac803f85-53a0dc0b64900da3:
- Memory Current: 308.85 KB (Peak: 1.35 MB)
TabletSchemaCache[number]@metadata@id=a84e26071f753efd-f9c4658595191489:
- Memory Current: 3.94 MB (Peak: 6.31 MB)
W20260415 14:25:32.021682 1271504 memory_profile.cpp:343]
CacheMemorySnapshot:
QueryCache@cache@id=6d45405fa2e5d037-8d9ff16896728cbc:
- Memory Current: 0 (Peak: 0)
DataPageCache[size]@cache@id=a94eab9491a6906d-cbd6d8eafccac9a1:
- Memory Current: 1.74 GB (Peak: 10.16 GB)
IndexPageCache[size]@cache@id=00484024e1e1ecbc-868f3fc3457a228a:
- Memory Current: 138.50 MB (Peak: 1.53 GB)
PKIndexPageCache[size]@cache@id=b645f8b5999aca33-4fb6d5f1e69ea28f:
- Memory Current: 18.27 MB (Peak: 711.37 MB)
PointQueryRowCache[size]@cache@id=b04a57fa21752fc8-ebfec4eadfe19498:
- Memory Current: 0 (Peak: 0)
CommonObjLRUCache[number]@cache@id=554e92186354fa97-ac39728d5b6d889b:
- Memory Current: 0 (Peak: 0)
PointQueryLookupConnectionCache[number]@cache@id=054f5f0625da75d6-042e69e5ef16a983:
- Memory Current: 0 (Peak: 0)
InvertedIndexSearcherCache[size]@cache@id=2a4087f68d2abafd-5a8e5a2f4f1acc9a:
- Memory Current: 9.65 MB (Peak: 1.56 GB)
InvertedIndexQueryCache[size]@cache@id=9d4cd9505afa90bb-458c439db9f7d5ac:
- Memory Current: 57.04 MB (Peak: 67.49 MB)
QueryCache[size]@cache@id=a64af03ca506d8df-d267e4f02293ee9a:
- Memory Current: 0 (Peak: 0)
LastSuccessChannelCache[size]@cache@id=394a24573abf9d02-9e3101f5948118b3:
- Memory Current: 0 (Peak: 0)
TabletColumnObjectPool[number]@cache@id=a84ebaf1bad5418b-20c108f998781591:
- Memory Current: 3.68 MB (Peak: 3.68 MB)
MowTabletVersionCache[number]@cache@id=2246a14283dbaea5-2c47052c562e888c:
- Memory Current: 11.35 MB (Peak: 11.35 MB)
CreateTabletRRIdxCache[number]@cache@id=5d43e34440fcc836-38d8440bba48c49b:
- Memory Current: 63.23 KB (Peak: 114.22 KB)
MowDeleteBitmapAggCache[size]@cache@id=744736d130b6f0e6-aab37b699b3b8387:
- Memory Current: 11.48 MB (Peak: 317.85 MB)
W20260415 14:25:32.021731 1271504 memory_profile.cpp:344]
TopMemoryTasksSnapshot:
Query#Id=359570d0dd9d43bd-99865eb1720d25cb@query@id=c24dc03ed4e5dcc2-3646b21052bfb693:
- Limit: 2.00 GB
- Memory Current: 1.91 GB (Peak: 2.21 GB)
Query#Id=454030fbbeff40aa-a33ae1d3d5450007@query@id=93484436cb0e5bcd-dcc5b701e8164abb:
- Limit: 2.00 GB
- Memory Current: 52.83 MB (Peak: 63.21 MB)
BufferControlBlock#FragmentInstanceId=a541d90a88ee4f0b-86bb636f20a9548c@query@id=b646accfd57acca1-7868bd4a66558ca9:
- Memory Current: 0 (Peak: 0)
BufferControlBlock#FragmentInstanceId=c21b949670594ea9-be292b5af3e56386@query@id=2544c18a055ebe04-676a207a76efe290:
- Memory Current: 0 (Peak: 0)
BufferControlBlock#FragmentInstanceId=f7fb8c34883d48d8-a6bb37f0abf6e5dd@query@id=d04252d23a05f52d-7c8a9c1bd505bdb0:
- Memory Current: 0 (Peak: 0)
BufferControlBlock#FragmentInstanceId=116c73b259dc4a2c-85d76e5b077283cb@query@id=19492a43ae68130d-a43d57febe9d70a5:
- Memory Current: 0 (Peak: 0)
BufferControlBlock#FragmentInstanceId=b2b43483cd5c4bb9-8320e28b23688f7b@query@id=624a64b29077b8a8-b3746ce55488748c:
- Memory Current: 0 (Peak: 0)
BufferControlBlock#FragmentInstanceId=26b119997ae24ee7-8bb99988f61814a2@query@id=5d480e41b85c5d3a-c7ba6c9c6999bd8c:
- Memory Current: 0 (Peak: 0)
BufferControlBlock#FragmentInstanceId=24f4d17cbbbe4462-910f4cf210fa6c3b@query@id=204f3f9c9b66e016-f5cdcc6b040c6da2:
- Memory Current: 0 (Peak: 0)
SnapshotManager@other_task@id=6441d457671b8565-97d13c3de72a19b2:
- Memory Current: 0 (Peak: 0)
StreamLoadPipe@load@id=b6424f3d9075cfd4-69f0a5d35960248e:
- Memory Current: 0 (Peak: 143.38 MB)
SegCompaction@compaction@id=7b49e2dadc47d5c7-0eacd062b3d96582:
- Memory Current: 0 (Peak: 0)
BufferControlBlock#FragmentInstanceId=55ea59c92a464b08-a911b2ad26b636bf@query@id=af49d1326b54bbe1-aac1ada346061c91:
- Memory Current: 0 (Peak: 0)
BufferControlBlock#FragmentInstanceId=bebea5ee5f5142a8-a77e06efdc3f4ac0@query@id=77475289b102baee-cf1873b861a5eb9e:
- Memory Current: 0 (Peak: 0)
BufferControlBlock#FragmentInstanceId=888f5d95c14d4b9e-bbe728afd2f73db0@query@id=2a4c55ec1a47018e-78281114e00d64a9:
- Memory Current: 0 (Peak: 0)
I20260415 14:25:32.021806 1271504 memory_reclamation.cpp:244] [MemoryGC] start GC work load group that enable overcommit, number of group: 1, request_free_memory:3027395911, total_exceeded_memory:0, request more than exceeded, try free size = (group used - group limit).
I20260415 14:25:32.021814 1271504 mem_tracker_limiter.cpp:551] [MemoryGC] GC free process top memory overcommit query, , start seek all query, running query and load num: 2
I20260415 14:25:32.021855 1271504 mem_tracker_limiter.cpp:590] [MemoryGC] GC free process top memory overcommit query, seek finished, seek 14 tasks. among them, 0 tasks can be canceled; 14 small tasks that were skipped; 0 tasks is being canceled and has not been completed yet;
I20260415 14:25:32.021859 1271504 mem_tracker_limiter.cpp:598] [MemoryGC] GC free process top memory overcommit query, finished, no task need be canceled.
I20260415 14:25:32.021865 1271504 memory_reclamation.cpp:43] [MemoryGC] end minor GC, free memory 0. cost(us): 88, details: :
FreeTopOvercommitMemoryQuery:
WorkloadGroup:
