2.1.7 版本doris集群中的一个be节点，凌晨3点7分突然离线重启

Viewed 51
2.1

微信截图_20250626085955.png

查看此be节点的日志，发现在这个时间点前，频繁的报此类日志：
I20250626 03:05:43.912931 3051468 tablet.cpp:3538] [Before Commit] construct delete bitmap tablet: 7475436, rowset_ids to add:
0, rowset_ids to del: 0, cur max_version: 2038229, transaction_id: 2567492, total rows: 0
I20250626 03:05:43.912940 3051468 rowset_builder.cpp:301] Got result of calc delete bitmap task from executor, tablet_id: 7475
382, txn_id: 2567492
————————————————————————
I20250626 03:05:43.979095 3051539 tablet.cpp:3538] [Before Commit] construct delete bitmap tablet: 7475412, rowset_ids to add:
0, rowset_ids to del: 0, cur max_version: 2038230, transaction_id: 2567493, total rows: 0
I20250626 03:05:43.979126 3051539 rowset_builder.cpp:287] submit calc delete bitmap task to executor, tablet_id: 7475418, txn_
id: 2567493
I20250626 03:05:43.979143 3051539 tablet.cpp:3186] skip to construct delete bitmap tablet: 7475418 rowset: 0200000000cf1785034
307d252b9998058ac4b9b89873f94
——————————————————————————
I20250626 03:05:47.386040 3052067 daemon.cpp:221] os physical memory 15.39 GB. process memory used 1.30 GB(= 1.73 GB[vm/rss] -
442.74 MB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), limit 13.85 GB, soft limit 12.46 GB. sys available memory
5.25 GB(= 5.25 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), low water mark 787.72 MB, warning water mark 1.54 GB.
I20250626 03:05:47.487803 3052067 daemon.cpp:221] os physical memory 15.39 GB. process memory used 702.91 MB(= 1.12 GB[vm/rss]

  • 442.72 MB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), limit 13.85 GB, soft limit 12.46 GB. sys available memor
    y 6.27 GB(= 6.27 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), low water mark 787.72 MB, warning water mark 1.54 GB
    .
    I20250626 03:05:48.198930 3052067 daemon.cpp:221] os physical memory 15.39 GB. process memory used 1.17 GB(= 1.60 GB[vm/rss] -
    442.75 MB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), limit 13.85 GB, soft limit 12.46 GB. sys available memory
    5.79 GB(= 5.79 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), low water mark 787.72 MB, warning water mark 1.54 GB.
    I20250626 03:05:48.300678 3052067 daemon.cpp:221] os physical memory 15.39 GB. process memory used 1.80 GB(= 2.23 GB[vm/rss] -
    442.75 MB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), limit 13.85 GB, soft limit 12.46 GB. sys available memory
    6.27 GB(= 6.27 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), low water mark 787.72 MB, warning water mark 1.54 GB.
    ———————————————————————————————————
    I20250626 03:06:07.329941 3051257 beta_rowset.cpp:222] deleting /data/doris/data/data/370/7475436/1179448453/0200000000cf146b0
    34307d252b9998058ac4b9b89873f94_0.dat
    I20250626 03:06:07.330850 3051257 beta_rowset.cpp:222] deleting /data/doris/data/data/370/7475436/1179448453/0200000000cf140e0
    34307d252b9998058ac4b9b89873f94_0.dat
    I20250626 03:06:07.330909 3051257 beta_rowset.cpp:222] deleting /data/doris/data/data/370/7475436/1179448453/0200000000cf13ea0
    34307d252b9998058ac4b9b89873f94_0.dat
    I20250626 03:06:07.330951 3051257 beta_rowset.cpp:222] deleting /data/doris/data/data/370/7475436/1179448453/0200000000cf13710
    34307d252b9998058ac4b9b89873f94_0.dat
    I20250626 03:06:07.330991 3051257 beta_rowset.cpp:222] deleting /data/doris/data/data/370/7475436/1179448453/0200000000cf11e50
    34307d252b9998058ac4b9b89873f94_0.dat
    I20250626 03:06:07.331056 3051257 beta_rowset.cpp:222] deleting /data/doris/data/data/369/7475430/1179448453/0200000000cf13de0
    34307d252b9998058ac4b9b89873f94_0.dat
2 Answers

参考这个帖子先看下：BE 宕机问题排查

可以将故障节点的 be.out 信息补充下，同时有可以加我主页微信一起看看

阿渊@SelectDB8000

的确触发了oom。当时时间的be.out的日志信息，请帮忙看下如何调优下，感谢大佬：
W20250626 03:07:13.029119 3052068 mem_tracker_limiter.cpp:514]
Process Memory Summary:
os physical memory 15.39 GB. process memory used 3.62 GB(= 6.55 GB[vm/rss] - 2.93 GB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), limit 13
.85 GB, soft limit 12.46 GB. sys available memory 861.62 MB(= 861.62 MB[proc/available] - 0[reserved] - 0B[waiting_refresh]), low water mark 787.72 MB, wa
rning water mark 1.54 GB.
Memory Tracker Summary:
MemTrackerLimiter Label=other, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=schema_change, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=compaction, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=891.66 MB(934971196 B)
MemTrackerLimiter Label=load, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=4.86 GB(5220600013 B)
MemTrackerLimiter Label=query, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=580.42 MB(608616296 B)
MemTrackerLimiter Label=global, Type=overview, Limit=-1.00 B(-1 B), Used=134.25 MB(140776449 B), Peak=2.56 GB(2744214162 B)
MemTrackerLimiter Label=tc/jemalloc_cache, Type=overview, Limit=-1.00 B(-1 B), Used=2.93 GB(3149018184 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=tc/jemalloc_metadata, Type=overview, Limit=-1.00 B(-1 B), Used=133.53 MB(140014840 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=reserved_memory, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=sum_of_all_trackers, Type=overview, Limit=-1.00 B(-1 B), Used=3.19 GB(3429809473 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=VmRSS(process resident memory), Type=overview, Limit=-1.00 B(-1 B), Used=6.55 GB(7032406016 B), Peak=6.55 GB(7032406016 B)
MemTrackerLimiter Label=VmSize(process virtual memory), Type=overview, Limit=-1.00 B(-1 B), Used=51.22 GB(54992437248 B), Peak=51.22 GB(54992437248 B)
MemTrackerLimiter Label=Orphan, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=DetailsTrackerSet, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTracker Label=PageNoCache, Parent Label=DetailsTrackerSet, Used=0(0 B), Peak=214.60 MB(225024422 B)
MemTracker Label=IOBufBlockMemory, Parent Label=DetailsTrackerSet, Used=28.34 MB(29720576 B), Peak=276.54 MB(289972224 B)
MemTracker Label=SegmentCache[size], Parent Label=DetailsTrackerSet, Used=44.08 MB(46221700 B), Peak=191.04 MB(200319776 B)
MemTracker Label=SchemaCache[number], Parent Label=DetailsTrackerSet, Used=167.83 KB(171863 B), Peak=183.11 KB(187502 B)
MemTracker Label=AllMemTableMemory, Parent Label=DetailsTrackerSet, Used=0(0 B), Peak=3.34 GB(3583591609 B)
MemTracker Label=TabletSchemaCache[number], Parent Label=DetailsTrackerSet, Used=266.40 KB(272791 B), Peak=308.07 KB(315460 B)
MemTracker Label=TabletMeta(experimental), Parent Label=DetailsTrackerSet, Used=2.54 MB(2668000 B), Peak=2.68 MB(2810784 B)
MemTracker Label=MowTabletVersionCache[number], Parent Label=DetailsTrackerSet, Used=11.35 MB(11900000 B), Peak=11.35 MB(11900238 B)
MemTracker Label=CreateTabletRRIdxCache[number], Parent Label=DetailsTrackerSet, Used=28.57 KB(29258 B), Peak=28.68 KB(29367 B)
MemTracker Label=MowDeleteBitmapAggCache[size], Parent Label=DetailsTrackerSet, Used=2.39 MB(2509391 B), Peak=2.39 MB(2509391 B)
MemTrackerLimiter Label=SegCompaction, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=422.59 MB(443121133 B)
MemTrackerLimiter Label=PointQueryExecutor, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=BlockCompression, Type=global, Limit=-1.00 B(-1 B), Used=10.10 MB(10589342 B), Peak=10.35 MB(10851484 B)
MemTrackerLimiter Label=RowIdStorageReader, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=14.44 KB(14784 B)
MemTrackerLimiter Label=SubcolumnsTree, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=S3FileBuffer, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=DataPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=43.89 MB(46017994 B), Peak=2.48 GB(2658633660 B)
MemTrackerLimiter Label=IndexPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=21.38 MB(22417446 B), Peak=172.00 MB(180356858
B)
MemTrackerLimiter Label=PKIndexPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=58.89 MB(61751667 B), Peak=65.14 MB(68306169
B)
MemTrackerLimiter Label=StreamLoadPipe, Type=load, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=10.88 MB(11403264 B)
MemTrackerLimiter Label=MemTableTrackerSet, Type=load, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTracker Label=AllMemTableMemory, Parent Label=DetailsTrackerSet, Used=0(0 B), Peak=3.34 GB(3583591609 B)
W20250626 03:07:13.240897 3052068 mem_tracker_limiter.cpp:514]
Process Memory Summary:
os physical memory 15.39 GB. process memory used 3.92 GB(= 6.86 GB[vm/rss] - 2.93 GB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), limit 13
.85 GB, soft limit 12.46 GB. sys available memory 611.75 MB(= 611.75 MB[proc/available] - 0[reserved] - 0B[waiting_refresh]), low water mark 787.72 MB, wa
rning water mark 1.54 GB.
Memory Tracker Summary:
MemTrackerLimiter Label=other, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=schema_change, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=compaction, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=891.66 MB(934971196 B)
MemTrackerLimiter Label=load, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=4.86 GB(5220600013 B)
MemTrackerLimiter Label=query, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=580.42 MB(608616296 B)
MemTrackerLimiter Label=global, Type=overview, Limit=-1.00 B(-1 B), Used=134.25 MB(140776449 B), Peak=2.56 GB(2744214162 B)
MemTrackerLimiter Label=tc/jemalloc_cache, Type=overview, Limit=-1.00 B(-1 B), Used=2.93 GB(3150522928 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=tc/jemalloc_metadata, Type=overview, Limit=-1.00 B(-1 B), Used=134.34 MB(140866808 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=reserved_memory, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=sum_of_all_trackers, Type=overview, Limit=-1.00 B(-1 B), Used=3.20 GB(3432166185 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=VmRSS(process resident memory), Type=overview, Limit=-1.00 B(-1 B), Used=6.86 GB(7363559424 B), Peak=6.86 GB(7363559424 B)
MemTrackerLimiter Label=VmSize(process virtual memory), Type=overview, Limit=-1.00 B(-1 B), Used=54.75 GB(58784350208 B), Peak=54.75 GB(58784350208 B)
MemTrackerLimiter Label=Orphan, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=DetailsTrackerSet, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTracker Label=PageNoCache, Parent Label=DetailsTrackerSet, Used=0(0 B), Peak=214.60 MB(225024422 B)
MemTracker Label=IOBufBlockMemory, Parent Label=DetailsTrackerSet, Used=28.34 MB(29720576 B), Peak=276.54 MB(289972224 B)
MemTracker Label=SegmentCache[size], Parent Label=DetailsTrackerSet, Used=44.08 MB(46221700 B), Peak=191.04 MB(200319776 B)
MemTracker Label=SchemaCache[number], Parent Label=DetailsTrackerSet, Used=167.83 KB(171863 B), Peak=183.11 KB(187502 B)
MemTracker Label=AllMemTableMemory, Parent Label=DetailsTrackerSet, Used=0(0 B), Peak=3.34 GB(3583591609 B)
MemTracker Label=TabletSchemaCache[number], Parent Label=DetailsTrackerSet, Used=266.40 KB(272791 B), Peak=308.07 KB(315460 B)
MemTracker Label=TabletMeta(experimental), Parent Label=DetailsTrackerSet, Used=2.54 MB(2668000 B), Peak=2.68 MB(2810784 B)
MemTracker Label=MowTabletVersionCache[number], Parent Label=DetailsTrackerSet, Used=11.35 MB(11900000 B), Peak=11.35 MB(11900238 B)
MemTracker Label=CreateTabletRRIdxCache[number], Parent Label=DetailsTrackerSet, Used=28.57 KB(29258 B), Peak=28.68 KB(29367 B)
MemTracker Label=MowDeleteBitmapAggCache[size], Parent Label=DetailsTrackerSet, Used=2.39 MB(2509391 B), Peak=2.39 MB(2509391 B)
MemTrackerLimiter Label=SegCompaction, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=422.59 MB(443121133 B)
MemTrackerLimiter Label=PointQueryExecutor, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=BlockCompression, Type=global, Limit=-1.00 B(-1 B), Used=10.10 MB(10589342 B), Peak=10.35 MB(10851484 B)
MemTrackerLimiter Label=RowIdStorageReader, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=14.44 KB(14784 B)
MemTrackerLimiter Label=SubcolumnsTree, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=S3FileBuffer, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=DataPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=43.89 MB(46017994 B), Peak=2.48 GB(2658633660 B)
MemTrackerLimiter Label=IndexPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=21.38 MB(22417446 B), Peak=172.00 MB(180356858
B)
MemTrackerLimiter Label=PKIndexPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=58.89 MB(61751667 B), Peak=65.14 MB(68306169
B)
MemTrackerLimiter Label=StreamLoadPipe, Type=load, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=10.88 MB(11403264 B)
MemTrackerLimiter Label=MemTableTrackerSet, Type=load, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTracker Label=AllMemTableMemory, Parent Label=DetailsTrackerSet, Used=0(0 B), Peak=3.34 GB(3583591609 B)
W20250626 03:07:13.241711 3052068 mem_tracker_limiter.cpp:635] [MemoryGC] GC free process top memory used load, Task ID parsing failed, label: MemTableTra
ckerSet
W20250626 03:07:13.241724 3052068 mem_tracker_limiter.cpp:635] [MemoryGC] GC free process top memory used load, Task ID parsing failed, label: StreamLoadP
ipe
W20250626 03:07:35.743474 3422047 timezone_utils.cpp:94] Meet illegal tzdata file: zone.tab. skipped
W20250626 03:07:35.755882 3422047 timezone_utils.cpp:94] Meet illegal tzdata file: Africa/Casablanca. skipped
W20250626 03:07:35.760195 3422047 timezone_utils.cpp:94] Meet illegal tzdata file: zone1970.tab. skipped
W20250626 03:07:35.767633 3422047 timezone_utils.cpp:94] Meet illegal tzdata file: iso3166.tab. skipped
W20250626 03:07:35.780906 3422047 timezone_utils.cpp:94] Meet illegal tzdata file: tzdata.zi. skipped
W20250626 03:07:35.782308 3422047 timezone_utils.cpp:94] Meet illegal tzdata file: leapseconds. skipped
E20250626 03:07:35.962038 3422047 bfd_parser.cpp:98] set default target to elf64-x86-64 failed.
E20250626 03:07:37.144542 3422047 variable.cpp:179] Already exposed doris_cache_data_page_cache' whose value is 0'
E20250626 03:07:37.144587 3422047 variable.cpp:179] Already exposed doris_cache_data_page_cache_persecond' whose value is 0'
E20250626 03:07:37.144685 3422047 variable.cpp:179] Already exposed doris_cache_index_page_cache' whose value is 0'
E20250626 03:07:37.144699 3422047 variable.cpp:179] Already exposed doris_cache_index_page_cache_persecond' whose value is 0'
E20250626 03:07:37.144773 3422047 variable.cpp:179] Already exposed doris_cache_pkindex_page_cache' whose value is 0'
E20250626 03:07:37.144788 3422047 variable.cpp:179] Already exposed doris_cache_pkindex_page_cache_persecond' whose value is 0'
E20250626 03:07:37.144868 3422047 variable.cpp:179] Already exposed doris_cache_point_query_row_cache' whose value is 0'
E20250626 03:07:37.144882 3422047 variable.cpp:179] Already exposed doris_cache_point_query_row_cache_persecond' whose value is 0'
E20250626 03:07:37.144932 3422047 variable.cpp:179] Already exposed doris_cache_segment_cache' whose value is 0'
E20250626 03:07:37.144944 3422047 variable.cpp:179] Already exposed doris_cache_segment_cache_persecond' whose value is 0'
E20250626 03:07:37.144990 3422047 variable.cpp:179] Already exposed doris_cache_schema_cache' whose value is 0'
E20250626 03:07:37.145001 3422047 variable.cpp:179] Already exposed doris_cache_schema_cache_persecond' whose value is 0'
E20250626 03:07:37.145043 3422047 variable.cpp:179] Already exposed doris_cache_common_obj_lrucache' whose value is 0'
E20250626 03:07:37.145066 3422047 variable.cpp:179] Already exposed doris_cache_common_obj_lrucache_persecond' whose value is 0'
E20250626 03:07:37.145110 3422047 variable.cpp:179] Already exposed doris_cache_point_query_lookup_connection_cache' whose value is 0'
E20250626 03:07:37.145123 3422047 variable.cpp:179] Already exposed doris_cache_point_query_lookup_connection_cache_persecond' whose value is 0'
E20250626 03:07:37.145191 3422047 variable.cpp:179] Already exposed doris_cache_inverted_index_searcher_cache' whose value is 0'
E20250626 03:07:37.145205 3422047 variable.cpp:179] Already exposed doris_cache_inverted_index_searcher_cache_persecond' whose value is 0'
E20250626 03:07:37.145298 3422047 variable.cpp:179] Already exposed doris_cache_inverted_index_query_cache' whose value is 0'
E20250626 03:07:37.145311 3422047 variable.cpp:179] Already exposed doris_cache_inverted_index_query_cache_persecond' whose value is 0'
E20250626 03:07:37.147846 3422047 variable.cpp:179] Already exposed doris_cache_last_success_channel_cache' whose value is 0'
E20250626 03:07:37.147876 3422047 variable.cpp:179] Already exposed doris_cache_last_success_channel_cache_persecond' whose value is 0'
E20250626 03:07:37.148617 3422047 variable.cpp:179] Already exposed doris_cache_tablet_schema_cache' whose value is 0'
E20250626 03:07:37.148638 3422047 variable.cpp:179] Already exposed doris_cache_tablet_schema_cache_persecond' whose value is 0'
E20250626 03:07:37.149147 3422047 variable.cpp:179] Already exposed doris_cache_mow_tablet_version_cache' whose value is 0'
E20250626 03:07:37.149170 3422047 variable.cpp:179] Already exposed doris_cache_mow_tablet_version_cache_persecond' whose value is 0'
E20250626 03:07:37.149209 3422047 variable.cpp:179] Already exposed doris_cache_create_tablet_rridx_cache' whose value is 0'
E20250626 03:07:37.149220 3422047 variable.cpp:179] Already exposed doris_cache_create_tablet_rridx_cache_persecond' whose value is 0'
E20250626 03:07:37.394691 3422674 variable.cpp:179] Already exposed doris_cache_mow_delete_bitmap_agg_cache' whose value is 0'
E20250626 03:07:37.394730 3422674 variable.cpp:179] Already exposed doris_cache_mow_delete_bitmap_agg_cache_persecond' whose value is 0'
W20250626 03:07:37.854015 3422706 olap_server.cpp:670] Have not get FE Master heartbeat yet
W20250626 03:08:47.440640 3424631 doris_main.cpp:123] thrift internal message: TConnectedClient processing exception: Bad version identifier
W20250626 03:09:29.276840 3423500 mem_tracker_limiter.cpp:514]
Process Memory Summary:
os physical memory 15.39 GB. process memory used 4.04 GB(= 8.29 GB[vm/rss] - 4.24 GB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), limit 13
.85 GB, soft limit 12.46 GB. sys available memory 1.15 GB(= 1.15 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), low water mark 787.72 MB, warnin
g water mark 1.54 GB.
Memory Tracker Summary:
MemTrackerLimiter Label=other, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=schema_change, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=compaction, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=load, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=query, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=global, Type=overview, Limit=-1.00 B(-1 B), Used=289.61 KB(296557 B), Peak=289.61 KB(296557 B)
MemTrackerLimiter Label=tc/jemalloc_cache, Type=overview, Limit=-1.00 B(-1 B), Used=4.24 GB(4556715528 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=tc/jemalloc_metadata, Type=overview, Limit=-1.00 B(-1 B), Used=88.54 MB(92844296 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=reserved_memory, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=sum_of_all_trackers, Type=overview, Limit=-1.00 B(-1 B), Used=4.33 GB(4649856381 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=VmRSS(process resident memory), Type=overview, Limit=-1.00 B(-1 B), Used=8.29 GB(8899067904 B), Peak=8.50 GB(9128247296 B)
MemTrackerLimiter Label=VmSize(process virtual memory), Type=overview, Limit=-1.00 B(-1 B), Used=40.16 GB(43117051904 B), Peak=40.16 GB(43117051904 B)
MemTrackerLimiter Label=Orphan, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=DetailsTrackerSet, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTracker Label=PageNoCache, Parent Label=DetailsTrackerSet, Used=0(0 B), Peak=230.41 KB(235936 B)
MemTracker Label=IOBufBlockMemory, Parent Label=DetailsTrackerSet, Used=880.00 KB(901120 B), Peak=880.00 KB(901120 B)
MemTracker Label=SegmentCache[size], Parent Label=DetailsTrackerSet, Used=51.35 KB(52584 B), Peak=51.35 KB(52584 B)
MemTracker Label=AllMemTableMemory, Parent Label=DetailsTrackerSet, Used=0(0 B), Peak=38.81 KB(39744 B)
MemTracker Label=TabletSchemaCache[number], Parent Label=DetailsTrackerSet, Used=266.40 KB(272791 B), Peak=266.40 KB(272791 B)
MemTracker Label=TabletMeta(experimental), Parent Label=DetailsTrackerSet, Used=2.54 MB(2667264 B), Peak=2.54 MB(2668000 B)
MemTracker Label=MowTabletVersionCache[number], Parent Label=DetailsTrackerSet, Used=16.62 KB(17017 B), Peak=16.62 KB(17017 B)
MemTrackerLimiter Label=SegCompaction, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=PointQueryExecutor, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=BlockCompression, Type=global, Limit=-1.00 B(-1 B), Used=288.21 KB(295123 B), Peak=288.21 KB(295123 B)
MemTrackerLimiter Label=RowIdStorageReader, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=SubcolumnsTree, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=S3FileBuffer, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=DataPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=488.00 B(488 B), Peak=488.00 B(488 B)
MemTrackerLimiter Label=IndexPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=PKIndexPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=946.00 B(946 B), Peak=946.00 B(946 B)
MemTrackerLimiter Label=StreamLoadPipe, Type=load, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=128.00 KB(131072 B)
MemTrackerLimiter Label=MemTableTrackerSet, Type=load, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTracker Label=AllMemTableMemory, Parent Label=DetailsTrackerSet, Used=0(0 B), Peak=38.81 KB(39744 B)
W20250626 03:09:29.709313 3423500 mem_tracker_limiter.cpp:514]
Process Memory Summary:
os physical memory 15.39 GB. process memory used 3.17 GB(= 8.66 GB[vm/rss] - 5.50 GB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), limit 13
.85 GB, soft limit 12.46 GB. sys available memory 774.19 MB(= 774.19 MB[proc/available] - 0[reserved] - 0B[waiting_refresh]), low water mark 787.72 MB, wa
rning water mark 1.54 GB.
Memory Tracker Summary:
MemTrackerLimiter Label=other, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=schema_change, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=compaction, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=load, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=query, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=global, Type=overview, Limit=-1.00 B(-1 B), Used=289.61 KB(296557 B), Peak=289.61 KB(296557 B)
MemTrackerLimiter Label=tc/jemalloc_cache, Type=overview, Limit=-1.00 B(-1 B), Used=5.50 GB(5900384000 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=tc/jemalloc_metadata, Type=overview, Limit=-1.00 B(-1 B), Used=88.56 MB(92859272 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=reserved_memory, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=sum_of_all_trackers, Type=overview, Limit=-1.00 B(-1 B), Used=5.58 GB(5993539829 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=VmRSS(process resident memory), Type=overview, Limit=-1.00 B(-1 B), Used=8.66 GB(9303687168 B), Peak=8.66 GB(9303687168 B)
MemTrackerLimiter Label=VmSize(process virtual memory), Type=overview, Limit=-1.00 B(-1 B), Used=40.16 GB(43125112832 B), Peak=40.17 GB(43133960192 B)
MemTrackerLimiter Label=Orphan, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=DetailsTrackerSet, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTracker Label=PageNoCache, Parent Label=DetailsTrackerSet, Used=0(0 B), Peak=230.41 KB(235936 B)
MemTracker Label=IOBufBlockMemory, Parent Label=DetailsTrackerSet, Used=880.00 KB(901120 B), Peak=880.00 KB(901120 B)
MemTracker Label=SegmentCache[size], Parent Label=DetailsTrackerSet, Used=51.35 KB(52584 B), Peak=51.35 KB(52584 B)
MemTracker Label=AllMemTableMemory, Parent Label=DetailsTrackerSet, Used=0(0 B), Peak=38.81 KB(39744 B)
MemTracker Label=TabletSchemaCache[number], Parent Label=DetailsTrackerSet, Used=266.40 KB(272791 B), Peak=266.40 KB(272791 B)
MemTracker Label=TabletMeta(experimental), Parent Label=DetailsTrackerSet, Used=2.54 MB(2667264 B), Peak=2.54 MB(2668000 B)
MemTracker Label=MowTabletVersionCache[number], Parent Label=DetailsTrackerSet, Used=16.62 KB(17017 B), Peak=16.62 KB(17017 B)
MemTrackerLimiter Label=SegCompaction, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=PointQueryExecutor, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=BlockCompression, Type=global, Limit=-1.00 B(-1 B), Used=288.21 KB(295123 B), Peak=288.21 KB(295123 B)
MemTrackerLimiter Label=RowIdStorageReader, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=SubcolumnsTree, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=S3FileBuffer, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=DataPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=488.00 B(488 B), Peak=488.00 B(488 B)
MemTrackerLimiter Label=IndexPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=PKIndexPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=946.00 B(946 B), Peak=946.00 B(946 B)
MemTrackerLimiter Label=StreamLoadPipe, Type=load, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=128.00 KB(131072 B)
MemTrackerLimiter Label=MemTableTrackerSet, Type=load, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTracker Label=AllMemTableMemory, Parent Label=DetailsTrackerSet, Used=0(0 B), Peak=38.81 KB(39744 B)
W20250626 03:09:29.710094 3423500 mem_tracker_limiter.cpp:635] [MemoryGC] GC free process top memory used load, Task ID parsing failed, label: MemTableTra
ckerSet
W20250626 03:09:29.710105 3423500 mem_tracker_limiter.cpp:635] [MemoryGC] GC free process top memory used load, Task ID parsing failed, label: StreamLoadP
ipe
W20250626 03:09:30.992501 3423500 mem_tracker_limiter.cpp:514]
Process Memory Summary:
os physical memory 15.39 GB. process memory used 3.20 GB(= 8.96 GB[vm/rss] - 5.75 GB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), limit 13
.85 GB, soft limit 12.46 GB. sys available memory 480.62 MB(= 480.62 MB[proc/available] - 0[reserved] - 0B[waiting_refresh]), low water mark 787.72 MB, wa
rning water mark 1.54 GB.
Memory Tracker Summary:
MemTrackerLimiter Label=other, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=schema_change, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=compaction, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=load, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=query, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=global, Type=overview, Limit=-1.00 B(-1 B), Used=289.61 KB(296557 B), Peak=289.61 KB(296557 B)
MemTrackerLimiter Label=tc/jemalloc_cache, Type=overview, Limit=-1.00 B(-1 B), Used=5.75 GB(6177343456 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=tc/jemalloc_metadata, Type=overview, Limit=-1.00 B(-1 B), Used=88.75 MB(93059464 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=reserved_memory, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=sum_of_all_trackers, Type=overview, Limit=-1.00 B(-1 B), Used=5.84 GB(6270699477 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=VmRSS(process resident memory), Type=overview, Limit=-1.00 B(-1 B), Used=8.96 GB(9617604608 B), Peak=8.96 GB(9621471232 B)
MemTrackerLimiter Label=VmSize(process virtual memory), Type=overview, Limit=-1.00 B(-1 B), Used=43.16 GB(46347452416 B), Peak=43.17 GB(46355906560 B)
MemTrackerLimiter Label=Orphan, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=DetailsTrackerSet, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTracker Label=PageNoCache, Parent Label=DetailsTrackerSet, Used=0(0 B), Peak=230.41 KB(235936 B)
MemTracker Label=IOBufBlockMemory, Parent Label=DetailsTrackerSet, Used=920.00 KB(942080 B), Peak=920.00 KB(942080 B)
MemTracker Label=SegmentCache[size], Parent Label=DetailsTrackerSet, Used=51.35 KB(52584 B), Peak=51.35 KB(52584 B)
MemTracker Label=AllMemTableMemory, Parent Label=DetailsTrackerSet, Used=0(0 B), Peak=38.81 KB(39744 B)
MemTracker Label=TabletSchemaCache[number], Parent Label=DetailsTrackerSet, Used=266.40 KB(272791 B), Peak=266.40 KB(272791 B)
MemTracker Label=TabletMeta(experimental), Parent Label=DetailsTrackerSet, Used=2.54 MB(2667264 B), Peak=2.54 MB(2668000 B)
MemTracker Label=MowTabletVersionCache[number], Parent Label=DetailsTrackerSet, Used=17.90 KB(18326 B), Peak=17.90 KB(18326 B)
MemTrackerLimiter Label=SegCompaction, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=PointQueryExecutor, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=BlockCompression, Type=global, Limit=-1.00 B(-1 B), Used=288.21 KB(295123 B), Peak=288.21 KB(295123 B)
MemTrackerLimiter Label=RowIdStorageReader, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=SubcolumnsTree, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=S3FileBuffer, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=DataPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=488.00 B(488 B), Peak=488.00 B(488 B)
MemTrackerLimiter Label=IndexPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=PKIndexPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=946.00 B(946 B), Peak=946.00 B(946 B)
MemTrackerLimiter Label=StreamLoadPipe, Type=load, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=128.00 KB(131072 B)
MemTrackerLimiter Label=MemTableTrackerSet, Type=load, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTracker Label=AllMemTableMemory, Parent Label=DetailsTrackerSet, Used=0(0 B), Peak=38.81 KB(39744 B)
W20250626 03:09:30.993204 3423500 mem_tracker_limiter.cpp:635] [MemoryGC] GC free process top memory used load, Task ID parsing failed, label: MemTableTra
ckerSet
W20250626 03:09:30.993225 3423500 mem_tracker_limiter.cpp:635] [MemoryGC] GC free process top memory used load, Task ID parsing failed, label: StreamLoadP
ipe
W20250626 03:09:33.008575 3423500 mem_tracker_limiter.cpp:514]
Process Memory Summary:
os physical memory 15.39 GB. process memory used 4.21 GB(= 8.46 GB[vm/rss] - 4.25 GB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), limit 13
.85 GB, soft limit 12.46 GB. sys available memory 976.50 MB(= 976.50 MB[proc/available] - 0[reserved] - 0B[waiting_refresh]), low water mark 787.72 MB, wa
rning water mark 1.54 GB.
Memory Tracker Summary:
MemTrackerLimiter Label=other, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=schema_change, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=compaction, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=load, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=query, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=global, Type=overview, Limit=-1.00 B(-1 B), Used=289.61 KB(296557 B), Peak=289.61 KB(296557 B)
MemTrackerLimiter Label=tc/jemalloc_cache, Type=overview, Limit=-1.00 B(-1 B), Used=4.25 GB(4565294152 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=tc/jemalloc_metadata, Type=overview, Limit=-1.00 B(-1 B), Used=88.75 MB(93059464 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=reserved_memory, Type=overview, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=sum_of_all_trackers, Type=overview, Limit=-1.00 B(-1 B), Used=4.34 GB(4658650173 B), Peak=-1.00 B(-1 B)
MemTrackerLimiter Label=VmRSS(process resident memory), Type=overview, Limit=-1.00 B(-1 B), Used=8.46 GB(9081978880 B), Peak=8.96 GB(9621471232 B)
MemTrackerLimiter Label=VmSize(process virtual memory), Type=overview, Limit=-1.00 B(-1 B), Used=43.16 GB(46338998272 B), Peak=43.17 GB(46355906560 B)
MemTrackerLimiter Label=Orphan, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=DetailsTrackerSet, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTracker Label=PageNoCache, Parent Label=DetailsTrackerSet, Used=0(0 B), Peak=230.41 KB(235936 B)
MemTracker Label=IOBufBlockMemory, Parent Label=DetailsTrackerSet, Used=920.00 KB(942080 B), Peak=920.00 KB(942080 B)
MemTracker Label=SegmentCache[size], Parent Label=DetailsTrackerSet, Used=51.35 KB(52584 B), Peak=51.35 KB(52584 B)
MemTracker Label=AllMemTableMemory, Parent Label=DetailsTrackerSet, Used=0(0 B), Peak=38.81 KB(39744 B)
MemTracker Label=TabletSchemaCache[number], Parent Label=DetailsTrackerSet, Used=266.40 KB(272791 B), Peak=266.40 KB(272791 B)
MemTracker Label=TabletMeta(experimental), Parent Label=DetailsTrackerSet, Used=2.54 MB(2667264 B), Peak=2.54 MB(2668000 B)
MemTracker Label=MowTabletVersionCache[number], Parent Label=DetailsTrackerSet, Used=17.90 KB(18326 B), Peak=17.90 KB(18326 B)
MemTrackerLimiter Label=SegCompaction, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=PointQueryExecutor, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=BlockCompression, Type=global, Limit=-1.00 B(-1 B), Used=288.21 KB(295123 B), Peak=288.21 KB(295123 B)
MemTrackerLimiter Label=RowIdStorageReader, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=SubcolumnsTree, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=S3FileBuffer, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=DataPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=488.00 B(488 B), Peak=488.00 B(488 B)
MemTrackerLimiter Label=IndexPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTrackerLimiter Label=PKIndexPageCachesize, Type=global, Limit=-1.00 B(-1 B), Used=946.00 B(946 B), Peak=946.00 B(946 B)
MemTrackerLimiter Label=StreamLoadPipe, Type=load, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=128.00 KB(131072 B)
MemTrackerLimiter Label=MemTableTrackerSet, Type=load, Limit=-1.00 B(-1 B), Used=0(0 B), Peak=0(0 B)
MemTracker Label=AllMemTableMemory, Parent Label=DetailsTrackerSet, Used=0(0 B), Peak=38.81 KB(39744 B)

dongfeng1
Related Questions
stream load离线一批导入大量数据时，实时导入延迟
1 answers
Doris通过s3上传数据到minio，报错curlCode: 28, Timeout was reached, exception , error code -1
集成Doris的connector的时候数据不同步
doris删除数据后新增__DORIS_DELETE_SIGN__=1的数据，请问删除后的数据在什么情况下才会触发compaction操作
求助Be挂掉异常分析
1 answers
VARIANT嵌套子列格式内容问题