2.1.7 版本doris集群中的一个be节点,凌晨3点7分突然离线重启

Viewed 12

微信截图_20250626085955.png

查看此be节点的日志,发现在这个时间点前,频繁的报此类日志:
I20250626 03:05:43.912931 3051468 tablet.cpp:3538] [Before Commit] construct delete bitmap tablet: 7475436, rowset_ids to add:
0, rowset_ids to del: 0, cur max_version: 2038229, transaction_id: 2567492, total rows: 0
I20250626 03:05:43.912940 3051468 rowset_builder.cpp:301] Got result of calc delete bitmap task from executor, tablet_id: 7475
382, txn_id: 2567492
————————————————————————
I20250626 03:05:43.979095 3051539 tablet.cpp:3538] [Before Commit] construct delete bitmap tablet: 7475412, rowset_ids to add:
0, rowset_ids to del: 0, cur max_version: 2038230, transaction_id: 2567493, total rows: 0
I20250626 03:05:43.979126 3051539 rowset_builder.cpp:287] submit calc delete bitmap task to executor, tablet_id: 7475418, txn_
id: 2567493
I20250626 03:05:43.979143 3051539 tablet.cpp:3186] skip to construct delete bitmap tablet: 7475418 rowset: 0200000000cf1785034
307d252b9998058ac4b9b89873f94
——————————————————————————
I20250626 03:05:47.386040 3052067 daemon.cpp:221] os physical memory 15.39 GB. process memory used 1.30 GB(= 1.73 GB[vm/rss] -
442.74 MB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), limit 13.85 GB, soft limit 12.46 GB. sys available memory
5.25 GB(= 5.25 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), low water mark 787.72 MB, warning water mark 1.54 GB.
I20250626 03:05:47.487803 3052067 daemon.cpp:221] os physical memory 15.39 GB. process memory used 702.91 MB(= 1.12 GB[vm/rss]

  • 442.72 MB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), limit 13.85 GB, soft limit 12.46 GB. sys available memor
    y 6.27 GB(= 6.27 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), low water mark 787.72 MB, warning water mark 1.54 GB
    .
    I20250626 03:05:48.198930 3052067 daemon.cpp:221] os physical memory 15.39 GB. process memory used 1.17 GB(= 1.60 GB[vm/rss] -
    442.75 MB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), limit 13.85 GB, soft limit 12.46 GB. sys available memory
    5.79 GB(= 5.79 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), low water mark 787.72 MB, warning water mark 1.54 GB.
    I20250626 03:05:48.300678 3052067 daemon.cpp:221] os physical memory 15.39 GB. process memory used 1.80 GB(= 2.23 GB[vm/rss] -
    442.75 MB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), limit 13.85 GB, soft limit 12.46 GB. sys available memory
    6.27 GB(= 6.27 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), low water mark 787.72 MB, warning water mark 1.54 GB.
    ———————————————————————————————————
    I20250626 03:06:07.329941 3051257 beta_rowset.cpp:222] deleting /data/doris/data/data/370/7475436/1179448453/0200000000cf146b0
    34307d252b9998058ac4b9b89873f94_0.dat
    I20250626 03:06:07.330850 3051257 beta_rowset.cpp:222] deleting /data/doris/data/data/370/7475436/1179448453/0200000000cf140e0
    34307d252b9998058ac4b9b89873f94_0.dat
    I20250626 03:06:07.330909 3051257 beta_rowset.cpp:222] deleting /data/doris/data/data/370/7475436/1179448453/0200000000cf13ea0
    34307d252b9998058ac4b9b89873f94_0.dat
    I20250626 03:06:07.330951 3051257 beta_rowset.cpp:222] deleting /data/doris/data/data/370/7475436/1179448453/0200000000cf13710
    34307d252b9998058ac4b9b89873f94_0.dat
    I20250626 03:06:07.330991 3051257 beta_rowset.cpp:222] deleting /data/doris/data/data/370/7475436/1179448453/0200000000cf11e50
    34307d252b9998058ac4b9b89873f94_0.dat
    I20250626 03:06:07.331056 3051257 beta_rowset.cpp:222] deleting /data/doris/data/data/369/7475430/1179448453/0200000000cf13de0
    34307d252b9998058ac4b9b89873f94_0.dat
1 Answers

参考这个帖子先看下:BE 宕机问题排查

可以将故障节点的 be.out 信息补充下,同时有可以加我主页微信一起看看