Doris版本:2.1.9
在监控中发现有一个BE节点的文件描述符持续增长,已经逼近阈值200k了(看监控2.1.6;2.1.8;2.1.9三个版本都有这个问题【3.29-4.20期间做了升级】)。
整个集群只有这一个节点be的文件描述符超高,关键是这个节点的任务量不大。
在重启后文件描述符还是使用这么多,4.25才升级到2.1.9,可以看到监控在启动后直接就飙到190k。
大佬们看下这个问题该怎么解决,如果超出200k又会导致什么现象呢?
该节点的配置与其他节点配置相同
日志中没有和这个相关的异常报错,下面是集群内报错最多的内容
W20250428 07:41:54.832345 9663 fragment_mgr.cpp:1305] Could not find the query id:7bc3a7f1d92148b0-9a766826ffd3ed4c fragment id:1 to cancel
W20250428 07:41:54.889052 9548 fragment_mgr.cpp:1305] Could not find the query id:2b19e8a2d7f3412b-88e39d6c3976050a fragment id:2 to cancel
W20250428 07:41:54.889072 9662 fragment_mgr.cpp:1305] Could not find the query id:2b19e8a2d7f3412b-88e39d6c3976050a fragment id:0 to cancel
W20250428 07:41:54.889123 9662 fragment_mgr.cpp:1305] Could not find the query id:2b19e8a2d7f3412b-88e39d6c3976050a fragment id:1 to cancel
W20250428 07:44:26.322192 10197 ref_count_closure.h:119] RPC meet error status: [INVALID_ARGUMENT]PStatus: (172.16.24.152)[INVALID_ARGUMENT]not found entity, query-id:be3711b726b1426b-8885d232f8f73e72
W20250428 07:44:26.324616 10418 ref_count_closure.h:119] RPC meet error status: [INVALID_ARGUMENT]PStatus: (172.16.24.152)[INVALID_ARGUMENT]not found entity, query-id:be3711b726b1426b-8885d232f8f73e72
W20250428 07:44:26.326709 10212 ref_count_closure.h:119] RPC meet error status: [INVALID_ARGUMENT]PStatus: (172.16.24.152)[INVALID_ARGUMENT]not found entity, query-id:be3711b726b1426b-8885d232f8f73e72
W20250428 07:49:26.575433 10383 ref_count_closure.h:119] RPC meet error status: [INVALID_ARGUMENT]PStatus: (172.16.24.152)[INVALID_ARGUMENT]not found entity, query-id:306e62cb1de24018-849c287c4a92bc9d
W20250428 07:49:26.577114 10228 ref_count_closure.h:119] RPC meet error status: [INVALID_ARGUMENT]PStatus: (172.16.24.152)[INVALID_ARGUMENT]not found entity, query-id:306e62cb1de24018-849c287c4a92bc9d
W20250428 07:49:26.578711 10244 ref_count_closure.h:119] RPC meet error status: [INVALID_ARGUMENT]PStatus: (172.16.24.152)[INVALID_ARGUMENT]not found entity, query-id:306e62cb1de24018-849c287c4a92bc9d
W20250428 08:24:43.907970 10269 ref_count_closure.h:119] RPC meet error status: [INVALID_ARGUMENT]PStatus: (172.16.24.152)[INVALID_ARGUMENT]query-id: bb57b9627d2046ac-809900c2e5d0ac60
W20250428 08:24:43.908025 10380 ref_count_closure.h:119] RPC meet error status: [INVALID_ARGUMENT]PStatus: (172.16.24.145)[INVALID_ARGUMENT]query-id: bb57b9627d2046ac-809900c2e5d0ac60
W20250428 08:44:46.754751 10401 ref_count_closure.h:119] RPC meet error status: [INVALID_ARGUMENT]PStatus: (172.16.24.152)[INVALID_ARGUMENT]not found entity, query-id:ddaa247752b94127-869b1a92d3b263d9
W20250428 08:51:57.580899 10280 ref_count_closure.h:119] RPC meet error status: [INVALID_ARGUMENT]PStatus: (172.16.24.145)[INVALID_ARGUMENT]not found entity, query-id:e1bf209c410745c6-82913cc2f679002d
W20250428 08:51:57.583657 10280 ref_count_closure.h:119] RPC meet error status: [INVALID_ARGUMENT]PStatus: (172.16.24.145)[INVALID_ARGUMENT]not found entity, query-id:e1bf209c410745c6-82913cc2f679002d
W20250428 08:51:57.586469 10312 ref_count_closure.h:119] RPC meet error status: [INVALID_ARGUMENT]PStatus: (172.16.24.145)[INVALID_ARGUMENT]not found entity, query-id:e1bf209c410745c6-82913cc2f679002d
W20250428 09:04:30.897217 10449 ref_count_closure.h:119] RPC meet error status: [INVALID_ARGUMENT]PStatus: (172.16.24.145)[INVALID_ARGUMENT]not found entity, query-id:b44df78add1d465f-95a4d1d0eb116abd
W20250428 09:04:30.900873 10263 ref_count_closure.h:119] RPC meet error status: [INVALID_ARGUMENT]PStatus: (172.16.24.145)[INVALID_ARGUMENT]not found entity, query-id:b44df78add1d465f-95a4d1d0eb116abd
W20250428 09:04:30.903903 10263 ref_count_closure.h:119] RPC meet error status: [INVALID_ARGUMENT]PStatus: (172.16.24.145)[INVALID_ARGUMENT]not found entity, query-id:b44df78add1d465f-95a4d1d0eb116abd
W20250428 09:04:30.925575 10395 ref_count_closure.h:119] RPC meet error status: [INVALID_ARGUMENT]PStatus: (172.16.24.145)[INVALID_ARGUMENT]not found entity, query-id:42975e0e5bdb475f-ab1d903cfaefb058
W20250428 09:04:30.927440 10200 ref_count_closure.h:119] RPC meet error status: [INVALID_ARGUMENT]PStatus: (172.16.24.145)[INVALID_ARGUMENT]not found entity, query-id:42975e0e5bdb475f-ab1d903cfaefb058
W20250428 09:04:30.929565 10218 ref_count_closure.h:119] RPC meet error status: [INVALID_ARGUMENT]PStatus: (172.16.24.145)[INVALID_ARGUMENT]not found entity, query-id:42975e0e5bdb475f-ab1d903cfaefb058
W20250428 09:11:30.128854 10269 ref_count_closure.h:119] RPC meet error status: [INVALID_ARGUMENT]PStatus: (172.16.24.145)[INVALID_ARGUMENT]query-id: 16d09a3fe9d44437-a345c0d8211acf88
W20250428 09:11:30.130993 10294 ref_count_closure.h:119] RPC meet error status: [INVALID_ARGUMENT]PStatus: (172.16.24.152)[INVALID_ARGUMENT]not found entity, query-id:796fbc0f52fd4be0-a929177d49496196
W20250428 09:11:30.132566 10235 ref_count_closure.h:119] RPC meet error status: [INVALID_ARGUMENT]PStatus: (172.16.24.152)[INVALID_ARGUMENT]not found entity, query-id:796fbc0f52fd4be0-a929177d49496196
W20250428 09:11:30.134981 10418 ref_count_closure.h:119] RPC meet error status: [INVALID_ARGUMENT]PStatus: (172.16.24.152)[INVALID_ARGUMENT]not found entity, query-id:796fbc0f52fd4be0-a929177d49496196
W20250428 09:29:23.012358 10269 ref_count_closure.h:119] RPC meet error status: [INVALID_ARGUMENT]PStatus: (172.16.24.152)[INVALID_ARGUMENT]not found entity, query-id:7406334ae0f44e97-bcec6477ed8b73ce
W20250428 09:29:23.013780 10442 ref_count_closure.h:119] RPC meet error status: [INVALID_ARGUMENT]PStatus: (172.16.24.152)[INVALID_ARGUMENT]not found entity, query-id:7406334ae0f44e97-bcec6477ed8b73ce
W20250428 09:29:23.015049 10380 ref_count_closure.h:119] RPC meet error status: [INVALID_ARGUMENT]PStatus: (172.16.24.152)[INVALID_ARGUMENT]not found entity, query-id:7406334ae0f44e97-bcec6477ed8b73ce
W20250428 09:34:22.047510 10250 ref_count_closure.h:119] RPC meet error status: [INVALID_ARGUMENT]PStatus: (172.16.24.152)[INVALID_ARGUMENT]not found entity, query-id:94bab1b27e3a4c79-8b34cf2044046ee3
W20250428 09:34:22.048712 10448 ref_count_closure.h:119] RPC meet error status: [INVALID_ARGUMENT]PStatus: (172.16.24.152)[INVALID_ARGUMENT]not found entity, query-id:94bab1b27e3a4c79-8b34cf2044046ee3
W20250428 09:34:22.049530 10274 ref_count_closure.h:119] RPC meet error status: [INVALID_ARGUMENT]PStatus: (172.16.24.152)[INVALID_ARGUMENT]not found entity, query-id:94bab1b27e3a4c79-8b34cf2044046ee3
W20250428 09:36:18.415361 10427 ref_count_closure.h:119] RPC meet error status: [INVALID_ARGUMENT]PStatus: (172.16.24.145)[INVALID_ARGUMENT]not found entity, query-id:42fd4f45fdeb4547-99003bee6d32f1ce
W20250428 09:36:18.417569 10282 ref_count_closure.h:119] RPC meet error status: [INVALID_ARGUMENT]PStatus: (172.16.24.145)[INVALID_ARGUMENT]not found entity, query-id:42fd4f45fdeb4547-99003bee6d32f1ce
W20250428 09:36:18.419951 10241 ref_count_closure.h:119] RPC meet error status: [INVALID_ARGUMENT]PStatus: (172.16.24.145)[INVALID_ARGUMENT]not found entity, query-id:42fd4f45fdeb4547-99003bee6d32f1ce
W20250428 09:39:34.203712 10302 ref_count_closure.h:119] RPC meet error status: [INVALID_ARGUMENT]PStatus: (172.16.24.152)[INVALID_ARGUMENT]query-id: 78ba3a05ce894dcf-98033037044d7a54
W20250428 09:39:34.203722 10405 ref_count_closure.h:119] RPC meet error status: [INVALID_ARGUMENT]PStatus: (172.16.24.145)[INVALID_ARGUMENT]query-id: 78ba3a05ce894dcf-98033037044d7a54
查看/proc/*/fd/内容,发现都是最后修改日期都是在24号(24号做了升级2.1.8->2.1.9)