集群夯住，查询报错report error status: PStatus: timeout when waiting for send fragments rpc,

Viewed 22
3.0

W20251222 10:28:28.375526 991943 status.h:444] meet error status: [INTERNAL_ERROR]PStatus: timeout when waiting for send fragments rpc, query timeout:900
, left timeout for this operation:30

    0#  doris::Status doris::Status::create<true>(doris::PStatus const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../i

nclude/c++/11/bits/basic_string.h:187
1# std::_Function_handler<void (), doris::PInternalService::cancel_plan_fragment(google::protobuf::RpcController*, doris::PCancelPlanFragmentReq
uest const*, doris::PCancelPlanFragmentResult*, google::protobuf::Closure*)::$_0>::_M_invoke(std::_Any_data const&) at /home/zcp/repo_center/doris_releas
e/doris/be/src/service/internal_service.cpp:0
2# doris::WorkThreadPool::work_thread(int) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits
/atomic_base.h:646
3# execute_native_thread_routine at /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/unique_ptr.h:85
4# ?
5# ?
I20251222 10:28:28.375582 991943 internal_service.cpp:659] Cancel query 5747302d9bc64f7a-96e09b5a06bba166, reason: [INTERNAL_ERROR]PStatus: timeout when
waiting for send fragments rpc, query timeout:900, left timeout for this operation:30

    0#  doris::Status doris::Status::create<true>(doris::PStatus const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:187
    1#  std::_Function_handler<void (), doris::PInternalService::cancel_plan_fragment(google::protobuf::RpcController*, doris::PCancelPlanFragmentRequest const*, doris::PCancelPlanFragmentResult*, google::protobuf::Closure*)::$_0>::_M_invoke(std::_Any_data const&) at /home/zcp/repo_center/doris_release/doris/be/src/service/internal_service.cpp:0
    2#  doris::WorkThreadPool<false>::work_thread(int) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:646
    3#  execute_native_thread_routine at /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/unique_ptr.h:85
    4#  ?
    5#  ?

I20251222 10:28:28.375582 991943 internal_service.cpp:659] Cancel query 5747302d9bc64f7a-96e09b5a06bba166, reason: [INTERNAL_ERROR]PStatus: timeout when
waiting for send fragments rpc, query timeout:900, left timeout for this operation:30

    0#  doris::Status doris::Status::create<true>(doris::PStatus const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../i

nclude/c++/11/bits/basic_string.h:187
1# std::_Function_handler<void (), doris::PInternalService::cancel_plan_fragment(google::protobuf::RpcController*, doris::PCancelPlanFragmentReq
uest const*, doris::PCancelPlanFragmentResult*, google::protobuf::Closure*)::$_0>::_M_invoke(std::_Any_data const&) at /home/zcp/repo_center/doris_releas
e/doris/be/src/service/internal_service.cpp:0
2# doris::WorkThreadPool::work_thread(int) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits
/atomic_base.h:646
3# execute_native_thread_routine at /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/unique_ptr.h:85
4# ?
5# ?
I20251222 10:28:28.375643 991943 pipeline_fragment_context.cpp:171] PipelineFragmentContext::cancel|query_id=5747302d9bc64f7a-96e09b5a06bba166|fragment_id=1|reason=[INTERNAL_ERROR]PStatus: timeout when waiting for send fragments rpc, query timeout:900, left timeout for this operation:30

    0#  doris::Status doris::Status::create<true>(doris::PStatus const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:187
    1#  std::_Function_handler<void (), doris::PInternalService::cancel_plan_fragment(google::protobuf::RpcController*, doris::PCancelPlanFragmentRequest const*, doris::PCancelPlanFragmentResult*, google::protobuf::Closure*)::$_0>::_M_invoke(std::_Any_data const&) at /home/zcp/repo_center/doris_release/doris/be/src/service/internal_service.cpp:0
    2#  doris::WorkThreadPool<false>::work_thread(int) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:646
    3#  execute_native_thread_routine at /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/unique_ptr.h:85
    4#  ?
    5#  ?

W20251222 10:28:28.375654 991943 pipeline_fragment_context.cpp:202] PipelineFragmentContext cancel instance: 5747302d9bc64f7a-96e09b5a06bba168
I20251222 10:28:28.375664 991943 fragment_mgr.cpp:895] Query 5747302d9bc64f7a-96e09b5a06bba166 is cancelled and removed. Reason: [INTERNAL_ERROR]PStatus: timeout when waiting for send fragments rpc, query timeout:900, left timeout for this operation:30

    0#  doris::Status doris::Status::create<true>(doris::PStatus const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:187
    1#  std::_Function_handler<void (), doris::PInternalService::cancel_plan_fragment(google::protobuf::RpcController*, doris::PCancelPlanFragmentRequest const*, doris::PCancelPlanFragmentResult*, google::protobuf::Closure*)::$_0>::_M_invoke(std::_Any_data const&) at /home/zcp/repo_center/doris_release/doris/be/src/service/internal_service.cpp:0
    2#  doris::WorkThreadPool<false>::work_thread(int) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:646
    3#  execute_native_thread_routine at /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/unique_ptr.h:85
    4#  ?
    5#  ?

I20251222 10:28:28.375728 995196 pipeline_fragment_context.cpp:171] PipelineFragmentContext::cancel|query_id=5747302d9bc64f7a-96e09b5a06bba166|fragment_id=1|reason=[INTERNAL_ERROR]PStatus: timeout when waiting for send fragments rpc, query timeout:900, left timeout for this operation:30

    0#  doris::Status doris::Status::create<true>(doris::PStatus const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:187
    1#  std::_Function_handler<void (), doris::PInternalService::cancel_plan_fragment(google::protobuf::RpcController*, doris::PCancelPlanFragmentRequest const*, doris::PCancelPlanFragmentResult*, google::protobuf::Closure*)::$_0>::_M_invoke(std::_Any_data const&) at /home/zcp/repo_center/doris_release/doris/be/src/service/internal_service.cpp:0
    2#  doris::WorkThreadPool<false>::work_thread(int) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:646
    3#  execute_native_thread_routine at /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/unique_ptr.h:85
    4#  ?
    5#  ?
2 Answers
  1. 具体的版本是多少？show variables like "%versio_comment%";
  2. 是每个sql 查询都报这个错，还是固定的sql？
  3. 维持时间大概多久？当时集群的压力如何，cpu io 这些
阿渊@SelectDB (没回帖直接加我主页微信)9035

20251222-111404.jpg20251222-111412.jpg

hht1
Related Questions
集群夯住，查询报错report error status: PStatus: timeout when waiting for send fragments rpc,
2 answers
3.0.8运行一段时间版本报错 ：Reach limit of connections. Total: 1024, User: 100, Current: 100，什么原因会导致这样，要改哪个配置？
2 answers
3.x版本webUI点击具体表，报unexpected exception
1 answers
Doris主键点查比Clickhouse慢一个数量级原因
1 answers
使用doris cluster manater升级从3.0.7到3.1.0
1 answers
官网文档为什么一直无法下载？
1 answers