问题描述:在Doris中使用SQL创建ROUTINE LOAD Kafka导入任务时,执行SQL后等很久,然后报错Failed to get real offsets of kafka topic。
Doris版本:doris-2.1.10
创建ROUTINE LOAD语句:
CREATE ROUTINE LOAD testdb.at_pmmodulepassworkcenter_dsg ON at_pmmodulepassworkcenter_dsg
PROPERTIES
(
"desired_concurrent_number"="3",
"format" = "json",
"strict_mode" = "false"
)
FROM KAFKA
(
"kafka_broker_list" = "**.**.**:**",
"kafka_topic" = "orc_mes_module_hs_logs",
"kafka_partitions" = "18",
"property.group.id" = "module_doris_zx",
"property.kafka_default_offsets" = "OFFSET_BEGINNING",
"property.security.protocol" = "SASL_PLAINTEXT",
"property.sasl.kerberos.service.name" = "kafka",
"property.sasl.kerberos.keytab" = "/data/doris/kafka/kerberos/krb5.keytab",
"property.sasl.kerberos.keytab"="/data/doris/kafka/kerberos/***.keytab",
"property.sasl.kerberos.principal" = "***@***.***"
);
报错:
[2025-10-22 14:54:36] [HY000][1105] errCode = 2, detailMessage = Failed to get real offsets of kafka topic: orc_mes_module_hs_logs. error: errCode = 2, detailMessage = Failed to get info
be.INFO中报错
W20251027 14:48:43.964395 42962 data_consumer.cpp:477] failed to get latest offset for partition: 18, err: Local: All broker connections are down
W20251027 14:48:43.964951 42962 status.h:415] meet error status: [INTERNAL_ERROR]failed to get latest offset for partition: 18, err: Local: All broker connections are down
0# doris::KafkaDataConsumer::get_real_offsets_for_partitions(std::vector<doris::PIntegerPair, std::allocator<doris::PIntegerPair> > const&, std::vector<doris::PIntegerPair, std::allocator<doris::PIntegerPair> >*, int) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:187
1# doris::RoutineLoadTaskExecutor::get_kafka_real_offsets_for_partitions(doris::PKafkaMetaProxyRequest const&, std::vector<doris::PIntegerPair, std::allocator<doris::PIntegerPair> >*, int) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:680
2# std::_Function_handler<void (), doris::PInternalServiceImpl::get_info(google::protobuf::RpcController*, doris::PProxyRequest const*, doris::PProxyResult*, google::protobuf::Closure*)::$_0>::_M_invoke(std::_Any_data const&) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:494
3# doris::ThreadPool::dispatch_thread() at /home/zcp/repo_center/doris_release/doris/be/src/util/threadpool.cpp:0
4# doris::Thread::supervise_thread(void*) at /var/local/ldb-toolchain/bin/../usr/include/pthread.h:562
5# start_thread
6# __clone
I20251027 14:48:43.965011 42962 data_consumer.h:94] kafka error: Local: Broker transport failure, event: **.**.**:9092/bootstrap: Disconnected: verify that security.protocol is correctly configured, broker might require SASL authentication (after 162ms in state UP)
I20251027 14:48:43.965025 42962 data_consumer.h:94] kafka error: Local: Broker transport failure, event: **.**.**:9092/bootstrap: Disconnected: verify that security.protocol is correctly configured, broker might require SASL authentication (after 100ms in state UP, 1 identical error(s) suppressed)
I20251027 14:48:43.965035 42962 data_consumer.h:94] kafka error: Local: Broker transport failure, event: kc003.center.longi:9092/bootstrap: Disconnected: verify that security.protocol is correctly configured, broker might require SASL authentication (after 221ms in state UP)