创建ROUTINE LOAD Kafka任务报错Failed to get real offsets of kafka topic

Viewed 55

问题描述:在Doris中使用SQL创建ROUTINE LOAD Kafka导入任务时,执行SQL后等很久,然后报错Failed to get real offsets of kafka topic。
Doris版本:doris-2.1.10
创建ROUTINE LOAD语句:

CREATE ROUTINE LOAD testdb.at_pmmodulepassworkcenter_dsg ON at_pmmodulepassworkcenter_dsg
        PROPERTIES
        (
            "desired_concurrent_number"="3",
            "format" = "json",
            "strict_mode" = "false"
        )
        FROM KAFKA
        (
            "kafka_broker_list" = "**.**.**:**",
            "kafka_topic" = "orc_mes_module_hs_logs",
            "kafka_partitions" = "18",
            "property.group.id" = "module_doris_zx",
            "property.kafka_default_offsets" = "OFFSET_BEGINNING",
            "property.security.protocol" = "SASL_PLAINTEXT",
            "property.sasl.kerberos.service.name" = "kafka",
            "property.sasl.kerberos.keytab" = "/data/doris/kafka/kerberos/krb5.keytab",
            "property.sasl.kerberos.keytab"="/data/doris/kafka/kerberos/***.keytab",
            "property.sasl.kerberos.principal" = "***@***.***"
        );

报错:

[2025-10-22 14:54:36] [HY000][1105] errCode = 2, detailMessage = Failed to get real offsets of kafka topic: orc_mes_module_hs_logs. error: errCode = 2, detailMessage = Failed to get info

be.INFO中报错

W20251027 14:48:43.964395 42962 data_consumer.cpp:477] failed to get latest offset for partition: 18, err: Local: All broker connections are down
W20251027 14:48:43.964951 42962 status.h:415] meet error status: [INTERNAL_ERROR]failed to get latest offset for partition: 18, err: Local: All broker connections are down

	0#  doris::KafkaDataConsumer::get_real_offsets_for_partitions(std::vector<doris::PIntegerPair, std::allocator<doris::PIntegerPair> > const&, std::vector<doris::PIntegerPair, std::allocator<doris::PIntegerPair> >*, int) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:187
	1#  doris::RoutineLoadTaskExecutor::get_kafka_real_offsets_for_partitions(doris::PKafkaMetaProxyRequest const&, std::vector<doris::PIntegerPair, std::allocator<doris::PIntegerPair> >*, int) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:680
	2#  std::_Function_handler<void (), doris::PInternalServiceImpl::get_info(google::protobuf::RpcController*, doris::PProxyRequest const*, doris::PProxyResult*, google::protobuf::Closure*)::$_0>::_M_invoke(std::_Any_data const&) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:494
	3#  doris::ThreadPool::dispatch_thread() at /home/zcp/repo_center/doris_release/doris/be/src/util/threadpool.cpp:0
	4#  doris::Thread::supervise_thread(void*) at /var/local/ldb-toolchain/bin/../usr/include/pthread.h:562
	5#  start_thread
	6#  __clone
I20251027 14:48:43.965011 42962 data_consumer.h:94] kafka error: Local: Broker transport failure, event: **.**.**:9092/bootstrap: Disconnected: verify that security.protocol is correctly configured, broker might require SASL authentication (after 162ms in state UP)
I20251027 14:48:43.965025 42962 data_consumer.h:94] kafka error: Local: Broker transport failure, event: **.**.**:9092/bootstrap: Disconnected: verify that security.protocol is correctly configured, broker might require SASL authentication (after 100ms in state UP, 1 identical error(s) suppressed)
I20251027 14:48:43.965035 42962 data_consumer.h:94] kafka error: Local: Broker transport failure, event: kc003.center.longi:9092/bootstrap: Disconnected: verify that security.protocol is correctly configured, broker might require SASL authentication (after 221ms in state UP)
1 Answers
  1. 当前的日志是现象,需要详细的日志堆栈进行的分析。
  2. 需要具体的版本号:show variables like "%version_comment%"; 查看。

你可以加我主页微信,发下日志