CREATE ROUTINE LOAD遇到的问题

Viewed 27

CREATE ROUTINE LOAD

FROM KAFKA
(
    "kafka_broker_list" = "10.15.8.72:9092",
    "kafka_topic" = "test",
    -- "property.group.id" = "console-consumer-38930",
    "property.kafka_default_offsets" = "OFFSET_BEGINNING"
    -- 分区和offsets
    -- "kafka_partitions"="0",
    -- "kafka_offsets" = "OFFSET_BEGINNING",
    -- 安全认证
    -- "property.security.protocol" = "ssl",
    -- "property.ssl.ca.location" = "FILE:ca.pem",
    -- "property.ssl.certificate.location" = "FILE:client.pem",
    -- "property.ssl.key.location" = "FILE:client.key",
    -- "property.ssl.key.password" = "ssl_passwd"
);

按照上面的配置进行创建,一直RUNNING,但数据导入不成功,执行SHOW ROUTINE LOAD,Progress和Lag都是0:0,问题是

failed to get latest partition offset. {}errCode = 2, detailMessage = Failed to get latest offsets of kafka topic: test. error: errCode = 2, detailMessage = Failed to get info

后面修改 ROUTINE LOAD,指定消费组,数据导入成功,但出现新的问题:

2025-10-27 18:15:04:[ANALYSIS_ERROR]TStatus: errCode = 2, detailMessage = txn 3016007 could not be COMMITTED while task 87a572f2041e4146-bac551221f63d0e6 has been aborted.

于是就暂停、再恢复,又出现问题:

failed to get latest partition offset. {}errCode = 2, detailMessage = Failed to get latest offsets of kafka topic: test. error: errCode = 2, detailMessage = Failed to get info

企业微信截图_17615609199350.png
现在卡在kafka导入doris这一步,有时直接创建就能成功,有时暂停后恢复才成功。
跪求大佬解决。(我玩虚拟机没遇到过这些问题,数据同步配置都很顺畅)

2 Answers

得看下BE的日志中的详细错误,拿着你的job_id 在be.INFO 中搜下

没指定消费者组group_id
企业微信截图_1761618729550.png
企业微信截图_17616187568106.png
企业微信截图_17616187744741.png
重新创建,指定消费者组
(补充:修改,添加消费者组报错,此外第一次创建的消费者组group_id,kafka查询does not exist,重新创建指定后可以查询到)

2025-10-28 10:17:28:[ANALYSIS_ERROR]TStatus: errCode = 2, detailMessage = txn 3045512 could not be COMMITTED while task bda7da1b20794620-92302d7a1daea029 has been aborted.

企业微信截图_17616192755057.png
企业微信截图_17616194969909.png
企业微信截图_17616195185564.png
因为txn_id一直显示-1,于是暂停kafka_test_job2,Other_Message

failed to get latest partition offset. {}errCode = 2, detailMessage = Failed to get latest offsets of kafka topic: test. error: errCode = 2, detailMessage = Failed to get info

仍然是-1。但目前测试情况,似乎要先指定消费者组,第一次同步会正常进行,但之后的任务就不行