doris版本:3.0.7
问题描述:Logstash通过stream load写入doris表中,并发量大的时候偶然会出现报错,detailMessage = Label [xxx] has already been used,导致导入失败。标签是由doris-logstash插件自动生成的,规则是logstash_{表名}{时间戳}{随机的uuid}_{随机数},理论上不会出现重复。想问一下是否有可能是因为上一次streamload在某一个阶段的执行时间超时,导致数据重放,造成冲突,以及有没有办法进行优化。
Logstash相关的日志如下,Logstash日志中没有找到占用这个标签的任务的成功导入记录
2025-10-28T15:45:32,067][INFO ][logstash.outputs.doris ][main][45a90c3f32cb58c6caf712257c806e18b8ee816e6e0dd3c3882eccdff87cb075] doris stream load response:
{
"TxnId": -1,
"Label": "node19_log_analysis_mbb_mobilebank_sit_20251028_154432_003_05f8022c-e5e6-478f-896f-7ea72a2f0640",
"Comment": "",
"TwoPhaseCommit": "false",
"Status": "Label Already Exists",
"ExistingJobStatus": "FINISHED",
"Message": "[LABEL_ALREADY_EXISTS]TStatus: errCode = 2, detailMessage = Label [node19_log_analysis_mbb_mobilebank_sit_20251028_154432_003_05f8022c-e5e6-478f-896f-7ea72a2f0640] has already been used, relate to txn [90358995], status [VISIBLE].",
"NumberTotalRows": 0,
"NumberLoadedRows": 0,
"NumberFilteredRows": 0,
"NumberUnselectedRows": 0,
"LoadBytes": 0,
"LoadTimeMs": 0,
"BeginTxnTimeMs": 0,
"StreamLoadPutTimeMs": 0,
"ReadDataTimeMs": 0,
"WriteDataTimeMs": 0,
"ReceiveDataTimeMs": 0,
"CommitAndPublishTimeMs": 0
}
be中相关的日志如下,包括已存在标签的日志和目前导入报错的日志
I20251028 15:44:32.018785 3216190 stream_load_executor.cpp:72] begin to execute stream load. label=node19_log_analysis_mbb_mobilebank_sit_20251028_154432_003_05f8022c-e5e6-478f-896f-7ea72a2f0640, txn_id=90358995, query_id=1d421e3bc534e7cc-cd824cc1c3d301b4
I20251028 15:44:32.019673 3216190 stream_load.cpp:223] finished to handle HTTP header, id=1d421e3bc534e7cc-cd824cc1c3d301b4, job_id=-1, txn_id=90358995, label=node19_log_analysis_mbb_mobilebank_sit_20251028_154432_003_05f8022c-e5e6-478f-896f-7ea72a2f0640, elapse(s)=0
I20251028 15:44:32.024420 1963788 vtablet_writer.cpp:1016] VNodeChannel[1754268843103-1749453084642], load_id=1d421e3bc534e7cc-cd824cc1c3d301b4, txn_id=90358995, node=10.238.166.232:8060 mark closed, left pending batch size: 1
I20251028 15:44:32.024431 1963788 vtablet_writer.cpp:1016] VNodeChannel[1754268843103-1749453083391], load_id=1d421e3bc534e7cc-cd824cc1c3d301b4, txn_id=90358995, node=10.238.166.72:8060 mark closed, left pending batch size: 1
I20251028 15:44:32.024439 1963788 vtablet_writer.cpp:1016] VNodeChannel[1754268843103-1749453084654], load_id=1d421e3bc534e7cc-cd824cc1c3d301b4, txn_id=90358995, node=10.238.166.192:8060 mark closed, left pending batch size: 1
I20251028 15:44:32.025534 3214263 tablets_channel.cpp:283] txn 90358995: close tablets channel of index 1754268843103 , sender id: 0, backend 1749453083391, remain senders: 0
I20251028 15:45:27.790774 3214263 load_channel.cpp:243] txn 90358995 closed tablets_channel 1754268843103
I20251028 15:45:27.792523 1963788 vtablet_writer.cpp:1676] finished to close olap table sink. load_id=1d421e3bc534e7cc-cd824cc1c3d301b4, txn_id=90358995, node add batch time(ms)/wait execution time(ms)/close time(ms)/num: {1749453084654:(16)(0)(55768)(1)} {1749453083391:(55766)(0)(55768)(1)} {1749453084642:(49)(0)(51)(1)}
I20251028 15:45:27.822113 1817172 task_worker_pool.cpp:425] successfully submit task|type=PUBLISH_VERSION|signature=90358995
I20251028 15:45:27.861884 3214036 engine_publish_version_task.cpp:488] publish version successfully on tablet, table_id=1750739678080, tablet=1756389123855, transaction_id=90358995, version=62234, num_rows=0, res=[OK], cost: 39641(us)
I20251028 15:45:27.862105 3214027 engine_publish_version_task.cpp:488] publish version successfully on tablet, table_id=1750739678080, tablet=1756389123861, transaction_id=90358995, version=62234, num_rows=0, res=[OK], cost: 39819(us)
I20251028 15:45:27.867695 3214054 engine_publish_version_task.cpp:488] publish version successfully on tablet, table_id=1750739678080, tablet=1756389123849, transaction_id=90358995, version=62234, num_rows=0, res=[OK], cost: 45494(us)
I20251028 15:45:27.869901 3214106 engine_publish_version_task.cpp:370] finish to publish version on transaction.transaction_id=90358995, cost(us): 47773, error_tablet_size=0, res=[OK]
I20251028 15:45:27.869944 3214106 task_worker_pool.cpp:1967] successfully publish version|signature=90358995|transaction_id=90358995|tablets_num=3|cost(s)=0
W20251028 15:45:32.065254 3216214 stream_load_executor.cpp:204] begin transaction failed, errmsg=[LABEL_ALREADY_EXISTS]TStatus: errCode = 2, detailMessage = Label [node19_log_analysis_mbb_mobilebank_sit_20251028_154432_003_05f8022c-e5e6-478f-896f-7ea72a2f0640] has already been used, relate to txn [90358995], status [VISIBLE].id=c24c0aa45a845362-1fcb02a9174cbc8d, job_id=-1, txn_id=-1, label=node19_log_analysis_mbb_mobilebank_sit_20251028_154432_003_05f8022c-e5e6-478f-896f-7ea72a2f0640, elapse(s)=0
W20251028 15:45:33.073698 3216168 stream_load_executor.cpp:204] begin transaction failed, errmsg=[LABEL_ALREADY_EXISTS]TStatus: errCode = 2, detailMessage = Label [node19_log_analysis_mbb_mobilebank_sit_20251028_154432_003_05f8022c-e5e6-478f-896f-7ea72a2f0640] has already been used, relate to txn [90358995], status [VISIBLE].id=104f7a566059d118-53508272af9b3595, job_id=-1, txn_id=-1, label=node19_log_analysis_mbb_mobilebank_sit_20251028_154432_003_05f8022c-e5e6-478f-896f-7ea72a2f0640, elapse(s)=0
I20251028 15:45:36.802642 3216190 stream_load.cpp:140] finished to execute stream load. label=node19_log_analysis_mbb_mobilebank_sit_20251028_154432_003_05f8022c-e5e6-478f-896f-7ea72a2f0640, txn_id=90358995, query_id=1d421e3bc534e7cc-cd824cc1c3d301b4, load_cost_ms=55930, receive_data_cost_ms=9, read_data_cost_ms=0, write_data_cost_ms=55774, commit_and_publish_txn_cost_ms=146, number_total_rows=17, number_loaded_rows=17, receive_bytes=11316, loaded_bytes=8579, error_url=