Logstash写入doris表偶发报错 detailMessage = Label [xxx] has already been used

Viewed 2

doris版本:3.0.7
问题描述:Logstash通过stream load写入doris表中,并发量大的时候偶然会出现报错,detailMessage = Label [xxx] has already been used,导致导入失败。标签是由doris-logstash插件自动生成的,规则是logstash_{表名}{时间戳}{随机的uuid}_{随机数},理论上不会出现重复。想问一下是否有可能是因为上一次streamload在某一个阶段的执行时间超时,导致数据重放,造成冲突,以及有没有办法进行优化。

Logstash相关的日志如下,Logstash日志中没有找到占用这个标签的任务的成功导入记录

2025-10-28T15:45:32,067][INFO ][logstash.outputs.doris   ][main][45a90c3f32cb58c6caf712257c806e18b8ee816e6e0dd3c3882eccdff87cb075] doris stream load response:
{
    "TxnId": -1,
    "Label": "node19_log_analysis_mbb_mobilebank_sit_20251028_154432_003_05f8022c-e5e6-478f-896f-7ea72a2f0640",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Label Already Exists",
    "ExistingJobStatus": "FINISHED",
    "Message": "[LABEL_ALREADY_EXISTS]TStatus: errCode = 2, detailMessage = Label [node19_log_analysis_mbb_mobilebank_sit_20251028_154432_003_05f8022c-e5e6-478f-896f-7ea72a2f0640] has already been used, relate to txn [90358995], status [VISIBLE].",
    "NumberTotalRows": 0,
    "NumberLoadedRows": 0,
    "NumberFilteredRows": 0,
    "NumberUnselectedRows": 0,
    "LoadBytes": 0,
    "LoadTimeMs": 0,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 0,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 0,
    "ReceiveDataTimeMs": 0,
    "CommitAndPublishTimeMs": 0
}

be中相关的日志如下,包括已存在标签的日志和目前导入报错的日志

I20251028  15:44:32.018785  3216190  stream_load_executor.cpp:72]  begin  to  execute  stream  load.  label=node19_log_analysis_mbb_mobilebank_sit_20251028_154432_003_05f8022c-e5e6-478f-896f-7ea72a2f0640,  txn_id=90358995,  query_id=1d421e3bc534e7cc-cd824cc1c3d301b4
I20251028  15:44:32.019673  3216190  stream_load.cpp:223]  finished  to  handle  HTTP  header,  id=1d421e3bc534e7cc-cd824cc1c3d301b4,  job_id=-1,  txn_id=90358995,  label=node19_log_analysis_mbb_mobilebank_sit_20251028_154432_003_05f8022c-e5e6-478f-896f-7ea72a2f0640,  elapse(s)=0
I20251028  15:44:32.024420  1963788  vtablet_writer.cpp:1016]  VNodeChannel[1754268843103-1749453084642],  load_id=1d421e3bc534e7cc-cd824cc1c3d301b4,  txn_id=90358995,  node=10.238.166.232:8060  mark  closed,  left  pending  batch  size:  1
I20251028  15:44:32.024431  1963788  vtablet_writer.cpp:1016]  VNodeChannel[1754268843103-1749453083391],  load_id=1d421e3bc534e7cc-cd824cc1c3d301b4,  txn_id=90358995,  node=10.238.166.72:8060  mark  closed,  left  pending  batch  size:  1
I20251028  15:44:32.024439  1963788  vtablet_writer.cpp:1016]  VNodeChannel[1754268843103-1749453084654],  load_id=1d421e3bc534e7cc-cd824cc1c3d301b4,  txn_id=90358995,  node=10.238.166.192:8060  mark  closed,  left  pending  batch  size:  1
I20251028  15:44:32.025534  3214263  tablets_channel.cpp:283]  txn  90358995:  close  tablets  channel  of  index  1754268843103  ,  sender  id:  0,  backend  1749453083391,  remain  senders:  0
I20251028  15:45:27.790774  3214263  load_channel.cpp:243]  txn  90358995  closed  tablets_channel  1754268843103
I20251028  15:45:27.792523  1963788  vtablet_writer.cpp:1676]  finished  to  close  olap  table  sink.  load_id=1d421e3bc534e7cc-cd824cc1c3d301b4,  txn_id=90358995,  node  add  batch  time(ms)/wait  execution  time(ms)/close  time(ms)/num:  {1749453084654:(16)(0)(55768)(1)}  {1749453083391:(55766)(0)(55768)(1)}  {1749453084642:(49)(0)(51)(1)}  
I20251028  15:45:27.822113  1817172  task_worker_pool.cpp:425]  successfully  submit  task|type=PUBLISH_VERSION|signature=90358995
I20251028  15:45:27.861884  3214036  engine_publish_version_task.cpp:488]  publish  version  successfully  on  tablet,  table_id=1750739678080,  tablet=1756389123855,  transaction_id=90358995,  version=62234,  num_rows=0,  res=[OK],  cost:  39641(us)  
I20251028  15:45:27.862105  3214027  engine_publish_version_task.cpp:488]  publish  version  successfully  on  tablet,  table_id=1750739678080,  tablet=1756389123861,  transaction_id=90358995,  version=62234,  num_rows=0,  res=[OK],  cost:  39819(us)  
I20251028  15:45:27.867695  3214054  engine_publish_version_task.cpp:488]  publish  version  successfully  on  tablet,  table_id=1750739678080,  tablet=1756389123849,  transaction_id=90358995,  version=62234,  num_rows=0,  res=[OK],  cost:  45494(us)  
I20251028  15:45:27.869901  3214106  engine_publish_version_task.cpp:370]  finish  to  publish  version  on  transaction.transaction_id=90358995,  cost(us):  47773,  error_tablet_size=0,  res=[OK]
I20251028  15:45:27.869944  3214106  task_worker_pool.cpp:1967]  successfully  publish  version|signature=90358995|transaction_id=90358995|tablets_num=3|cost(s)=0
W20251028  15:45:32.065254  3216214  stream_load_executor.cpp:204]  begin  transaction  failed,  errmsg=[LABEL_ALREADY_EXISTS]TStatus:  errCode  =  2,  detailMessage  =  Label  [node19_log_analysis_mbb_mobilebank_sit_20251028_154432_003_05f8022c-e5e6-478f-896f-7ea72a2f0640]  has  already  been  used,  relate  to  txn  [90358995],  status  [VISIBLE].id=c24c0aa45a845362-1fcb02a9174cbc8d,  job_id=-1,  txn_id=-1,  label=node19_log_analysis_mbb_mobilebank_sit_20251028_154432_003_05f8022c-e5e6-478f-896f-7ea72a2f0640,  elapse(s)=0
W20251028  15:45:33.073698  3216168  stream_load_executor.cpp:204]  begin  transaction  failed,  errmsg=[LABEL_ALREADY_EXISTS]TStatus:  errCode  =  2,  detailMessage  =  Label  [node19_log_analysis_mbb_mobilebank_sit_20251028_154432_003_05f8022c-e5e6-478f-896f-7ea72a2f0640]  has  already  been  used,  relate  to  txn  [90358995],  status  [VISIBLE].id=104f7a566059d118-53508272af9b3595,  job_id=-1,  txn_id=-1,  label=node19_log_analysis_mbb_mobilebank_sit_20251028_154432_003_05f8022c-e5e6-478f-896f-7ea72a2f0640,  elapse(s)=0
I20251028  15:45:36.802642  3216190  stream_load.cpp:140]  finished  to  execute  stream  load.  label=node19_log_analysis_mbb_mobilebank_sit_20251028_154432_003_05f8022c-e5e6-478f-896f-7ea72a2f0640,  txn_id=90358995,  query_id=1d421e3bc534e7cc-cd824cc1c3d301b4,  load_cost_ms=55930,  receive_data_cost_ms=9,  read_data_cost_ms=0,  write_data_cost_ms=55774,  commit_and_publish_txn_cost_ms=146,  number_total_rows=17,  number_loaded_rows=17,  receive_bytes=11316,  loaded_bytes=8579,  error_url=
0 Answers