CCR 上游Doris fe master 切换,导致 fullsync

Viewed 7

1.问题描述:
创建库级别ccr同步任务后,ccr正常启动全量同步,然后进入增量同步阶段,一切正常。但是当上游doris fe master切换至其他节点后,ccr触发了fullsync,重新拉取数据。
因为数据量较多,同步时间长,且对生产影响较大;

2.背景:
上游Doris版本:2.1.11-x64;
下游Doris版本:2.1.11-arm64;
CCR版本:ccr-syncer-3.0.6-rc05-arm64;
同步库大小:900张表,20TB,tablet 10w以下;

3.报错信息:
[2026-05-14 09:33:51.786] WARN call [:0] error: GetBinlog error: remote or network error: get connection error: dial tcp :0: connection has been closed by peer, req: TGetBinlogRequest({Cluster: User:0x40001a8378 Passwd:0x40001a8388 Db:0x40001a83a8 Table: TableId: UserIp: Token: PrevCommitSeq:0x400082e928 NumAcquired:0x400082e930}): [rpc] remote or network error: get connection error: dial tcp :0: connection has been closed by peer, try next addr job=CCR_PROD_ZHBB line=rpc/fe.go:259
[2026-05-14 09:33:51.786] WARN call [:0] error: GetBinlog error: remote or network error: get connection error: dial tcp :0: connection has been closed by peer, req: TGetBinlogRequest({Cluster: User:0x40007d2078 Passwd:0x40007d2088 Db:0x40007d20a8 Table: TableId: UserIp: Token: PrevCommitSeq:0x400906b188 NumAcquired:0x400906b190}): [rpc] remote or network error: get connection error: dial tcp :0: connection has been closed by peer, try next addr job=CCR_PROD_DW line=rpc/fe.go:259
[2026-05-14 09:33:51.786] WARN call [:0] error: GetBinlogLag error: remote or network error: get connection error: dial tcp :0: connection has been closed by peer, req: TGetBinlogRequest({Cluster: User:0x4003b04678 Passwd:0x4003b04688 Db:0x4003b046a8 Table: TableId: UserIp: Token: PrevCommitSeq:0x400906b158 NumAcquired:}): [rpc] remote or network error: get connection error: dial tcp :0: connection has been closed by peer, try next addr line=rpc/fe.go:259
[2026-05-14 09:33:51.887] INFO consume prev txn id: 37099393 job=CCR_PROD_ZHBB line=ccr/job_pipeline.go:483
[2026-05-14 09:33:51.900] INFO begin txn 37099394, label: ccrj-b277:db_sync:129802:44566176:9813974458, db: 44566176 job=CCR_PROD_DW line=ccr/job_pipeline.go:680
[2026-05-14 09:33:52.095] INFO txn 37099394 ingest binlog: run 1 tablet ingest jobs line=ccr/ingest_binlog_job.go:720
[2026-05-14 09:33:52.095] INFO begin txn 37099395, label: ccrj-5b09:db_sync:129802:44566176:9813974460, db: 44566176 job=CCR_PROD_DW line=ccr/job_pipeline.go:680
[2026-05-14 09:33:52.105] INFO txn 37099395 ingest binlog: run 1 tablet ingest jobs line=ccr/ingest_binlog_job.go:720
[2026-05-14 09:33:52.106] INFO consume prev txn id: 37099388 job=CCR_PROD_DW line=ccr/job_pipeline.go:483
[2026-05-14 09:33:52.121] INFO commit txn 37099394 success, commit seq: 9813974458 job=CCR_PROD_DW line=ccr/job_pipeline.go:808
[2026-05-14 09:33:52.133] INFO commit txn 37099395 success, commit seq: 9813974460 job=CCR_PROD_DW line=ccr/job_pipeline.go:808
[2026-05-14 09:33:52.141] INFO begin txn 37099396, label: ccrj-2780:db_sync:129802:44566176:9813974485, db: 44566176 job=CCR_PROD_DW line=ccr/job_pipeline.go:680
[2026-05-14 09:33:52.147] WARN txn 37099396 ingest binlog: prepare table 100598113 failed, err: [meta] index ids is empty, dropped tables map[], dropped partitions map[], dropped indexes map[] job=CCR_PROD_DW line=ccr/ingest_binlog_job.go:711
[2026-05-14 09:33:52.147] WARN a meta error occurred, retry to handle upsert binlog again, commitSeq: 9813974485 job=CCR_PROD_DW line=ccr/job_pipeline.go:740
[2026-05-14 09:33:52.149] WARN txn 37099396 ingest binlog: prepare table 100598113 failed, err: [meta] index ids is empty, dropped tables map[], dropped partitions map[], dropped indexes map[] job=CCR_PROD_DW line=ccr/ingest_binlog_job.go:711
[2026-05-14 09:33:52.149] WARN job sync failed, job: CCR_PROD_DW, err: [meta] index ids is empty
github.com/selectdb/ccr_syncer/pkg/ccr.(*IngestBinlogJob).preparePartition
/root/ccr-syncer/pkg/ccr/ingest_binlog_job.go:487
github.com/selectdb/ccr_syncer/pkg/ccr.(*IngestBinlogJob).prepareTable
/root/ccr-syncer/pkg/ccr/ingest_binlog_job.go:678
github.com/selectdb/ccr_syncer/pkg/ccr.(*IngestBinlogJob).prepareTabletIngestJobs
/root/ccr-syncer/pkg/ccr/ingest_binlog_job.go:707
github.com/selectdb/ccr_syncer/pkg/ccr.(*IngestBinlogJob).Prepare
/root/ccr-syncer/pkg/ccr/ingest_binlog_job.go:901
github.com/selectdb/ccr_syncer/pkg/ccr.(*Job).prepareIngestJob
/root/ccr-syncer/pkg/ccr/job_pipeline.go:769
github.com/selectdb/ccr_syncer/pkg/ccr.(*Job).prepareIngestJobWithRetry
/root/ccr-syncer/pkg/ccr/job_pipeline.go:741
github.com/selectdb/ccr_syncer/pkg/ccr.(*Job).launchIngestJob
/root/ccr-syncer/pkg/ccr/job_pipeline.go:693
github.com/selectdb/ccr_syncer/pkg/ccr.(*Job).launchTxn
/root/ccr-syncer/pkg/ccr/job_pipeline.go:537
github.com/selectdb/ccr_syncer/pkg/ccr.(*Job).pipelineSync
/root/ccr-syncer/pkg/ccr/job_pipeline.go:251
github.com/selectdb/ccr_syncer/pkg/ccr.(*Job).incrementalSync
/root/ccr-syncer/pkg/ccr/job.go:3736
github.com/selectdb/ccr_syncer/pkg/ccr.(*Job).dbSync
/root/ccr-syncer/pkg/ccr/job.go:3888
github.com/selectdb/ccr_syncer/pkg/ccr.(*Job).sync
/root/ccr-syncer/pkg/ccr/job.go:3936
github.com/selectdb/ccr_syncer/pkg/ccr.(*Job).run
/root/ccr-syncer/pkg/ccr/job.go:3995
github.com/selectdb/ccr_syncer/pkg/ccr.(*Job).Run
/root/ccr-syncer/pkg/ccr/job.go:4123
github.com/selectdb/ccr_syncer/pkg/ccr.(*JobManager).runJob.func1
/root/ccr-syncer/pkg/ccr/job_manager.go:184
runtime.goexit
[2026-05-14 09:33:52.149] WARN receive meta category error, make new snapshot, job: CCR_PROD_DW, err: [meta] index ids is empty job=CCR_PROD_DW line=ccr/job.go:3958
[2026-05-14 09:33:52.149] INFO new snapshot, commitSeq: 9813974460, prevCommitSeq: 9813974460, prevSyncState: DBIncrementalSync, prevSubSyncState: LaunchTransaction job=CCR_PROD_DW line=ccr/job.go:4007
[2026-05-14 09:33:53.597] INFO fullsync status: create snapshot with prefix ccrs_CCR_PROD_DW_1778668141 job=CCR_PROD_DW line=ccr/job.go:973
[2026-05-14 09:33:53.694] INFO fullsync status: create snapshot ccrs_CCR_PROD_DW_1778668141_1778722433 job=CCR_PROD_DW line=ccr/job.go:1019
[2026-05-14 09:33:53.694] INFO create snapshot PROD_DW.ccrs_CCR_PROD_DW_1778668141_1778722433, backup snapshot sql: BACKUP SNAPSHOT PROD_DW.ccrs_CCR_PROD_DW_1778668141_1778722433 TO __keep_on_local__ PROPERTIES ("type" = "full") job=CCR_PROD_DW line=base/spec.go:771

0 Answers