使用x2doris从 doris v1.2 迁移数据到 doris v3.0
v1.2集群有3台BE,有些表无法同步完整,大部分时候进度33%,偶尔重试时进度67%,然后迁移任务的状态就变成「完成」
看日志只从一个BE上同步了数据,连接到另两个BE时,连上后直接就close了
x2doris到各个BE的访问是通的,可以看到connect success
2025-07-24 16:40:20 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:40:20 INFO ScalableIterator: buffer is cleared and batch iterator is closed
2025-07-24 16:40:31 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:40:31 INFO ScalableIterator: buffer is cleared and batch iterator is closed
2025-07-24 16:40:41 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:40:41 INFO ScalableIterator: buffer is cleared and batch iterator is closed
2025-07-24 16:40:46 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:40:46 INFO ScalableIterator: buffer is cleared and batch iterator is closed
2025-07-24 16:40:46 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:40:46 INFO BackendClient: CloseScanner to Doris BE 'Doris BE{host='10.128.5.49', port=9060}' success.
2025-07-24 16:40:46 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:40:46 INFO BackendClient: Closed a connection to Doris BE{host='10.128.5.49', port=9060}.
2025-07-24 16:40:46 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:40:46 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1218 bytes result sent to driver
2025-07-24 16:40:46 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:40:46 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1) (x2doris-0.x2doris-svc.doris.svc.cluster.local, executor driver, partition 1, ANY, 14640 bytes) taskResourceAssignments Map()
2025-07-24 16:40:46 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:40:46 INFO Executor: Running task 1.0 in stage 0.0 (TID 1)
2025-07-24 16:40:46 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:40:46 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 731491 ms on x2doris-0.x2doris-svc.doris.svc.cluster.local (executor driver) (1/3)
2025-07-24 16:40:46 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:40:46 INFO BackendClient: Success connect to Doris BE{host='10.128.3.21', port=9060}.
2025-07-24 16:40:53 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:40:53 INFO BackendClient: CloseScanner to Doris BE 'Doris BE{host='10.128.3.21', port=9060}' success.
2025-07-24 16:40:53 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:40:53 INFO BackendClient: Closed a connection to Doris BE{host='10.128.3.21', port=9060}.
2025-07-24 16:40:53 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:40:53 INFO Executor: Finished task 1.0 in stage 0.0 (TID 1). 1175 bytes result sent to driver
2025-07-24 16:40:53 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:40:53 INFO TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2) (x2doris-0.x2doris-svc.doris.svc.cluster.local, executor driver, partition 2, ANY, 14641 bytes) taskResourceAssignments Map()
2025-07-24 16:40:53 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:40:53 INFO Executor: Running task 2.0 in stage 0.0 (TID 2)
2025-07-24 16:40:53 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:40:53 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 7109 ms on x2doris-0.x2doris-svc.doris.svc.cluster.local (executor driver) (2/3)
2025-07-24 16:40:53 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:40:53 INFO BackendClient: Success connect to Doris BE{host='10.128.1.227', port=9060}.
2025-07-24 16:41:00 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:41:00 INFO BackendClient: CloseScanner to Doris BE 'Doris BE{host='10.128.1.227', port=9060}' success.
2025-07-24 16:41:00 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:41:00 INFO BackendClient: Closed a connection to Doris BE{host='10.128.1.227', port=9060}.
2025-07-24 16:41:00 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:41:00 INFO Executor: Finished task 2.0 in stage 0.0 (TID 2). 1175 bytes result sent to driver
2025-07-24 16:41:00 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:41:00 INFO TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 7037 ms on x2doris-0.x2doris-svc.doris.svc.cluster.local (executor driver) (3/3)
2025-07-24 16:41:00 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:41:00 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
2025-07-24 16:41:00 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:41:00 INFO DAGScheduler: ResultStage 0 (foreachPartition at DorisWriter.scala:66) finished in 745.819 s
2025-07-24 16:41:00 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:41:00 INFO DAGScheduler: Job 0 is finished. Cancelling potential speculative or zombie tasks for this job
2025-07-24 16:41:00 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:41:00 INFO TaskSchedulerImpl: Killing all running tasks in stage 0: Stage finished
2025-07-24 16:41:00 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:41:00 INFO DAGScheduler: Job 0 finished: foreachPartition at DorisWriter.scala:66, took 745.876163 s
2025-07-24 16:41:00 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:41:00 INFO SparkContext: Invoking stop() from shutdown hook
2025-07-24 16:41:00 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:41:00 INFO SparkUI: Stopped Spark web UI at http://x2doris-0.x2doris-svc.doris.svc.cluster.local:4041
2025-07-24 16:41:00 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:41:00 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
2025-07-24 16:41:00 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:41:00 INFO MemoryStore: MemoryStore cleared
2025-07-24 16:41:00 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:41:00 INFO BlockManager: BlockManager stopped
2025-07-24 16:41:00 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:41:00 INFO BlockManagerMaster: BlockManagerMaster stopped
2025-07-24 16:41:00 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:41:00 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
2025-07-24 16:41:00 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:41:00 INFO SparkContext: Successfully stopped SparkContext
2025-07-24 16:41:00 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:41:00 INFO ShutdownHookManager: Shutdown hook called
2025-07-24 16:41:00 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:41:00 INFO ShutdownHookManager: Deleting directory /tmp/spark-423e7bed-b004-4d13-b441-d11f34c45a58
2025-07-24 16:41:00 | INFO | launcher-proc-63 | org.apache.spark.launcher.app.doris:middle_order:64] 25/07/24 16:41:00 INFO ShutdownHookManager: Deleting directory /tmp/spark-ebf2c175-e047-4722-82b5-0aecc5edc2dd