X2Doris 迁移数据失败 字段类型无法识别问题

Viewed 4

使用X2Doris1.2版本从doris1.2.6迁移数据到doris2.1.11中,一部分表没有问题,另一部分表就报错,查看日志发现是由于遇到无法识别的聚合数据类型,可是我创建的表没有聚合类型的数据,不知道x2doris内部是如何判断识别字段类型的,如何解决?
报错日志如下:

2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] get schema from Fe: ---------------------------^M

2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: arch_id  : VARCHAR
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: personname  : VARCHAR
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: idcard  : VARCHAR
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: dt  : DATEV2
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: age  : INT
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: month_floor  : INT
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: gender_name  : STRING
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: bmi  : BIGINT
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: bmi_name  : STRING
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: waistline  : BIGINT
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: waistline_name  : STRING
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: diastolic_pressure_left  : BIGINT
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: diastolic_pressure_left_name  : STRING
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: systolic_pressure_left  : BIGINT
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: systolic_pressure_left_name  : STRING
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: diastolic_pressure_right  : BIGINT
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: systolic_pressure_right_name  : STRING
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: fbs_mmol  : BIGINT
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: fbs_mmol_name  : STRING
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: triglyceridevalue  : BIGINT
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: triglyceridevalue_name  : STRING
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: cholesterol_value  : BIGINT
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: cholesterol_value_name  : STRING
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: cholesterol_low  : BIGINT
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: cholesterol_low_name  : STRING
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: cholesterol_high  : BIGINT
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: cholesterol_high_name  : STRING
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: erythrocyte  : BIGINT
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: erythrocyte_name  : STRING
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: hemoglobin  : BIGINT
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: hemoglobin_name  : STRING
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: leukocyte  : BIGINT
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: leukocyte_name  : STRING
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: platelet  : BIGINT
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: platelet_name  : STRING
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: bun  : BIGINT
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: bun_name  : STRING
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: afp  : BIGINT
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: afp_name  : STRING
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: cea  : BIGINT
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: cea_name  : STRING
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: urine_glu  : BIGINT
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: hba1c  : BIGINT
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] doris schema: hba1c_name  : STRING
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] convertToStruct: ---------------------------^M

2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: ---------------------------^M

2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: arch_id  : string
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: personname  : string
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: idcard  : string
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: dt  : string
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: age  : integer
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: month_floor  : integer
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: gender_name  : string
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: bmi  : long
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: bmi_name  : string
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: waistline  : long
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: waistline_name  : string
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: diastolic_pressure_left  : long
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: diastolic_pressure_left_name  : string
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: systolic_pressure_left  : long
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: systolic_pressure_left_name  : string
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: diastolic_pressure_right  : long
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: diastolic_pressure_right_name  : string
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: systolic_pressure_right  : long
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: systolic_pressure_right_name  : string
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: fbs_mmol  : long
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: fbs_mmol_name  : string
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: triglyceridevalue  : long
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: triglyceridevalue_name  : string
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: cholesterol_value  : long
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: cholesterol_value_name  : string
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: cholesterol_low  : long
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: cholesterol_low_name  : string
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: cholesterol_high  : long
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: cholesterol_high_name  : string
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: erythrocyte  : long
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: erythrocyte_name  : string
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: hemoglobin  : long
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: hemoglobin_name  : string
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: leukocyte  : long
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: leukocyte_name  : string
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: platelet  : long
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: platelet_name  : string
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: bun  : long
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: bun_name  : string
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: afp  : long
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: afp_name  : string
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: cea  : long
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: cea_name  : string
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: urine_glu  : long
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: urine_glu_name  : string
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: hba1c  : long
2026-06-12 16:05:34 INFO  DorisUtils:43 - [SelectDB] spark struct: hba1c_name  : string

2026-06-12 15:34:14,716 WARN endpoint.BackendClient: The status of open scanner result from Doris BE{host='172.31.252.36', port=9060} is 'INVALID_ARGUMENT', error message is: [(172.31.252.36)[INVALID_ARGUMENT]Unknown primitive type(25)].
2026-06-12 15:34:14,716 WARN endpoint.BackendClient: The status of open scanner result from Doris BE{host='172.31.252.35', port=9060} is 'INVALID_ARGUMENT', error message is: [(172.31.252.35)[INVALID_ARGUMENT]Unknown primitive type(25)].
2026-06-12 15:34:14,716 WARN endpoint.BackendClient: The status of open scanner result from Doris BE{host='172.31.252.35', port=9060} is 'INVALID_ARGUMENT', error message is: [(172.31.252.35)[INVALID_ARGUMENT]Unknown primitive type(25)].
2026-06-12 15:34:14,716 WARN endpoint.BackendClient: The status of open scanner result from Doris BE{host='172.31.252.36', port=9060} is 'INVALID_ARGUMENT', error message is: [(172.31.252.36)[INVALID_ARGUMENT]Unknown primitive type(25)].
2026-06-12 15:34:14,716 WARN endpoint.BackendClient: The status of open scanner result from Doris BE{host='172.31.252.36', port=9060} is 'INVALID_ARGUMENT', error message is: [(172.31.252.36)[INVALID_ARGUMENT]Unknown primitive type(25)].
2026-06-12 15:34:14,716 WARN endpoint.BackendClient: The status of open scanner result from Doris BE{host='172.31.252.35', port=9060} is 'INVALID_ARGUMENT', error message is: [(172.31.252.35)[INVALID_ARGUMENT]Unknown primitive type(25)].
2026-06-12 15:34:14,726 WARN endpoint.BackendClient: The status of open scanner result from Doris BE{host='172.31.252.35', port=9060} is 'INVALID_ARGUMENT', error message is: [(172.31.252.35)[INVALID_ARGUMENT]Unknown primitive type(25)].
2026-06-12 15:34:14,727 WARN endpoint.BackendClient: The status of open scanner result from Doris BE{host='172.31.252.35', port=9060} is 'INVALID_ARGUMENT', error message is: [(172.31.252.35)[INVALID_ARGUMENT]Unknown primitive type(25)].
2026-06-12 15:34:14,727 WARN endpoint.BackendClient: The status of open scanner result from Doris BE{host='172.31.252.35', port=9060} is 'INVALID_ARGUMENT', error message is: [(172.31.252.35)[INVALID_ARGUMENT]Unknown primitive type(25)].
2026-06-12 15:34:14,728 WARN endpoint.BackendClient: The status of open scanner result from Doris BE{host='172.31.252.36', port=9060} is 'INVALID_ARGUMENT', error message is: [(172.31.252.36)[INVALID_ARGUMENT]Unknown primitive type(25)].
2026-06-12 15:34:14,728 WARN endpoint.BackendClient: The status of open scanner result from Doris BE{host='172.31.252.36', port=9060} is 'INVALID_ARGUMENT', error message is: [(172.31.252.36)[INVALID_ARGUMENT]Unknown primitive type(25)].
2026-06-12 15:34:14,729 WARN endpoint.BackendClient: The status of open scanner result from Doris BE{host='172.31.252.36', port=9060} is 'INVALID_ARGUMENT', error message is: [(172.31.252.36)[INVALID_ARGUMENT]Unknown primitive type(25)].
2026-06-12 15:34:14,732 WARN endpoint.BackendClient: The status of open scanner result from Doris BE{host='172.31.252.35', port=9060} is 'INVALID_ARGUMENT', error message is: [(172.31.252.35)[INVALID_ARGUMENT]Unknown primitive type(25)].
2026-06-12 15:34:14,732 ERROR endpoint.BackendClient: Connect Doris BE{host='172.31.252.35', port=9060} failed.
2026-06-12 15:34:14,733 ERROR reader.DorisRowValueReader: Failed to initialize DorisReader for partition [be:172.31.252.37:9060, tablets:[198987]]
com.selectdb.x2doris.connector.doris.exception.ConnectedFailedException: Connect to Doris BE{host='172.31.252.35', port=9060}failed.
	at com.selectdb.x2doris.connector.doris.endpoint.BackendClient.openScanner(BackendClient.java:184)
	at com.selectdb.x2doris.connector.doris.reader.DorisReader.$anonfun$init$4(DorisReader.scala:120)
	at com.selectdb.x2doris.connector.doris.reader.DorisReader.lockClient(DorisReader.scala:207)
	at com.selectdb.x2doris.connector.doris.reader.DorisReader.init(DorisReader.scala:120)
	at com.selectdb.x2doris.connector.doris.reader.DorisReader.<init>(DorisReader.scala:66)
	at com.selectdb.x2doris.connector.doris.reader.DorisRowValueReader.<init>(DorisRowValueReader.scala:29)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at com.selectdb.x2doris.connector.doris.reader.AbstractRDDIterator.reader$lzycompute(AbstractRDDIterator.scala:43)
	at com.selectdb.x2doris.connector.doris.reader.AbstractRDDIterator.reader(AbstractRDDIterator.scala:34)
	at com.selectdb.x2doris.connector.doris.reader.AbstractRDDIterator.hasNext(AbstractRDDIterator.scala:56)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:755)
	at com.selectdb.x2doris.connector.doris.writer.PartitionHandler$.tabletControlWrite(DorisWriter.scala:131)
	at com.selectdb.x2doris.connector.doris.writer.DorisWriter.$anonfun$write$2(DorisWriter.scala:59)
	at com.selectdb.x2doris.connector.doris.writer.DorisWriter.$anonfun$write$2$adapted(DorisWriter.scala:59)
	at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2(RDD.scala:1020)
	at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2$adapted(RDD.scala:1020)
	at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2236)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:131)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:498)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:501)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
0 Answers