2.1.2版本历史数据迁移问题(旧表为JSON字段类型,新表为Variant字段类型)

Viewed 47

旧表(28GB,1000万行左右的数据)为一个Duplicate模型,某个字段是JSON类型,现在新建了一张表为Unique模型,某个字段是Variant类型,在尝试旧表数据迁移到新表,并且Variant字段类型去替代JSON字段类型的insert into select过程中,每次执行这个语句,be节点都会挂掉或者这个sql执行的时候cancelled。

3 Answers

方便提供一下复现的sql、相应的表结构和部分数据吗

create table t1(
target_index varchar(1024),
primarykey varchar(1024),
content_version varchar(1024),
content JSON,
time datetime default current_timestamp
)DUPLICATE KEY(target_index,primarykey,content_version)
DISTRIBUTED by hash(target_index) buckets 10
properties(
"replication_allocation" = "tag.location.default:1",
"min_load_replica_num" = "-1",
"is_being_synced" = "false",
"storage_medium" = "hdd",
"storage_format" = "V2",
"inverted_index_storage_format" = "V1",
"light_scheme_change" = "true",
"storage_row_column" = "true",
"disable_auto_compaction" = "false",
"enable_single_replica_compaction" = "false",
"group_commit_interval_ms" = "1000",
"group_commit_data_bytes" = "134217728"
);

create table t2(
target_index varchar(1024),
primarykey varchar(1024),
content_version varchar(1024),
content Variant,
time datetime default current_timestamp
)UNQIUE KEY(target_index,primarykey,content_version)
DISTRIBUTED by hash(target_index) buckets 10
properties(
"replication_allocation" = "tag.location.default:1",
"min_load_replica_num" = "-1",
"is_being_synced" = "false",
"storage_medium" = "hdd",
"storage_format" = "V2",
"inverted_index_storage_format" = "V1",
"enable_unique_key_merge_on_write" = "true",
"light_scheme_change" = "true",
"storage_row_column" = "true",
"disable_auto_compaction" = "false",
"enable_single_replica_compaction" = "false",
"group_commit_interval_ms" = "1000",
"group_commit_data_bytes" = "134217728"
)

insert into t2
select
target_index,
primarykey,
content_version,
content
from t1;

content数据类似如下:
{
"name": "test",
"age": "18",
"picture": [
{
"bz": "test",
"id": "1",
"url": "http://asdasdas/asda/asd.jpg"
},
{
"bz": "test2",
"id": "2",
"url": "http://asdasdas/asda/asd1.jpg"
}
]
}
里面除了picture是确定的,其他字段都是不确定的。

尝试升级到2.1.5,如果还有问题,可以贴一下be.out的栈