streamload使用jsonroot导入超大数组chunk问题

Viewed 4

我在使用2.1版本,使用s3读取minio上的json文件,内容是{
"output": [
{
"name": "26512000000066314296",
"age": "2026-01-06",
"score": null,
},{
"name": "26512000000066314296",
"age": "2026-01-06",
"score": null,
}
]
}。s3语句是select
count(*)
from
s3 (
"uri" = "s3://bucket/001.txt",
"s3.endpoint" = "http://ip:9000",
"s3.access_key" = "access",
"s3.secret_key" = "secret",
"s3.region" = "us-east-1",
"format" = "json",
"strip_outer_array" = "true",
"json_root" = "$.output" ),我测试后发现,如果这个数组有4G大小,doris好像会多节点并行chunk,导致一直提示json不合法,如果小于1G可以正常查询,不知道是不是我猜测的这种原因

0 Answers