doris 读取iceberg 文件报错

Viewed 16

doris 版本4.0.5,
创建catalog 语句:
CREATE CATALOG iceberg PROPERTIES (
"warehouse" = "hdfs://127.0.0.1:8020/hadoop_catalog",
"type" = "iceberg",
"iceberg.catalog.type" = "hadoop",
'hadoop.username' = 'hdfs',
"fs.defaultFS" = "hdfs://127.0.0.1:8020"
);
doris 通过catalog 读取iceberg 数据,如果iceberg的数据存放路径为类似:hdfs:/hadoop_catalog/fdm/f_csm_t_consume_info/data/data_dt=20220406/00000-0-6024a354-afab-4253-baea-338a777b7432-00001.parquet,路径前缀为hdfs:/
那么在解析表的第二个分区的文件时(hdfs:/hadoop_catalog/fdm/f_csm_t_consume_info/data/data_dt=20220407/00000-0-2f3724c9-86b0-40e4-8517-1319c9002bd6-00001.parquet)
将报错,报错内容如下:
java.lang.RuntimeException: StoragePropertiesException: Invalid location, missing authority: /hadoop_catalog/fdm/f_csm_t_consume_info/data/data_dt=20220407/00000-0-2f3724c9-86b0-40e4-8517-1319c9002bd6-00001.parquet
at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.translatePlan(PhysicalPlanTranslator.java:318)
at org.apache.doris.nereids.NereidsPlanner.splitFragments(NereidsPlanner.java:592)
at org.apache.doris.nereids.NereidsPlanner.distribute(NereidsPlanner.java:686)
at org.apache.doris.nereids.NereidsPlanner.lambda$plan$0(NereidsPlanner.java:166)
at org.apache.doris.nereids.NereidsPlanner.planWithLock(NereidsPlanner.java:264)
at org.apache.doris.nereids.NereidsPlanner.plan(NereidsPlanner.java:162)
at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:749)
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:542)
at org.apache.doris.qe.StmtExecutor.queryRetry(StmtExecutor.java:501)
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:486)
at org.apache.doris.qe.ConnectProcessor.executeQuery(ConnectProcessor.java:312)
at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:199)
at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:231)
at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:259)
at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:403)
at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.RuntimeException: StoragePropertiesException: Invalid location, missing authority: /hadoop_catalog/fdm/f_csm_t_consume_info/data/data_dt=20220407/00000-0-2f3724c9-86b0-40e4-8517-1319c9002bd6-00001.parquet
at org.apache.doris.datasource.iceberg.source.IcebergScanNode.getSplits(IcebergScanNode.java:388)
at org.apache.doris.datasource.FileQueryScanNode.createScanRangeLocations(FileQueryScanNode.java:366)
at org.apache.doris.datasource.iceberg.source.IcebergScanNode.createScanRangeLocations(IcebergScanNode.java:358)
at org.apache.doris.datasource.FileQueryScanNode.doFinalize(FileQueryScanNode.java:204)
at org.apache.doris.datasource.FileQueryScanNode.finalizeForNereids(FileQueryScanNode.java:195)
at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.translatePlan(PhysicalPlanTranslator.java:316)
... 18 more
Caused by: org.apache.doris.datasource.property.storage.exception.StoragePropertiesException: Invalid location, missing authority: /hadoop_catalog/fdm/f_csm_t_consume_info/data/data_dt=20220407/00000-0-2f3724c9-86b0-40e4-8517-1319c9002bd6-00001.parquet
at org.apache.doris.common.util.LocationPath.ofWithCache(LocationPath.java:255)
at org.apache.doris.datasource.iceberg.source.IcebergScanNode.createLocationPathWithCache(IcebergScanNode.java:735)
at org.apache.doris.datasource.iceberg.source.IcebergScanNode.createIcebergSplit(IcebergScanNode.java:744)
at org.apache.doris.datasource.iceberg.source.IcebergScanNode.lambda$doGetSplits$7(IcebergScanNode.java:819)
at java.base/java.lang.Iterable.forEach(Iterable.java:75)
at org.apache.doris.datasource.iceberg.source.IcebergScanNode.doGetSplits(IcebergScanNode.java:819)
at org.apache.doris.datasource.iceberg.source.IcebergScanNode.lambda$getSplits$0(IcebergScanNode.java:382)
at org.apache.doris.common.security.authentication.ExecutionAuthenticator.execute(ExecutionAuthenticator.java:44)
at org.apache.doris.datasource.iceberg.source.IcebergScanNode.getSplits(IcebergScanNode.java:382)
... 23 more

1 Answers