Iceberg Catalog 使用 MinIO 作为存储时, 读写权限报错

Viewed 76

Iceberg Catalog: LakeKeeper 0.11.2
MinIO: minio/minio:RELEASE.2024-10-29T16-01-48Z
Doris: 3.1.4/4.0.3

当使用 vended-credentials-enabled 时, 无论是读表还是写表都会报权限错误

failed to read from : AccessDenied Access Denied. code=FORBIDDEN, type=15, request_id=1895A6DC4C0D2E01failed to read. cur path: s3://iceberg/019c7515-e794-79b2-93a0-44a8d560d1a3/019c7557-e0f1-7232-b98b-9c7be71f7151/data/ba8c85b51c6a41fb-86a0962fecef90a9_a743038d-ce28-45d6-8131-0036c9af5991-0.zstd.parquet

fe log

java.lang.Exception: (be)[E-256]failed to read from : AccessDenied Access Denied. code=FORBIDDEN, type=15, request_id=1895A6DC4C0D2E01failed to read. cur path: s3://iceberg/019c7515-e794-79b2-93a0-44a8d560d1a3/019c7557-e0f1-7232-b98b-9c7be71f7151/data/ba8c85b51c6a41fb-86a0962fecef90a9_a743038d-ce28-45d6-8131-0036c9af5991-0.zstd.parquet
	at org.apache.doris.qe.Coordinator.getNext(Coordinator.java:1198)
	at org.apache.doris.qe.StmtExecutor.executeAndSendResult(StmtExecutor.java:1322)
	at org.apache.doris.qe.StmtExecutor.handleQueryStmt(StmtExecutor.java:1260)
	at org.apache.doris.qe.StmtExecutor.handleQueryWithRetry(StmtExecutor.java:917)
	at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:824)
	at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:541)
	at org.apache.doris.qe.StmtExecutor.queryRetry(StmtExecutor.java:500)
	at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:485)
	at org.apache.doris.qe.ConnectProcessor.executeQuery(ConnectProcessor.java:311)
	at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:198)
	at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:231)
	at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:259)
	at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:403)
	at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:842)

当使用 vended-credentials-enabled 时, 将 MinIO 桶权限设置为 public 时, 读写无问题.

补充: 当使用静态的 ak 和 sk 时, 读表和写表会卡住一直处在查询状态, Doris 3.0.8 上没有这个问题.

补充: 从湖里读写数据很慢.(极小数据量就可以测试, 非常慢)

2 Answers
  1. 使用 vended-credentials-enabled 是通过rest 服务生成临时凭证的方式访问 Minio ,如果 rest 服务并没有配置或支持凭证代发,那就会有权限问题。改成 public 权限肯定能访问,因为不需要凭证。
  2. 使用静态 AK SK 访问,在卡住的时候需要收集一些堆栈信息进行查看。比如FE的jstack 信息,BE的pstack信息之类,来进行排查
  3. 小数据量很慢,一般这种和带宽、网络等可能有直接关系,但是不排除其他因素,如果有监控的话可以先看看监控。或者打个profile看看
Summary:
   - Profile ID: 6a91ef2c5d044ef9-b3ae2ea508a1ad47
   - Task Type: QUERY
   - Start Time: 2026-02-24 10:55:40
   - End Time: 2026-02-24 10:55:56
   - Total: 15sec948ms
   - Task State: OK
   - User: root
   - Default Catalog: iceberg
   - Default Db: fin_eastmoney
   - Sql Statement: select * from test LIMIT 1001;
   - Distributed Plan: N/A
Execution Summary:
   - Workload Group: normal
   - Parse SQL Time: 1ms
   - Plan Time: 1sec877ms
     - Garbage Collect During Plan Time: 0ms
     - Nereids Lock Table Time: 57ms
     - Nereids Analysis Time: 211ms
     - Nereids Rewrite Time: 1ms
       - Nereids Fold Const By BE Time: 0ms
     - Nereids Collect Table Partition Time: 0ms
   - Nereids Pre Rewrite By Mv Time: N/A
     - Nereids Optimize Time: N/A
     - Nereids Translate Time: 1sec588ms
       - Init Scan Node Time: 61ms
       - Finalize Scan Node Time: 1sec476ms
         - Get Splits Time: 1sec468ms
         - Get Partitions Time: N/A
         - Get Partition Files Time: N/A
         - Sink Set Partition Values Time: N/A
       - Create Scan Range Time: 8ms
         - Iceberg Scan Metrics: N/A
         - Paimon Scan Metrics: N/A
     - Nereids Distribute Time: N/A
   - Get Meta Version Time: N/A
     - Get Partition Version Time: N/A
     - Get Partition Version Count (hasData): N/A
     - Get Partition Version Count: N/A
     - Get Table Version Time: N/A
     - Get Table Version Count: N/A
   - Schedule Time: 25ms
     - Fragment Assign Time: 10ms
     - Fragment Serialize Time: 11ms
     - Fragment RPC Phase1 Time: 3ms
     - Fragment RPC Phase2 Time: 1ms
     - Fragment Compressed Size: 2.69 KB
     - Fragment RPC Count: 2
   - Schedule Time Of BE: {"phase1":{"be: 8060":{"RPC Work Time":"2ms","RPC Latency From FE To BE":"1ms","RPC Work Queue Time":"0ms","RPC Latency From BE To FE":"0ms"}},"phase2":{"be: 8060":{"RPC Work Time":"0ms","RPC Latency From FE To BE":"1ms","RPC Work Queue Time":"0ms","RPC Latency From BE To FE":"0ms"}}}
   - Wait and Fetch Result Time: 14sec45ms
   - Fetch Result Time: 14sec44ms
   - Write Result Time: 0ms
   - Doris Version: doris-4.0.3-rc03-e9096296b8
   - Is Nereids: Yes
   - Is Cached: No
   - Total Instances Num: 2
   - Instances Num Per BE: be:8060:2
   - Parallel Fragment Exec Instance Num: 24
   - Trace ID: 2f3ae78a-bd26-455b-ae0e-c26e5c0ccc7f
   - Transaction Commit Time: N/A
   - System Message: N/A
   - Executed By Frontend: N/A
   - Splits Assignment Weight: N/A
  Iceberg Scan Metrics:
    Table Scan (iceberg.fin_eastmoney.test):
       - table: iceberg.fin_eastmoney.test
       - snapshot: 4177479596196165619
       - filter: true
       - columns: id
       - planning: 96ms (1 ops)
       - data_files: 3
       - delete_files: 0
       - skipped_data_files: 0
       - skipped_delete_files: 0
       - total_size: 2.874 KB
       - total_delete_size: 0.000 
       - scanned_manifests: 3
       - skipped_manifests: 0
       - scanned_delete_manifests: 0
       - skipped_delete_manifests: 0
       - indexed_delete_files: 0
       - equality_delete_files: 0
       - positional_delete_files: 0

ChangedSessionVariables:
[
  {
    "VarName": "enable_new_type_coercion_behavior",
    "CurrentValue": "false",
    "DefaultValue": "true"
  },
  {
    "VarName": "character_set_results",
    "CurrentValue": "NULL",
    "DefaultValue": "utf8mb4"
  },
  {
    "VarName": "enable_profile",
    "CurrentValue": "true",
    "DefaultValue": "false"
  },
  {
    "VarName": "enable_audit_plugin",
    "CurrentValue": "false",
    "DefaultValue": "true"
  },
  {
    "VarName": "insert_visible_timeout_ms",
    "CurrentValue": "10000",
    "DefaultValue": "60000"
  },
  {
    "VarName": "max_scan_key_num",
    "CurrentValue": "-1",
    "DefaultValue": "48"
  },
  {
    "VarName": "enable_unicode_name_support",
    "CurrentValue": "false",
    "DefaultValue": "true"
  },
  {
    "VarName": "fetch_splits_max_wait_time_ms",
    "CurrentValue": "4000",
    "DefaultValue": "1000"
  },
  {
    "VarName": "runtime_filter_max_in_num",
    "CurrentValue": "1024",
    "DefaultValue": "40960"
  },
  {
    "VarName": "sql_mode",
    "CurrentValue": "ONLY_FULL_GROUP_BY,PIPES_AS_CONCAT,STRICT_TRANS_TABLES",
    "DefaultValue": "ONLY_FULL_GROUP_BY"
  },
  {
    "VarName": "session_context",
    "CurrentValue": "trace_id:2f3ae78a-bd26-455b-ae0e-c26e5c0ccc7f",
    "DefaultValue": ""
  },
  {
    "VarName": "runtime_bloom_filter_min_size",
    "CurrentValue": "2048",
    "DefaultValue": "1048576"
  },
  {
    "VarName": "experimental_parallel_scan_min_rows_per_scanner",
    "CurrentValue": "16384",
    "DefaultValue": "2097152"
  },
  {
    "VarName": "parallel_exchange_instance_num",
    "CurrentValue": "-1",
    "DefaultValue": "100"
  },
  {
    "VarName": "enable_parallel_result_sink",
    "CurrentValue": "false",
    "DefaultValue": "true"
  },
  {
    "VarName": "enable_ansi_query_organization_behavior",
    "CurrentValue": "false",
    "DefaultValue": "true"
  },
  {
    "VarName": "experimental_parallel_scan_max_scanners_count",
    "CurrentValue": "48",
    "DefaultValue": "0"
  },
  {
    "VarName": "audit_plugin_max_sql_length",
    "CurrentValue": "4096",
    "DefaultValue": "2097152"
  },
  {
    "VarName": "exec_mem_limit",
    "CurrentValue": "2147483648",
    "DefaultValue": "100147483648"
  },
  {
    "VarName": "experimental_enable_nereids_distribute_planner",
    "CurrentValue": "false",
    "DefaultValue": "true"
  },
  {
    "VarName": "max_pushdown_conditions_per_column",
    "CurrentValue": "-1",
    "DefaultValue": "1024"
  },
  {
    "VarName": "runtime_bloom_filter_max_size",
    "CurrentValue": "16777216",
    "DefaultValue": "67108864"
  },
  {
    "VarName": "max_allowed_packet",
    "CurrentValue": "1048576",
    "DefaultValue": "16777216"
  }
]
MergedProfile:
     Fragments:
       Fragment 0:
         Pipeline 0(instance_num=1):
            - WaitWorkerTime: avg 388.589us, max 388.589us, min 388.589us
           RESULT_SINK_OPERATOR(id=2147483647):
             CommonCounters:
                - ExecTime: avg 721.516us, max 721.516us, min 721.516us
                - InputRows: sum 3, avg 3, max 3, min 3
                - MemoryUsage: sum 0.00 , avg 0.00 , max 0.00 , min 0.00 
                - MemoryUsagePeak: sum 0.00 , avg 0.00 , max 0.00 , min 0.00 
                - WaitForDependency[RESULT_SINK_OPERATOR_DEPENDENCY]Time: avg 0ns, max 0ns, min 0ns
             CustomCounters:
                - WaitForDependencyTime: avg 0ns, max 0ns, min 0ns
           EXCHANGE_OPERATOR(id=1):
              - PlanInfo
                 - offset: 0
                 - limit: 1001
             CommonCounters:
                - BlocksProduced: sum 3, avg 3, max 3, min 3
                - ExecTime: avg 50.526us, max 50.526us, min 50.526us
                - MemoryUsage: sum 0.00 , avg 0.00 , max 0.00 , min 0.00 
                - MemoryUsagePeak: sum 128.00 B, avg 128.00 B, max 128.00 B, min 128.00 B
                - RowsProduced: sum 3, avg 3, max 3, min 3
             CustomCounters:
                - WaitForDependencyTime: avg 0ns, max 0ns, min 0ns
                  - WaitForData0: avg 14sec42ms, max 14sec42ms, min 14sec42ms
       Fragment 1:
         Pipeline 0(instance_num=1):
            - WaitWorkerTime: avg 388.299us, max 388.299us, min 388.299us
           DATA_STREAM_SINK_OPERATOR(dest_id=1):
             CommonCounters:
                - ExecTime: avg 85.260us, max 85.260us, min 85.260us
                - InputRows: sum 3, avg 3, max 3, min 3
                - MemoryUsage: sum 0.00 , avg 0.00 , max 0.00 , min 0.00 
                - MemoryUsagePeak: sum 0.00 , avg 0.00 , max 0.00 , min 0.00 
                - WaitForDependencyTime: avg 0ns, max 0ns, min 0ns
                  - WaitForRpcBufferQueue: avg 0ns, max 0ns, min 0ns
             CustomCounters:
                - BlocksProduced: sum 4, avg 4, max 4, min 4
                - OverallThroughput: sum 0.0 /sec, avg 0.0 /sec, max 0.0 /sec, min 0.0 /sec
           FILE_SCAN_OPERATOR(nereids_id=73. table_name=test)(id=0):
              - PlanInfo
                 - table: iceberg.fin_eastmoney.test
                 - inputSplitNum=3, totalFileSize=2943, scanRanges=3
                 - partition=0/0
                 - cardinality=3, numNodes=1
                 - pushdown agg=NONE
                 - limit: 1001
             CommonCounters:
                - BlocksProduced: sum 3, avg 3, max 3, min 3
                - ExecTime: avg 14sec42ms, max 14sec42ms, min 14sec42ms
                - MemoryUsage: sum 0.00 , avg 0.00 , max 0.00 , min 0.00 
                - MemoryUsagePeak: sum 0.00 , avg 0.00 , max 0.00 , min 0.00 
                - RowsProduced: sum 3, avg 3, max 3, min 3
                - WaitForDependency[FILE_SCAN_OPERATOR_DEPENDENCY]Time: avg 14sec41ms, max 14sec41ms, min 14sec41ms
             CustomCounters:
                - RuntimeFilterInfo: sum , avg , max , min 
                - ScanBytes: sum 3.14 KB, avg 3.14 KB, max 3.14 KB, min 3.14 KB
                - ScanRows: sum 3, avg 3, max 3, min 3

DetailProfile(6a91ef2c5d044ef9-b3ae2ea508a1ad47):
  Fragments:
    Fragment 0:
      FragmentLevelProfile:(host=TNetworkAddress(hostname:be, port:9050)):
      Pipeline 0(host=TNetworkAddress(hostname:be, port:9050)):
        PipelineTask(index=0):
           - TaskState: FINALIZED
           - BlockedByDependency: 
           - WakeUpEarly: 0
           - WaitWorkerTime: 388.589us
          RESULT_SINK_OPERATOR(id=2147483647):
            CommonCounters:
               - ExecTime: 721.516us
               - InputRows: 3
               - MemoryUsage: 0.00 
               - MemoryUsagePeak: 0.00 
               - WaitForDependency[RESULT_SINK_OPERATOR_DEPENDENCY]Time: 0ns
            CustomCounters:
               - WaitForDependencyTime: 0ns
          EXCHANGE_OPERATOR(id=1):
            CommonCounters:
               - BlocksProduced: 3
               - ExecTime: 50.526us
               - MemoryUsage: 0.00 
               - MemoryUsagePeak: 128.00 B
               - RowsProduced: 3
            CustomCounters:
               - InstanceID: 6a91ef2c5d044ef9-b3ae2ea508a1ad49
               - WaitForDependencyTime: 0ns
                 - WaitForData0: 14sec42ms
    Fragment 1:
      FragmentLevelProfile:(host=TNetworkAddress(hostname:be, port:9050)):
      Pipeline 0(host=TNetworkAddress(hostname:be, port:9050)):
        PipelineTask(index=0):
           - TaskState: FINALIZED
           - BlockedByDependency: 
           - WakeUpEarly: 0
           - WaitWorkerTime: 388.299us
          DATA_STREAM_SINK_OPERATOR(dest_id=1):
            CommonCounters:
               - ExecTime: 85.260us
               - InputRows: 3
               - MemoryUsage: 0.00 
               - MemoryUsagePeak: 0.00 
               - WaitForDependencyTime: 0ns
                 - WaitForRpcBufferQueue: 0ns
            CustomCounters:
               - BlocksProduced: 4
               - OverallThroughput: 0.0 /sec
          FILE_SCAN_OPERATOR(nereids_id=73. table_name=test)(id=0):
            CommonCounters:
               - BlocksProduced: 3
               - ExecTime: 14sec42ms
               - MemoryUsage: 0.00 
               - MemoryUsagePeak: 0.00 
               - RowsProduced: 3
               - WaitForDependency[FILE_SCAN_OPERATOR_DEPENDENCY]Time: 14sec41ms
            CustomCounters:
               - TopNFilterSourceNodeIds: 
               - PushDownPredicates: 
               - RemainedDownPredicates: 
               - RuntimeFilterInfo: 
               - ScanBytes: 3.14 KB
               - ScanRows: 3
              Scanner:
                 - PerScannerRunningTime: [14s041ms, ]
                 - PerScannerRowsRead: [3, ]
                 - PerScannerWaitTime: [262.553us, ]
                 - PerScannerProjectionTime: [0.000ns, ]
                 - EOS: True
                 - EmptyFileNum: 0
                 - FileFooterHitCache: 0
                 - FileFooterReadCalls: 3
                 - FileNumber: 3
                 - FileReadBytes: 3.14 KB
                 - FileReadCalls: 6
                 - FileReadTime: 6sec19ms
                 - FileScannerCastInputBlockTime: 0ns
                 - FileScannerConvertOuputBlockTime: 0ns
                 - FileScannerFillMissingColumnTime: 0ns
                 - FileScannerGetBlockTime: 1.693ms
                 - FileScannerPreFilterTimer: 0ns
                 - FileScannerRuntimeFilterPartitionPruningTime: 0ns
                 - FullySkippedFileNum: 0
                 - NotFoundFileNum: 0
                 - PageIndexReadCalls: 0
                 - ParquetReader: 0ns
                   - BloomFilterReadTime: 0ns
                   - ColumnReadTime: 402.576us
                   - DecodeDictTime: 1.204ms
                   - DecodeLevelTime: 1.42us
                   - DecodeNullMapTime: 13.386us
                   - DecodeValueTime: 13.475us
                   - DecompressCount: 3
                   - DecompressTime: 2.805us
                   - DictFilterRewriteTime: 0ns
                   - FileNum: 3
                   - FileReaderCreateTime: 8sec19ms
                   - FilteredBytes: 0.00 
                   - FilteredRowsByGroup: 0
                   - FilteredRowsByLazyRead: 0
                   - FilteredRowsByPage: 0
                   - PageHeaderDecodeTime: 19.355us
                   - PageHeaderReadTime: 16.862us
                   - PageIndexFilterTime: 0ns
                   - PageIndexParseTime: 0ns
                   - PageIndexReadTime: 0ns
                   - ParseFooterTime: 6sec19ms
                   - ParseMetaTime: 16.671us
                   - ParsePageHeaderNum: 6
                   - PredicateFilterTime: 3.786us
                   - RawRowsRead: 3
                   - RowGroupFilterTime: 5.940us
                   - RowGroupsFiltered: 0
                   - RowGroupsFilteredByBloomFilter: 0
                   - RowGroupsFilteredByMinMax: 0
                   - RowGroupsReadNum: 3
                   - RowGroupsTotalNum: 3
                   - SkipPageHeaderNum: 0
                 - RuntimeFilterPartitionPrunedRangeNum: 0
  LoadChannels:

Appendix:

PhysicalPlan:
PhysicalResultSink[89] ( outputExprs=[id#0] )
     +--PhysicalLimit[85]@2 ( limit=1001, offset=0, phase=GLOBAL, stats=3 )
        +--PhysicalDistribute[81]@1 ( stats=3, distributionSpec=DistributionSpecGather )
           +--PhysicalLimit[77]@1 ( limit=1001, offset=0, phase=LOCAL, stats=3 )
              +--PhysicalFileScan[test]@0 ( stats=3 actualRows=3, qualified=iceberg.fin_eastmoney.test, selected partitions num=unknown, operativeCols=[] )
     PhysicalFileScan[test]@0 ( stats=3 actualRows=3, qualified=iceberg.fin_eastmoney.test, selected partitions num=unknown, operativeCols=[] )
       id#0 -> unknown(3.0)