doris与paimon湖仓一体查询，paimon表delete操作后在doris查询记录为空

Question

【环境】 doris4.0.2, paimon1.4 ,flink 1.20

【操作场景】

基于flink sql-client做建表、数据插入更新操作。

CREATE CATALOG my_catalog WITH (
'type' = 'paimon',
'warehouse' = 's3://paimon-warehouse',
's3.endpoint' = 'http://localhost:9000',
's3.access-key' = 'xxxxxxxxxxxxxx',
's3.secret-key' = 'xxxxxxxxxxxxxx',
's3.path-style-access' = 'true',
's3.connection-timeout' = '5000',
'ess' = 'true'
  );

use catalog my_catalog;
create database db_paimon;

CREATE TABLE `my_catalog`.`db_paimon`.`test_table` (
`c_custkey` INT NOT NULL,
`c_name` VARCHAR(25),
 PRIMARY KEY (`c_custkey`) NOT ENFORCED
) ;

set execution.runtime-mode =batch;

insert into `my_catalog`.`db_paimon`.`test_table` values (1,'aaa'),(2,'bbb'),(3,'ccc'),(4,'ddd'),(5,'eee'),(6,'fff'),(7,'ggg');

在doris中创建paimon-catalog

CREATE CATALOG paimon_catalog PROPERTIES (
   'type' = 'paimon',
   'warehouse' = 's3://paimon-warehouse',
   's3.endpoint' = 'http://localhost:9000',
   's3.access_key' = 'xxxxxxxxxxxxx',
   's3.secret_key' = 'xxxxxxxxxxxxx',
   's3.use_path_style_access' = 'true',
   's3.enable_ssl' = 'false',
   'paimon.s3.endpoint' = 'http://localhost:9000',
   'paimon.s3.access_key' = 'xxxxxxxxxxxxx',
   'paimon.s3.secret_key' = 'xxxxxxxxxxxxx',
   'paimon.s3.path-style-access' = 'true',
   'paimon.s3.enable_ssl' = 'false'
);

3.在doris中查询paimon表，此时可以查到前面插入的所有数据；

REFRESH CATALOG paimon_catalog;
select * from paimon_catalog.db_paimon.test_table;

继续在flink sql-client中对paimon表做删除操作(删除一条可能不会复现，需多次执行删除操作才能复现)

delete from `my_catalog`.`db_paimon`.`test_table` where id =1 ;

去doris中查询paimon表， 此时查询表记录为空

   REFRESH CATALOG paimon_catalog;
   select * from paimon_catalog.db_paimon.test_table;

再次在flink sql-client中插入新数据

insert into `my_catalog`.`db_paimon`.`test_table` values (8,'hhh')

去doris中查询该表，又可以查到数据了。

  REFRESH CATALOG paimon_catalog;
  select * from paimon_catalog.db_paimon.test_table;

【问题】
初步分析，可能是删除数据后触发了compaction操作，doris 的paimon reader不能解析compaction后的snapshot。请帮忙看下是哪个地方的配置有问题，还是当前doris版本就不支持读取compaction后的snapshot。

doris与paimon湖仓一体查询，paimon表delete操作后在doris查询记录为空

0 Answers