doris 版本 4.1.3,单be+单fe,存算分离模式
开启算子落盘之后,
如果读取的数据远超be的内存时,数据可以落盘成功,但是scan不能继续读取数据,导致任务挂起,以下两次执行命令SHOW PROC "/current_queries",间隔时间为一个70分钟但是读取的行数基本没涨。
select count(0) from iceberg.sdm.s_tft_tsm_t_trade where data_dt < '20220731';
+------------+
| count(0) |
+------------+
| 1966688311 |
+------------+
1 row in set (31.72 sec)
SHOW PROC "/current_queries";
+-----------------------------------+--------------+----------+----------+-----------+----------+----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+-----------+----------------+--------+--------------------+------------------------+-----------------+------------------+-----------------+---------------------------+----------------------------+-------------------------------+--------------------------------+---------------------+------------+---------------+----------+
| QueryId | ConnectionId | Catalog | Database | User | ExecTime | SqlHash | Statement | ScanRows | ScanBytes | ProcessRows | CpuMs | MaxPeakMemoryBytes | CurrentUsedMemoryBytes | WorkloadGroupId | ShuffleSendBytes | ShuffleSendRows | ScanBytesFromLocalStorage | ScanBytesFromRemoteStorage | SpillWriteBytesToLocalStorage | SpillReadBytesFromLocalStorage | BytesWriteIntoCache | TotalTasks | FinishedTasks | Progress |
+-----------------------------------+--------------+----------+----------+-----------+----------+----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+-----------+----------------+--------+--------------------+------------------------+-----------------+------------------+-----------------+---------------------------+----------------------------+-------------------------------+--------------------------------+---------------------+------------+---------------+----------+
| 9dbb75129f7b4738-8d638400f968f610 | 4 | internal | fdm | compute_a | 229226 | 0792ebfc30c9c63c412776f5170c036f | insert into fdm.f_csm_t_trade
select
t3.order_date,
t3.trade_id,
t3.trade_tn,
t3.userid,
t3.msisdn,
t3.order_type,
t3.order_state,
t3.order_info,
t3.update_time,
t3.check_msg,
t3.tftpk_tn,
t3.isnot_check,
t3.query_id,
t3.iccid,
t3.trade_channel,
t3.trade_amt,
t3.other_amt,
t3.appid,
t3.callback_url,
t3.business_state,
t3.qr_userid,
t3.acctno,
t3.buss_type,
t3.default_trade_channel,
t3.update_time table_update
from (select *,row_number() over(partition by trade_id order by data_dt) rk from iceberg.sdm.s_tft_tsm_t_trade where data_dt < '20220731') t3
where rk =1 | 131298401 Rows | 5.94 GB | 145717780 Rows | 538729 | 37.66 GB | 21.13 GB | 1785309225922 | 0.00 | 0 Rows | 0.00 | 5.94 GB | 3.22 GB | 0.00 | 0.00 | 121 | 0 | 0.0% |
+-----------------------------------+--------------+----------+----------+-----------+----------+----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+-----------+----------------+--------+--------------------+------------------------+-----------------+------------------+-----------------+---------------------------+----------------------------+-------------------------------+--------------------------------+---------------------+------------+---------------+----------+
SHOW PROC "/current_queries";
+-----------------------------------+--------------+----------+----------+-----------+----------+----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+-----------+----------------+--------+--------------------+------------------------+-----------------+------------------+-----------------+---------------------------+----------------------------+-------------------------------+--------------------------------+---------------------+------------+---------------+----------+
| QueryId | ConnectionId | Catalog | Database | User | ExecTime | SqlHash | Statement | ScanRows | ScanBytes | ProcessRows | CpuMs | MaxPeakMemoryBytes | CurrentUsedMemoryBytes | WorkloadGroupId | ShuffleSendBytes | ShuffleSendRows | ScanBytesFromLocalStorage | ScanBytesFromRemoteStorage | SpillWriteBytesToLocalStorage | SpillReadBytesFromLocalStorage | BytesWriteIntoCache | TotalTasks | FinishedTasks | Progress |
+-----------------------------------+--------------+----------+----------+-----------+----------+----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+-----------+----------------+--------+--------------------+------------------------+-----------------+------------------+-----------------+---------------------------+----------------------------+-------------------------------+--------------------------------+---------------------+------------+---------------+----------+
| 9dbb75129f7b4738-8d638400f968f610 | 4 | internal | fdm | compute_a | 4262622 | 0792ebfc30c9c63c412776f5170c036f | insert into fdm.f_csm_t_trade
select
t3.order_date,
t3.trade_id,
t3.trade_tn,
t3.userid,
t3.msisdn,
t3.order_type,
t3.order_state,
t3.order_info,
t3.update_time,
t3.check_msg,
t3.tftpk_tn,
t3.isnot_check,
t3.query_id,
t3.iccid,
t3.trade_channel,
t3.trade_amt,
t3.other_amt,
t3.appid,
t3.callback_url,
t3.business_state,
t3.qr_userid,
t3.acctno,
t3.buss_type,
t3.default_trade_channel,
t3.update_time table_update
from (select *,row_number() over(partition by trade_id order by data_dt) rk from iceberg.sdm.s_tft_tsm_t_trade where data_dt < '20220731') t3
where rk =1 | 135332549 Rows | 6.09 GB | 145783060 Rows | 539415 | 37.66 GB | 21.35 GB | 1785309225922 | 0.00 | 0 Rows | 0.00 | 6.09 GB | 3.22 GB | 0.00 | 0.00 | 121 | 0 | 0.0% |
+-----------------------------------+--------------+----------+----------+-----------+----------+----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+-----------+----------------+--------+--------------------+------------------------+-----------------+------------------+-----------------+---------------------------+----------------------------+-------------------------------+--------------------------------+---------------------+------------+---------------+----------+
如果查询任务读取的数据能全部装入be的内存,则数据落盘之后,后续任务将正常执行,整个任务成功执行
select count(0) from iceberg.sdm.s_tft_tsm_t_trade where data_dt between '2017' and '201911';
+-----------+
| count(0) |
+-----------+
| 202580679 |
+-----------+
SHOW PROC "/current_queries";
+-----------------------------------+--------------+----------+----------+------+----------+----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+-----------+----------------+---------+--------------------+------------------------+-----------------+------------------+-----------------+---------------------------+----------------------------+-------------------------------+--------------------------------+---------------------+------------+---------------+----------+
| QueryId | ConnectionId | Catalog | Database | User | ExecTime | SqlHash | Statement | ScanRows | ScanBytes | ProcessRows | CpuMs | MaxPeakMemoryBytes | CurrentUsedMemoryBytes | WorkloadGroupId | ShuffleSendBytes | ShuffleSendRows | ScanBytesFromLocalStorage | ScanBytesFromRemoteStorage | SpillWriteBytesToLocalStorage | SpillReadBytesFromLocalStorage | BytesWriteIntoCache | TotalTasks | FinishedTasks | Progress |
+-----------------------------------+--------------+----------+----------+------+----------+----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+-----------+----------------+---------+--------------------+------------------------+-----------------+------------------+-----------------+---------------------------+----------------------------+-------------------------------+--------------------------------+---------------------+------------+---------------+----------+
| f08a4234e5cd459b-a33dd930147635ed | 16 | internal | | root | 184869 | aa424c2bf548249bbdb8d08761b07b69 | insert into fdm.f_csm_t_trade
select
t3.order_date,
t3.trade_id,
t3.trade_tn,
t3.userid,
t3.msisdn,
t3.order_type,
t3.order_state,
t3.order_info,
t3.update_time,
t3.check_msg,
t3.tftpk_tn,
t3.isnot_check,
t3.query_id,
t3.iccid,
t3.trade_channel,
t3.trade_amt,
t3.other_amt,
t3.appid,
t3.callback_url,
t3.business_state,
t3.qr_userid,
t3.acctno,
t3.buss_type,
t3.default_trade_channel,
t3.update_time table_update
from (select *,row_number() over(partition by trade_id order by data_dt desc) rk from iceberg.sdm.s_tft_tsm_t_trade where data_dt between '2017' and '201911') t3
where rk =1 | 202580679 Rows | 12.27 GB | 784725785 Rows | 2611942 | 93.27 GB | 28.88 GB | 1785309225922 | 0.00 | 0 Rows | 0.00 | 12.27 GB | 6.54 GB | 4.29 GB | 0.00 | 121 | 81 | 66.9% |
+-----------------------------------+--------------+----------+----------+------+----------+----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+-----------+----------------+---------+--------------------+------------------------+-----------------+------------------+-----------------+---------------------------+----------------------------+-------------------------------+--------------------------------+---------------------+------------+---------------+----------+