基本信息:
- doris版本是2.1.6
- be:3个节点,32G16C
- 每个表数据的replica=3
执行遇到多个大表join的情况,会出现集群cpu和内存的飙升,想要对其优化
select o.id orderId
from or_rescue_order o
inner join or_task_count tc on tc.order_id = o.id and tc.is_delete = 0 and tc.company_id = o.company_id
inner join or_order_task task on tc.task_id = task.id and task.is_delete = 0
LEFT JOIN fm_apply_settlement fm ON fm.order_id = o.id
inner join or_rescue_order uo
on uo.public_no = o.public_no and uo.is_delete = 0 and uo.company_id = o.source_company
left join or_task_count utc on utc.order_id = uo.id and utc.is_delete = 0 and utc.company_id = uo.company_id
left join or_task_count_provider tcp on utc.id = tcp.id
left join vr_service_type st on utc.service_type = st.type
left join or_order_other oo on uo.id = oo.id
LEFT JOIN sp_company_relation scr on scr.relation_company_id = o.company_id and scr.is_deleted = 0 and
scr.customer_id = o.customer_id and
scr.expand_company_id = o.source_company and scr.status = 0
LEFT JOIN (SELECT pr.saa_order_id, SUM(pr.pay_amount - IFNULL(pr.refund_amount_count, 0)) AS total_pay
FROM applets_pay_record pr
WHERE pr.pay_status = 2
GROUP BY pr.saa_order_id) pr_sum ON pr_sum.saa_order_id = uo.id
WHERE o.book_time >= '2022-12-01 00:00:00'
AND uo.book_time >= '2022-12-01 00:00:00'
AND o.book_time <= '2022-12-04 23:59:59'
AND uo.book_time <= '2022-12-04 23:59:59'
and o.order_number='SAAJY13163345'
and o.source_company is not null
and o.is_delete = 0
and uo.is_delete = 0
and o.order_status in (10, 13, 15)
and o.company_id in (1037)
and utc.state = 2
and (uo.order_charge_mode = 0 or oo.cash_pay_type = 0 or uo.customer_id = 64843 or IFNULL(pr_sum.total_pay, 0) > 0)
AND utc.task_status != 5
and utc.update_time < date_add(now(), interval -3 day)
and (fm.ORDER_ID is null or fm.status = 1)
limit 50 offset 0;
看了profile文件,发现条件中order_numnber对应的数据没有下推到or_task_count表,导致扫的数据太多了
尝试过加大执行的并发,并没有用
目前doris对这种优化的相关资料比较少,想请教下怎么做这种优化,或者说有什么好的方法论没
(附件不能传profile,我贴部分的profile内容)
OLAP_SCAN_OPERATOR (id=16. table name = or_task_count(or_task_count)):
- PlanInfo
- TABLE: mirror_v3.or_task_count(or_task_count), PREAGGREGATION: ON
- PREDICATES: ((((state = 2) AND (update_time < '2025-08-26 14:20:55')) AND ((task_status != 5) AND (is_delete = 0))) AND (DORIS_DELETE_SIGN = 0))
- runtime filters: RF006[in_or_bloom] -> order_id, RF007[in_or_bloom] -> company_id
- partitions=85/508 (p201809,p201810,p201811,p201812,p201901,p201902,p201903,p201904,p201905,p201906,p201907,p201908,p201909,p201910,p201911,p201912,p202001,p202002,p202003,p202004,p202005,p202006,p202007,p202008,p202009,p202010,p202011,p202012,p202101,p202102,p202103,p202104,p202105,p202106,p202107,p202108,p202109,p202110,p202111,p202112,p202201,p202202,p202203,p202204,p202205,p202206,p202207,p202208,p202209,p202210,p202211,p202212,p202301,p202302,p202303,p202304,p202305,p202306,p202307,p202308,p202309,p202310,p202311,p202312,p202401,p202402,p202403,p202404,p202405,p202406,p202407,p202408,p202409,p202410,p202411,p202412,p202501,p202502,p202503,p202504,p202505,p202506,p202507,p202508,p204001)
- tablets=255/255, tabletList=10918050,10918054,10918058 ...
- cardinality=75738134, avgRowSize=0.0, numNodes=1
- pushAggOp=NONE
- projections: create_time, id
- project output tuple id: 23
- BlocksProduced: sum 1, avg 0, max 1, min 0
- CloseTime: avg 27.898ms, max 36.265ms, min 19.530ms
- ExecTime: avg 19s365ms, max 25s343ms, min 13s388ms
- InitTime: avg 122.794us, max 138.95us, min 107.494us
- MemoryUsage: sum , avg , max , min
- PeakMemoryUsage: sum 0.00 , avg 0.00 , max 0.00 , min 0.00
- OpenTime: avg 361.438ms, max 484.398ms, min 238.479ms
- ProjectionTime: avg 0ns, max 0ns, min 0ns
- RowsProduced: sum 1, avg 0, max 1, min 0
- RuntimeFilterInfo: sum , avg , max , min
- WaitForDependency[OLAP_SCAN_OPERATOR_DEPENDENCY]Time: avg 17s960ms, max 24s53ms, min 11s866ms
VScanner:
- MemoryUsage: sum , avg , max , min
- FreeBlocks: sum 0.00 , avg 0.00 , max 0.00 , min 0.00