复杂SQL优化,请求协助

Viewed 13

基本信息:

  • doris版本是2.1.6
  • be:3个节点,32G16C
  • 每个表数据的replica=3

执行遇到多个大表join的情况,会出现集群cpu和内存的飙升,想要对其优化

select o.id                                                     orderId
from or_rescue_order o
         inner join or_task_count tc on tc.order_id = o.id and tc.is_delete = 0 and tc.company_id = o.company_id
         inner join or_order_task task on tc.task_id = task.id and task.is_delete = 0
         LEFT JOIN fm_apply_settlement fm ON fm.order_id = o.id
         inner join or_rescue_order uo
                    on uo.public_no = o.public_no and uo.is_delete = 0 and uo.company_id = o.source_company
         left join or_task_count utc on utc.order_id = uo.id and utc.is_delete = 0 and utc.company_id = uo.company_id
         left join or_task_count_provider tcp on utc.id = tcp.id
         left join vr_service_type st on utc.service_type = st.type
         left join or_order_other oo on uo.id = oo.id
         LEFT JOIN sp_company_relation scr on scr.relation_company_id = o.company_id and scr.is_deleted = 0 and
                                              scr.customer_id = o.customer_id and
                                              scr.expand_company_id = o.source_company and scr.status = 0
         LEFT JOIN (SELECT pr.saa_order_id, SUM(pr.pay_amount - IFNULL(pr.refund_amount_count, 0)) AS total_pay
                    FROM applets_pay_record pr
                    WHERE pr.pay_status = 2
                    GROUP BY pr.saa_order_id) pr_sum ON pr_sum.saa_order_id = uo.id
WHERE o.book_time >= '2022-12-01 00:00:00'
  AND uo.book_time >= '2022-12-01 00:00:00'
  AND o.book_time <= '2022-12-04 23:59:59'
  AND uo.book_time <= '2022-12-04 23:59:59'
   and o.order_number='SAAJY13163345'
  and o.source_company is not null
  and o.is_delete = 0
  and uo.is_delete = 0
  and o.order_status in (10, 13, 15)
  and o.company_id in (1037)
  and utc.state = 2
  and (uo.order_charge_mode = 0 or oo.cash_pay_type = 0 or uo.customer_id = 64843 or IFNULL(pr_sum.total_pay, 0) > 0)
  AND utc.task_status != 5
  and utc.update_time < date_add(now(), interval -3 day)
  and (fm.ORDER_ID is null or fm.status = 1)
limit 50 offset 0;

看了profile文件,发现条件中order_numnber对应的数据没有下推到or_task_count表,导致扫的数据太多了
尝试过加大执行的并发,并没有用
目前doris对这种优化的相关资料比较少,想请教下怎么做这种优化,或者说有什么好的方法论没
(附件不能传profile,我贴部分的profile内容)
OLAP_SCAN_OPERATOR (id=16. table name = or_task_count(or_task_count)):
- PlanInfo
- TABLE: mirror_v3.or_task_count(or_task_count), PREAGGREGATION: ON
- PREDICATES: ((((state = 2) AND (update_time < '2025-08-26 14:20:55')) AND ((task_status != 5) AND (is_delete = 0))) AND (DORIS_DELETE_SIGN = 0))
- runtime filters: RF006[in_or_bloom] -> order_id, RF007[in_or_bloom] -> company_id
- partitions=85/508 (p201809,p201810,p201811,p201812,p201901,p201902,p201903,p201904,p201905,p201906,p201907,p201908,p201909,p201910,p201911,p201912,p202001,p202002,p202003,p202004,p202005,p202006,p202007,p202008,p202009,p202010,p202011,p202012,p202101,p202102,p202103,p202104,p202105,p202106,p202107,p202108,p202109,p202110,p202111,p202112,p202201,p202202,p202203,p202204,p202205,p202206,p202207,p202208,p202209,p202210,p202211,p202212,p202301,p202302,p202303,p202304,p202305,p202306,p202307,p202308,p202309,p202310,p202311,p202312,p202401,p202402,p202403,p202404,p202405,p202406,p202407,p202408,p202409,p202410,p202411,p202412,p202501,p202502,p202503,p202504,p202505,p202506,p202507,p202508,p204001)
- tablets=255/255, tabletList=10918050,10918054,10918058 ...
- cardinality=75738134, avgRowSize=0.0, numNodes=1
- pushAggOp=NONE
- projections: create_time, id
- project output tuple id: 23
- BlocksProduced: sum 1, avg 0, max 1, min 0
- CloseTime: avg 27.898ms, max 36.265ms, min 19.530ms
- ExecTime: avg 19s365ms, max 25s343ms, min 13s388ms
- InitTime: avg 122.794us, max 138.95us, min 107.494us
- MemoryUsage: sum , avg , max , min
- PeakMemoryUsage: sum 0.00 , avg 0.00 , max 0.00 , min 0.00
- OpenTime: avg 361.438ms, max 484.398ms, min 238.479ms
- ProjectionTime: avg 0ns, max 0ns, min 0ns
- RowsProduced: sum 1, avg 0, max 1, min 0
- RuntimeFilterInfo: sum , avg , max , min
- WaitForDependency[OLAP_SCAN_OPERATOR_DEPENDENCY]Time: avg 17s960ms, max 24s53ms, min 11s866ms
VScanner:
- MemoryUsage: sum , avg , max , min
- FreeBlocks: sum 0.00 , avg 0.00 , max 0.00 , min 0.00

1 Answers

OLAP_SCAN_OPERATOR (id=16. table name = or_task_count(or_task_count)):
- RuntimeFilters: : RuntimeFilter: (id = 6, type = in_or_bloomfilter, need_local_merge: false, is_broadcast: false, build_bf_cardinality: false, RuntimeFilter: (id = 7, type = in_or_bloomfilter, need_local_merge: false, is_broadcast: false, build_bf_cardinality: false,
- PushDownPredicates: [{is_delete IN [0]}, {state IN [2]}, {DORIS_DELETE_SIGN IN [0]}, {update_time << [2025-08-26 14:20:55]}, {task_status NOT IN [5]}]
- KeyRanges: ScanKeys:ScanKey=[null(0000-01-01 00:00:00) : 9999-12-31 23:59:59.999999]
- TabletIds: [10918054, 10918067, 10918080, 10918028, 10918041, 10918119, 10918132, 10918145, 10918093, 10918106, 10917937, 10917950, 10917898, 10917911, 10917924, 10917989, 10918002, 10918015, 10917963, 10917976, 10918314, 10918327, 10918340, 10918288, 10918301, 10918379, 10918392, 10918353, 10918366, 10918184, 10918197, 10918210, 10918158, 10918171, 10918249, 10918262, 10918275, 10918223, 10918236, 10918574, 10918587, 10918535, 10918548, 10918561, 10918639, 10918652, 10917625, 10918600, 10918613, 10918626, 10918444, 10918457, 10918405, 10918418, 10918431, 10918509, 10918522, 10918470, 10918483, 10918496, 10917807, 10917820, 10917768, 10917781, 10917794, 10917872, 10920953, 10917885, 10917833, 10917846, 10917859, 10918704, 10917677, 10917690, 10918665, 10917638, 10917651, 10918678, 10918691, 10917664, 10917742, 10917755, 10917703, 10917716, 10917729]
- UseSpecificThreadToken: False
- AcquireRuntimeFilterTime: 613ns
- BlocksProduced: 0
- CloseTime: 19.530ms
- ExecTime: 13s388ms
- InitTime: 107.494us
- KeyRangesNum: 0
- MaxScannerThreadNum: 48
- MemoryUsage:
- PeakMemoryUsage: 0.00
- NumScanners: 85
- OpenTime: 484.398ms
- ProcessConjunctTime: 204.653us
- ProjectionTime: 0ns
- RowsProduced: 0
- RowsRead: 21.546211M (21546211)
- RuntimeFilterInfo:
- ScanBytes: 2.88 GB
- ScanRows: 25.236726M (25236726)
- ScannerWorkerWaitTime: 3s942ms
- TabletNum: 85
- TotalReadThroughput: 0
- WaitForDependency[OLAP_SCAN_OPERATOR_DEPENDENCY]Time: 11s866ms
- WaitForRuntimeFilter: 997.36ms
RuntimeFilter: (id = 6, type = in_or_bloomfilter):
- Info: [IsPushDown = false, RuntimeFilterState = READY, HasRemoteTarget = false, HasLocalTarget = true, Ignored = false]
- RealRuntimeFilterType: in
- InFilterSize: 0
- always_true: 0
- expr_filtered_rows: 21.546211M (21546211)
- expr_input_rows: 21.546211M (21546211)
RuntimeFilter: (id = 7, type = in_or_bloomfilter):
- Info: [IsPushDown = false, RuntimeFilterState = READY, HasRemoteTarget = false, HasLocalTarget = true, Ignored = false]
- RealRuntimeFilterType: in
- InFilterSize: 0
- always_true: 0
- expr_filtered_rows: 0
- expr_input_rows: 0

OLAP_SCAN_OPERATOR (id=16. table name = or_task_count(or_task_count)):
- RuntimeFilters: : RuntimeFilter: (id = 6, type = in_or_bloomfilter, need_local_merge: false, is_broadcast: false, build_bf_cardinality: false, RuntimeFilter: (id = 7, type = in_or_bloomfilter, need_local_merge: false, is_broadcast: false, build_bf_cardinality: false,
- PushDownPredicates: [{is_delete IN [0]}, {state IN [2]}, {DORIS_DELETE_SIGN IN [0]}, {update_time << [2025-08-26 14:20:55]}, {task_status NOT IN [5]}]
- KeyRanges: ScanKeys:ScanKey=[null(0000-01-01 00:00:00) : 9999-12-31 23:59:59.999999]
- TabletIds: [10918050, 10918063, 10918076, 10918024, 10918037, 10918115, 10918128, 10918141, 10918089, 10918102, 10917933, 10917946, 10917894, 10917907, 10917920, 10917985, 10917998, 10918011, 10917959, 10917972, 10918310, 10918323, 10918336, 10918284, 10918297, 10918375, 10918388, 10918349, 10918362, 10918180, 10918193, 10918206, 10918154, 10918167, 10918245, 10918258, 10918271, 10918219, 10918232, 10918570, 10918583, 10918531, 10918544, 10918557, 10918635, 10918648, 10917621, 10918596, 10918609, 10918622, 10918440, 10918453, 10918401, 10918414, 10918427, 10918505, 10918518, 10918466, 10918479, 10918492, 10917803, 10917816, 10917764, 10917777, 10917790, 10917868, 10920949, 10917881, 10917829, 10917842, 10917855, 10918700, 10917673, 10917686, 10918661, 10917634, 10917647, 10918674, 10918687, 10917660, 10917738, 10917751, 10917699, 10917712, 10917725, 10918058, 10918071, 10918084, 10918032, 10918045, 10918123, 10918136, 10918149, 10918097, 10918110, 10917941, 10917954, 10917902, 10917915, 10917928, 10917993, 10918006, 10918019, 10917967, 10917980, 10918318, 10918331, 10918344, 10918292, 10918305, 10918383, 10918396, 10918357, 10918370, 10918188, 10918201, 10918214, 10918162, 10918175, 10918253, 10918266, 10918279, 10918227, 10918240, 10918578, 10918591, 10918539, 10918552, 10918565, 10918643, 10918656, 10917629, 10918604, 10918617, 10918630, 10918448, 10918461, 10918409, 10918422, 10918435, 10918513, 10918526, 10918474, 10918487, 10918500, 10917811, 10917824, 10917772, 10917785, 10917798, 10917876, 10920957, 10917889, 10917837, 10917850, 10917863, 10918708, 10917681, 10917694, 10918669, 10917642, 10917655, 10918682, 10918695, 10917668, 10917746, 10917759, 10917707, 10917720, 10917733]
- UseSpecificThreadToken: False
- AcquireRuntimeFilterTime: 581ns
- BlocksProduced: 1
- CloseTime: 36.265ms
- ExecTime: 25s343ms
- InitTime: 138.95us
- KeyRangesNum: 0
- MaxScannerThreadNum: 48
- MemoryUsage:
- PeakMemoryUsage: 0.00
- NumScanners: 170
- OpenTime: 238.479ms
- ProcessConjunctTime: 257.903us
- ProjectionTime: 0ns
- RowsProduced: 1
- RowsRead: 35.157589M (35157589)
- RuntimeFilterInfo:
- ScanBytes: 5.30 GB
- ScanRows: 46.353358M (46353358)
- ScannerWorkerWaitTime: 23s356ms
- TabletNum: 170
- TotalReadThroughput: 0
- WaitForDependency[OLAP_SCAN_OPERATOR_DEPENDENCY]Time: 24s53ms
- WaitForRuntimeFilter: 993.973ms
RuntimeFilter: (id = 6, type = in_or_bloomfilter):
- Info: [IsPushDown = false, RuntimeFilterState = READY, HasRemoteTarget = false, HasLocalTarget = true, Ignored = false]
- RealRuntimeFilterType: in
- InFilterSize: 1
- always_true: 0
- expr_filtered_rows: 39.75708M (39757080)
- expr_input_rows: 39.757081M (39757081)
RuntimeFilter: (id = 7, type = in_or_bloomfilter):
- Info: [IsPushDown = false, RuntimeFilterState = READY, HasRemoteTarget = false, HasLocalTarget = true, Ignored = false]
- RealRuntimeFilterType: in
- InFilterSize: 1
- always_true: 0
- expr_filtered_rows: 741
- expr_input_rows: 802