doris2.1.7的routineload方式应该如何权衡消费参数问题?

Viewed 10

当前我们doris版本是2.1.7,采用的是routine load方式消费kafka入库,
我们是也有一个单独的stream节点消费routinelaod,
当前我们主题分区为3个,消费方式如下所示:
CREATE ROUTINE LOAD WIP_TEST_DATA_STREAM1 ON mes_wip_test_data1
WITH APPEND
ORDER BY TEST_TIME
PROPERTIES
(
"desired_concurrent_number" = "3",
"max_error_number" = "0",
"max_filter_ratio" = "1.0",
"max_batch_interval" = "30",
"max_batch_rows" = "1000000",
"max_batch_size" = "2147483648",
"format" = "json",
"strip_outer_array" = "true",
"num_as_string" = "false",
"fuzzy_parse" = "false",
"strict_mode" = "false",
"timezone" = "Asia/Shanghai",
"exec_mem_limit" = "2147483648"
)
FROM KAFKA
(
"kafka_broker_list" = "10.11.148.19:8092",
"kafka_topic" = "WIP_TEST1",
"property.enable.auto.commit" = "false",
"property.kafka_default_offsets" = "OFFSET_BEGINNING",
"property.group.id" = "test_v3",
"kafka_partitions" = "0,1,2",
"kafka_offsets" = "OFFSET_BEGINNING,OFFSET_BEGINNING,OFFSET_BEGINNING"
);
但是监控该节点下的be出现:
consumer group done: 9749d9757a762f28-b66cdf42374ac98e. consume time(ms)=30214, received rows=3782, receiv
ed bytes=763429867, eos: 0, left_time: -214, left_rows: 996218, left_bytes: 1384053781, blocking get time(us): 22339635, blocking put time(us): 3759384, id=72ebf6b9e3b
342f7-9cc479ebcc4cab1d, job_id=60184695, txn_id=54635156, label=WIP_TEST_DATA_STREAM-60184695-72ebf6b9e3b342f7-9cc479ebcc4cab1d-54635156, elapse(s)=30
I20250813 09:13:39.821861 18163 data_consumer.cpp:496] kafka consumer cancelled. fb4ba29c1f965a36-acf8559b70fe5c8e
到底应该如何平衡盖下面的参数,尤其是max_batch_interval/max_batch_size等,请大佬哦解答下吧。多次调节参数后,该主题下经常报警消费慢,该stream节点下的磁盘是机械硬盘,该磁盘下的情况如下
DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util
09:22:34 AM vg01-data 736.00 32048.00 19064.00 69.45 1.03 1.43 0.76 56.30
09:22:35 AM vg01-data 373.00 31560.00 21464.00 142.16 0.80 2.14 1.39 51.80
09:22:36 AM vg01-data 529.00 40800.00 6088.00 88.64 0.86 1.58 0.91 48.40
09:22:37 AM vg01-data 1126.00 79048.00 77712.00 139.22 3.27 2.88 0.85 95.60
09:22:38 AM vg01-data 843.00 73488.00 83600.00 186.34 2.14 2.58 1.10 93.00
09:22:39 AM vg01-data 900.00 98752.00 4632.00 114.87 1.77 1.97 0.97 87.00
09:22:40 AM vg01-data 1370.00 63312.00 11936.00 54.93 2.32 1.68 0.65 88.60
09:22:41 AM vg01-data 1011.00 50504.00 288872.00 335.68 3.21 3.16 0.90 91.40
09:22:42 AM vg01-data 330.00 32032.00 8056.00 121.48 1.02 3.10 1.88 62.10
09:22:43 AM vg01-data 1356.00 38072.00 136504.00 128.74 2.96 2.15 0.62 83.70
Average: vg01-data 857.40 53961.60 65792.80 139.67 1.94 2.25 0.88 75.79

0 Answers