Doris 4.0.2 MD5函数导致BE内存过高问题

Viewed 10

问题说明:
运用场景:当前需要查询MD5字段不匹配的总数,执行过程提示内存不足,内存默认只给了2G,通过监控程序发现3个BE节点的程序内存均已达到20多G(总共70多G)

SQL语句:

select count(1) from t_pm_parameter_value where object_uid != upper(md5(object_fdn));

BE内存配置:

JAVA_OPTS_FOR_JDK_17="-Dfile.encoding=UTF-8 -Djol.skipHotspotSAAttach=true -Xmx2G -DlogPath=$LOG_DIR/jni.log -Xlog:gc*:$LOG_DIR/be.gc.log.$CUR_DATE:time,uptime:filecount=10,filesize=50M -Djavax.security.auth.useSubjectCredsOnly=false -Dsun.security.krb5.debug=true -Dsun.java.command=DorisBE -XX:-CriticalJNINatives -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED --add-opens=java.management/sun.management=ALL-UNNAMED -Darrow.enable_null_check_for_get=false"

表创建语句:

CREATE TABLE IF NOT EXISTS `t_pm_parameter_value` (
    `ems_name` VARCHAR(64) NOT NULL COMMENT "",
    `cycle_time` VARCHAR(20) NULL COMMENT "",
    `object_uid` VARCHAR(64) NULL COMMENT "",
    `object_type` VARCHAR(255) NULL COMMENT "",
    `granularity` SMALLINT NULL COMMENT "",
    `standard_parameter_name` VARCHAR(64) NULL COMMENT "",
    `layer_rate` VARCHAR(255) NULL COMMENT "",
    `location` VARCHAR(32) NULL COMMENT "",
    `parameter_name` VARCHAR(64) NULL COMMENT "",
    `value` VARCHAR(32) NULL COMMENT "",
    `unit` VARCHAR(32) NULL COMMENT "",
    `object_fdn` VARCHAR(255) NULL COMMENT "",
    `request_id` VARCHAR(20) NULL COMMENT "请求ID",
    `create_time` DATETIME NULL DEFAULT CURRENT_TIMESTAMP COMMENT ""
)
ENGINE=olap
DUPLICATE KEY(`ems_name`, `cycle_time`, `object_uid`,`object_type`, `granularity`,`standard_parameter_name`)
COMMENT "性能指标值表(纵表)"
DISTRIBUTED BY HASH(`ems_name`, `cycle_time`) BUCKETS 10
PROPERTIES (
  "replication_num" = "1",
  "bloom_filter_columns"="request_id"
);

疑问:
1、如何解决当前问题?
2、关于内存不足报错的问题,如何提前避免,如何知道当前执行SQL大概需要多少内存?

相关截图:
企业微信截图_17682003393278.png

企业微信截图_20260112150803.png

0 Answers