问题说明:
运用场景:当前需要查询MD5字段不匹配的总数,执行过程提示内存不足,内存默认只给了2G,通过监控程序发现3个BE节点的程序内存均已达到20多G(总共70多G)
SQL语句:
select count(1) from t_pm_parameter_value where object_uid != upper(md5(object_fdn));
BE内存配置:
JAVA_OPTS_FOR_JDK_17="-Dfile.encoding=UTF-8 -Djol.skipHotspotSAAttach=true -Xmx2G -DlogPath=$LOG_DIR/jni.log -Xlog:gc*:$LOG_DIR/be.gc.log.$CUR_DATE:time,uptime:filecount=10,filesize=50M -Djavax.security.auth.useSubjectCredsOnly=false -Dsun.security.krb5.debug=true -Dsun.java.command=DorisBE -XX:-CriticalJNINatives -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED --add-opens=java.management/sun.management=ALL-UNNAMED -Darrow.enable_null_check_for_get=false"
表创建语句:
CREATE TABLE IF NOT EXISTS `t_pm_parameter_value` (
`ems_name` VARCHAR(64) NOT NULL COMMENT "",
`cycle_time` VARCHAR(20) NULL COMMENT "",
`object_uid` VARCHAR(64) NULL COMMENT "",
`object_type` VARCHAR(255) NULL COMMENT "",
`granularity` SMALLINT NULL COMMENT "",
`standard_parameter_name` VARCHAR(64) NULL COMMENT "",
`layer_rate` VARCHAR(255) NULL COMMENT "",
`location` VARCHAR(32) NULL COMMENT "",
`parameter_name` VARCHAR(64) NULL COMMENT "",
`value` VARCHAR(32) NULL COMMENT "",
`unit` VARCHAR(32) NULL COMMENT "",
`object_fdn` VARCHAR(255) NULL COMMENT "",
`request_id` VARCHAR(20) NULL COMMENT "请求ID",
`create_time` DATETIME NULL DEFAULT CURRENT_TIMESTAMP COMMENT ""
)
ENGINE=olap
DUPLICATE KEY(`ems_name`, `cycle_time`, `object_uid`,`object_type`, `granularity`,`standard_parameter_name`)
COMMENT "性能指标值表(纵表)"
DISTRIBUTED BY HASH(`ems_name`, `cycle_time`) BUCKETS 10
PROPERTIES (
"replication_num" = "1",
"bloom_filter_columns"="request_id"
);
疑问:
1、如何解决当前问题?
2、关于内存不足报错的问题,如何提前避免,如何知道当前执行SQL大概需要多少内存?
相关截图:

