Doris 3.0.7 存算分离FE内存泄露问题

Viewed 39

集群情况:
Doris 版本 3.0.7,FE和BE都是独立节点部署。
FE 8C32G 3台
BE 12C48G 5台

JAVA_OPTS_FOR_JDK_17="-Djava.security.krb5.conf=/etc/krb5.conf -Dallow_weak_crypto=true -Dfile.encoding=UTF-8 -Djavax.security.auth.useSubjectCredsOnly=false -Xmx23552m  -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$LOG_DIR -Xlog:gc*,classhisto*=trace:$LOG_DIR/fe.gc.log.$CUR_DATE:time,uptime:filecount=10,filesize=50M --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens java.base/jdk.internal.ref=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED"

FE节点内存总是有这种直线上升情况,看gc日志,发现没有fullgc。这个下降是强制重启FE恢复正常,手工触发fullgc 也会正常。
image.png

jmap -histo top 20

 num     #instances         #bytes  class name (module)
-------------------------------------------------------
   1:     235393575     7352541896  [B (java.base@17.0.2)
   2:       4374583      104989992  com.google.common.base.Suppliers$NonSerializableMemoizingSupplier
   3:         78152       82698672  [I (java.base@17.0.2)
   4:       1154062       54774104  [Ljava.lang.Object; (java.base@17.0.2)
   5:        482205       42434040  org.apache.doris.nereids.trees.expressions.SlotReference
   6:       1617733       38825592  java.lang.String (java.base@17.0.2)
   7:        862048       27585536  java.util.HashMap$Node (java.base@17.0.2)
   8:        397685       25451840  org.apache.doris.nereids.trees.expressions.Alias
   9:        380025       24321600  org.apache.doris.nereids.trees.expressions.literal.VarcharLiteral
  10:        781698       18760752  java.util.ArrayList (java.base@17.0.2)
  11:       1020715       16331440  java.lang.Integer (java.base@17.0.2)
  12:         20190       16074688  [C (java.base@17.0.2)
  13:        108509       14011120  [Ljava.util.HashMap$Node; (java.base@17.0.2)
  14:        865799       13852784  org.apache.doris.nereids.trees.expressions.ExprId
  15:        551426       13234224  org.apache.doris.nereids.util.MutableState$SingleMutableState
  16:        818429       13094864  org.apache.doris.nereids.trees.expressions.Expression$$Lambda$445/0x00000008016bd840
  17:        818240       13091840  org.apache.doris.nereids.trees.expressions.Expression$$Lambda$444/0x00000008016bd390
  18:        813592       13017472  org.apache.doris.nereids.trees.expressions.Expression$$Lambda$446/0x00000008016bdcf0

如果手工触发fullgc,内存就会下降。这种情况应该如何排查处理。

0 Answers