使用doris 的be节点出现重启问题

Question

您好！
我在使用k8s部署doris测试的过程中，出现了如下的奔溃重启情况，请问我该如何调整我的配置，或者有没有最佳实践的配置指导？
pod状态：

pod奔溃日志

进程被系统给杀

apiVersion: doris.selectdb.com/v1
kind: DorisCluster
metadata:
  labels:
    app.kubernetes.io/name: doriscluster
    app.kubernetes.io/instance: doriscluster-sample
    app.kubernetes.io/part-of: doris-operator
  name: doriscluster-sample
spec:
  adminUser:
    name: root
    password: a123456
  feSpec:
    configMapInfo:
      configMapName: doriscluster-sample-conf
      resolveKey: fe.conf
    replicas: 3
    service:
      type: NodePort
    limits:
      cpu: 4
      memory: 8Gi
    requests:
      cpu: 2
      memory: 4Gi
    image: 192.168.18.165:8443/apache/doris.fe-ubuntu:2.0.11-1
    envVars:
      - name: TZ
        value: "Asia/Shanghai"
    persistentVolumes:
      - mountPath: /opt/apache-doris/fe/doris-meta
        name: fe-doris-meta
        persistentVolumeClaimSpec:
          storageClassName: local-path
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 10Gi
      - mountPath: /opt/apache-doris/fe/log
        name: fe-log
        persistentVolumeClaimSpec:
          storageClassName: local-path
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 10Gi
  beSpec:
    configMapInfo:
      configMapName: doriscluster-sample-conf
      resolveKey: be.conf
    replicas: 3
    service:
      type: NodePort
    limits:
      cpu: 4
      memory: 8Gi
    requests:
      cpu: 1
      memory: 2Gi
    image: 192.168.18.165:8443/apache/doris.be-ubuntu:2.0.11
    envVars:
      - name: TZ
        value: "Asia/Shanghai"
    systemInitialization:
      command: ["/sbin/sysctl", "-w", "vm.max_map_count=2000000"]
      initImage: 192.168.18.165:8443/apache/alpine:latest
    persistentVolumes:
      - mountPath: /opt/apache-doris/be/storage
        name: be-storage
        persistentVolumeClaimSpec:
          storageClassName: local-path
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 10Gi
      - mountPath: /opt/apache-doris/be/log
        name: be-log
        persistentVolumeClaimSpec:
          storageClassName: local-path
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 10Gi

apiVersion: v1
kind: ConfigMap
metadata:
  name: doriscluster-sample-conf
data:
  fe.conf: |
    CUR_DATE=`date +%Y%m%d-%H%M%S`

    # the output dir of stderr and stdout
    LOG_DIR = ${DORIS_HOME}/log

    JAVA_OPTS="-Djavax.security.auth.useSubjectCredsOnly=false -Xss4m -Xmx8192m -XX:+UseMembar -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xloggc:$DORIS_HOME/log/fe.gc.log.$CUR_DATE"

    # For jdk 9+, this JAVA_OPTS will be used as default JVM options
    JAVA_OPTS_FOR_JDK_9="-Djavax.security.auth.useSubjectCredsOnly=false -Xss4m -Xmx8192m -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xlog:gc*:$DORIS_HOME/log/fe.gc.log.$CUR_DATE:time"

    enable_fqdn_mode = true

    # INFO, WARN, ERROR, FATAL
    sys_log_level = INFO

    # NORMAL, BRIEF, ASYNC
    sys_log_mode = NORMAL

    # Default dirs to put jdbc drivers,default value is ${DORIS_HOME}/jdbc_drivers
    # jdbc_drivers_dir = ${DORIS_HOME}/jdbc_drivers

    http_port = 8030
    rpc_port = 9020
    query_port = 9030
    edit_log_port = 9010
  be.conf: |
    CUR_DATE=`date +%Y%m%d-%H%M%S`

    PPROF_TMPDIR="$DORIS_HOME/log/"

    JAVA_OPTS="-Xmx1024m -DlogPath=$DORIS_HOME/log/jni.log -Xloggc:$DORIS_HOME/log/be.gc.log.$CUR_DATE -Djavax.security.auth.useSubjectCredsOnly=false -Dsun.java.command=DorisBE -XX:-CriticalJNINatives -DJDBC_MIN_POOL=1 -DJDBC_MAX_POOL=100 -DJDBC_MAX_IDLE_TIME=300000 -DJDBC_MAX_WAIT_TIME=5000"

    # For jdk 9+, this JAVA_OPTS will be used as default JVM options
    JAVA_OPTS_FOR_JDK_9="-Xmx1024m -DlogPath=$DORIS_HOME/log/jni.log -Xlog:gc:$DORIS_HOME/log/be.gc.log.$CUR_DATE -Djavax.security.auth.useSubjectCredsOnly=false -Dsun.java.command=DorisBE -XX:-CriticalJNINatives -DJDBC_MIN_POOL=1 -DJDBC_MAX_POOL=100 -DJDBC_MAX_IDLE_TIME=300000 -DJDBC_MAX_WAIT_TIME=5000"

    # since 1.2, the JAVA_HOME need to be set to run BE process.
    # JAVA_HOME=/path/to/jdk/

    # https://github.com/apache/doris/blob/master/docs/zh-CN/community/developer-guide/debug-tool.md#jemalloc-heap-profile
    # https://jemalloc.net/jemalloc.3.html
    JEMALLOC_CONF="percpu_arena:percpu,background_thread:true,metadata_thp:auto,muzzy_decay_ms:15000,dirty_decay_ms:15000,oversize_threshold:0,lg_tcache_max:20,prof:false,lg_prof_interval:32,lg_prof_sample:19,prof_gdump:false,prof_accum:false,prof_leak:false,prof_final:false"
    JEMALLOC_PROF_PRFIX=""

    enable_fqdn_mode = true

    # INFO, WARNING, ERROR, FATAL
    sys_log_level = INFO

    # ports for admin, web, heartbeat service
    be_port = 9060
    webserver_port = 8040
    heartbeat_service_port = 9050
    brpc_port = 8060
    enable_https = false

阿渊@SelectDB · Accepted Answer

这个是因为内存资源给的太少，导致OOM，被kill了，这个需要加内存资源试试了

____ · Answer

这个内存资源是在哪些参数可以控制吗？比如类似myssql 的innodb_buffer_pool_size一样占用3/4内存计算公式，可否分享一下调优经验，如：1G数据需要多大的内存，有哪些控制参数可以设置，
我的数据压缩后，data占用总大小如下：

____ · Answer

还报了这个错误，集群节点一直起不来

2024-07-21 22:48:27,774 WARN (Analysis Job Executor-0|396) [OlapAnalysisTask.doExecute():75] Table 0.12002.12362, partitionNames for column ExchangeStates is null. ColToPartitions:[{}]
2024-07-21 22:48:27,776 WARN (Analysis Job Executor-0|396) [StmtExecutor.executeByLegacy():844] execute Exception. stmt[790, a91fa8783b4c450f-b98d1c836e06afe6]
org.apache.doris.common.UserException: errCode = 3, detailMessage = tablet 11135 alive replica num 1 < quorumreplica num 2, alive backends: [11002], detail: Visible Replicas:Visible version: 1, Replicas: [replicaId=11136, BackendId=11002, version=1, dataSize=0, rowCount=0, lastFailedVersion=-1, lastSuccessVersion=1, lastFailedTimestamp=0, schemaHash=409617337, state=NORMAL], Backends: Backend [id=11002, isDecommission: false, backendStatus: [lastSuccessReportTabletsTime='2024-07-21 22:48:01', lastStreamLoadTime=-1, isQueryDisabled=false, isLoadDisabled=false], isAlive: true, lastUpdateTime: 2024-07-21 22:48:23
        at org.apache.doris.planner.OlapTableSink.createLocation(OlapTableSink.java:440) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.OlapTableSink.complete(OlapTableSink.java:195) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.analysis.NativeInsertStmt.complete(NativeInsertStmt.java:947) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.OriginalPlanner.createPlanFragments(OriginalPlanner.java:249) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.OriginalPlanner.plan(OriginalPlanner.java:102) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.analyzeAndGenerateQueryPlan(StmtExecutor.java:1232) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.analyze(StmtExecutor.java:1060) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.executeByLegacy(StmtExecutor.java:735) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:507) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:475) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.statistics.AnalysisJob.executeWithExceptionOnFail(AnalysisJob.java:149) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.statistics.AnalysisJob.writeBuf(AnalysisJob.java:133) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.statistics.AnalysisJob.markOneTaskDone(AnalysisJob.java:87) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.statistics.AnalysisJob.appendBuf(AnalysisJob.java:75) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.statistics.OlapAnalysisTask.doExecute(OlapAnalysisTask.java:80) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.statistics.BaseAnalysisTask.executeWithRetry(BaseAnalysisTask.java:214) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.statistics.BaseAnalysisTask.execute(BaseAnalysisTask.java:199) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.statistics.AnalysisTaskWrapper.lambda$new$0(AnalysisTaskWrapper.java:43) ~[doris-fe.jar:1.2-SNAPSHOT]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_352-352]
        at org.apache.doris.statistics.AnalysisTaskWrapper.run(AnalysisTaskWrapper.java:66) ~[doris-fe.jar:1.2-SNAPSHOT]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_352-352]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_352-352]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_352-352]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_352-352]
        at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_352-352]
2024-07-21 22:48:27,820 WARN (Analysis Job Executor-0|396) [BaseAnalysisTask.executeWithRetry():220] Failed to execute analysis task, retried times: 2
java.lang.RuntimeException: Failed to analyze: Failed to insert : INSERT INTO internal.__internal_schema.column_statistics VALUES ('12362--1-GoodsNumbers',0,12002,12362,-1,'GoodsNumbers',null,1616,24,0,'1','40',6464,'2024-07-21 22:47:56'),('12362--1-States',0,12002,12362,-1,'States',null,1616,1,0,'1','1',6464,'2024-07-21 22:47:57'),('12362--1-ProvinceName',0,12002,12362,-1,'ProvinceName',null,1616,1,1616,'null','null',0,'2024-07-21 22:47:57'),('12362--1-VerifyCode',0,12002,12362,-1,'VerifyCode',null,1616,1018,599,'1645800385722','1719472066607',13221,'2024-07-21 22:47:57'),('12362--1-GoodsImage',0,12002,12362,-1,'GoodsImage',null,1616,98,0,'','https://county-media-img-oss.xinhuaapp.com/Party/Cms/goods/fc429/4ec01fc4294ec01c67e6bc697bcd43fc86849.png?x-oss-process=image/rotate,0/crop,x_583,y_0,w_1070,h_803/quality,q_80/format,png/resize,w_750',244725,'2024-07-21 22:47:57'),('12362--1-InputFormValue',0,12002,12362,-1,'InputFormValue',null,1616,25,787,'','������������������������������������������������������',750,'2024-07-21 22:47:57'),('12362--1-FormId',0,12002,12362,-1,'FormId',null,1616,1616,0,'125081372533536C0593','plugin-1814174369525770198',23678,'2024-07-21 22:47:57'),('12362--1-UserPost',0,12002,12362,-1,'UserPost',null,1616,1,0,'','',0,'2024-07-21 22:47:57'),('12362--1-ExchangeType',0,12002,12362,-1,'ExchangeType',null,1616,3,0,'1','8',6464,'2024-07-21 22:47:57'),('12362--1-UserName',0,12002,12362,-1,'UserName',null,1616,470,0,'**','������������������������������������������������',16257,'2024-07-21 22:47:57'),('12362--1-CreateTime',0,12002,12362,-1,'CreateTime',null,1616,1616,0,'2019-10-11 00:44:35','2024-06-27 23:07:46',12928,'2024-07-21 22:47:57'),('12362--1-GoodsTitle',0,12002,12362,-1,'GoodsTitle',null,1616,180,0,'(���������������������������)������������������������������������������������������������������������1.5L���������������������������','���������������������������������������������������������������������������������',33187,'2024-07-21 22:47:57'),('12362--1-ProjectId',0,12002,12362,-1,'ProjectId',null,1616,1,0,'15','15',12928,'2024-07-21 22:47:57'),('12362--1-CityName',0,12002,12362,-1,'CityName',null,1616,1,1616,'null','null',0,'2024-07-21 22:47:57'),('12362--1-CountyName',0,12002,12362,-1,'CountyName',null,1616,1,1616,'null','null',0,'2024-07-21 22:47:57'),('12362--1-GoodsPrices',0,12002,12362,-1,'GoodsPrices',null,1616,60,0,'1','12000',6464,'2024-07-21 22:47:57'),('12362--1-AppId',0,12002,12362,-1,'AppId',null,1616,10,0,'110151','110300',12928,'2024-07-21 22:47:57'),('12362--1-UserId',0,12002,12362,-1,'UserId',null,1616,470,0,'35098','4365971',12928,'2024-07-21 22:47:57'),('12362--1-UpdateTime',0,12002,12362,-1,'UpdateTime',null,1616,1616,0,'2019-10-11 00:44:35','2024-06-27 23:07:46',12928,'2024-07-21 22:47:57'),('12362--1-Id',0,12002,12362,-1,'Id',null,1616,1616,0,'533','238087',12928,'2024-07-21 22:47:57'),('12362--1-ProvinceCode',0,12002,12362,-1,'ProvinceCode',null,1616,1,1616,'null','null',0,'2024-07-21 22:47:57'),('12362--1-GoodsId',0,12002,12362,-1,'GoodsId',null,1616,100,0,'0','110959',12928,'2024-07-21 22:47:57'),('12362--1-UserMobile',0,12002,12362,-1,'UserMobile',null,1616,418,0,'','19997999169',16379,'2024-07-21 22:47:57'),('12362--1-CityCode',0,12002,12362,-1,'CityCode',null,1616,1,1616,'null','null',0,'2024-07-21 22:47:57'),('12362--1-CountyCode',0,12002,12362,-1,'CountyCode',null,1616,1,1616,'null','null',0,'2024-07-21 22:47:57'),('12362--1-CourierNumber',0,12002,12362,-1,'CourierNumber',null,1616,2,626,'','',0,'2024-07-21 22:47:57'),('12362--1-UserAddress',0,12002,12362,-1,'UserAddress',null,1616,4,0,'','���������������������������2���������',37,'2024-07-21 22:47:57'),('12362--1-ExchangeStates',0,12002,12362,-1,'ExchangeStates',null,1616,3,0,'1','4',12928,'2024-07-21 22:47:57'),('12362--1-ExchangeStates',0,12002,12362,-1,'ExchangeStates',null,0,0,0,NULL,NULL,0,'2024-07-21 22:48:27'),('12362--1-ExchangeStates',0,12002,12362,-1,'ExchangeStates',null,0,0,0,NULL,NULL,0,'2024-07-21 22:48:27')Error msg: errCode = 3, detailMessage = tablet 11135 alive replica num 1 < quorumreplica num 2, alive backends: [11002], detail: Visible Replicas:Visible version: 1, Replicas: [replicaId=11136, BackendId=11002, version=1, dataSize=0, rowCount=0, lastFailedVersion=-1, lastSuccessVersion=1, lastFailedTimestamp=0, schemaHash=409617337, state=NORMAL], Backends: Backend [id=11002, isDecommission: false, backendStatus: [lastSuccessReportTabletsTime='2024-07-21 22:48:01', lastStreamLoadTime=-1, isQueryDisabled=false, isLoadDisabled=false], isAlive: true, lastUpdateTime: 2024-07-21 22:48:23
        at org.apache.doris.statistics.AnalysisJob.writeBuf(AnalysisJob.java:135) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.statistics.AnalysisJob.markOneTaskDone(AnalysisJob.java:87) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.statistics.AnalysisJob.appendBuf(AnalysisJob.java:75) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.statistics.OlapAnalysisTask.doExecute(OlapAnalysisTask.java:80) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.statistics.BaseAnalysisTask.executeWithRetry(BaseAnalysisTask.java:214) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.statistics.BaseAnalysisTask.execute(BaseAnalysisTask.java:199) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.statistics.AnalysisTaskWrapper.lambda$new$0(AnalysisTaskWrapper.java:43) ~[doris-fe.jar:1.2-SNAPSHOT]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_352-352]
        at org.apache.doris.statistics.AnalysisTaskWrapper.run(AnalysisTaskWrapper.java:66) ~[doris-fe.jar:1.2-SNAPSHOT]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_352-352]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_352-352]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_352-352]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_352-352]
        at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_352-352]
2024-07-21 22:48:37,854 WARN (Analysis Job Executor-0|396) [OlapAnalysisTask.doExecute():75] Table 0.12002.12362, partitionNames for column ExchangeStates is null. ColToPartitions:[{}]
2024-07-21 22:48:37,857 WARN (Analysis Job Executor-0|396) [StmtExecutor.executeByLegacy():844] execute Exception. stmt[792, ae4df665552464d-99c3718500161f07]
org.apache.doris.common.UserException: errCode = 3, detailMessage = tablet 11135 alive replica num 1 < quorumreplica num 2, alive backends: [11002], detail: Visible Replicas:Visible version: 1, Replicas: [replicaId=11136, BackendId=11002, version=1, dataSize=0, rowCount=0, lastFailedVersion=-1, lastSuccessVersion=1, lastFailedTimestamp=0, schemaHash=409617337, state=NORMAL], Backends: Backend [id=11002, isDecommission: false, backendStatus: [lastSuccessReportTabletsTime='2024-07-21 22:48:01', lastStreamLoadTime=-1, isQueryDisabled=false, isLoadDisabled=false], isAlive: true, lastUpdateTime: 2024-07-21 22:48:33
        at org.apache.doris.planner.OlapTableSink.createLocation(OlapTableSink.java:440) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.OlapTableSink.complete(OlapTableSink.java:195) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.analysis.NativeInsertStmt.complete(NativeInsertStmt.java:947) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.OriginalPlanner.createPlanFragments(OriginalPlanner.java:249) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.OriginalPlanner.plan(OriginalPlanner.java:102) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.analyzeAndGenerateQueryPlan(StmtExecutor.java:1232) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.analyze(StmtExecutor.java:1060) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.executeByLegacy(StmtExecutor.java:735) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:507) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:475) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.statistics.AnalysisJob.executeWithExceptionOnFail(AnalysisJob.java:149) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.statistics.AnalysisJob.writeBuf(AnalysisJob.java:133) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.statistics.AnalysisJob.markOneTaskDone(AnalysisJob.java:87) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.statistics.AnalysisJob.appendBuf(AnalysisJob.java:75) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.statistics.OlapAnalysisTask.doExecute(OlapAnalysisTask.java:80) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.statistics.BaseAnalysisTask.executeWithRetry(BaseAnalysisTask.java:214) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.statistics.BaseAnalysisTask.execute(BaseAnalysisTask.java:199) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.statistics.AnalysisTaskWrapper.lambda$new$0(AnalysisTaskWrapper.java:43) ~[doris-fe.jar:1.2-SNAPSHOT]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_352-352]
        at org.apache.doris.statistics.AnalysisTaskWrapper.run(AnalysisTaskWrapper.java:66) ~[doris-fe.jar:1.2-SNAPSHOT]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_352-352]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_352-352]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_352-352]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_352-352]
        at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_352-352]
2024-07-21 22:48:37,866 WARN (Analysis Job Executor-0|396) [BaseAnalysisTask.executeWithRetry():220] Failed to execute analysis task, retried times: 3
java.lang.RuntimeException: Failed to analyze: Failed to insert : INSERT INTO internal.__internal_schema.column_statistics VALUES ('12362--1-GoodsNumbers',0,12002,12362,-1,'GoodsNumbers',null,1616,24,0,'1','40',6464,'2024-07-21 22:47:56'),('12362--1-States',0,12002,12362,-1,'States',null,1616,1,0,'1','1',6464,'2024-07-21 22:47:57'),('12362--1-ProvinceName',0,12002,12362,-1,'ProvinceName',null,1616,1,1616,'null','null',0,'2024-07-21 22:47:57'),('12362--1-VerifyCode',0,12002,12362,-1,'VerifyCode',null,1616,1018,599,'1645800385722','1719472066607',13221,'2024-07-21 22:47:57'),('12362--1-GoodsImage',0,12002,12362,-1,'GoodsImage',null,1616,98,0,'','https://county-media-img-oss.xinhuaapp.com/Party/Cms/goods/fc429/4ec01fc4294ec01c67e6bc697bcd43fc86849.png?x-oss-process=image/rotate,0/crop,x_583,y_0,w_1070,h_803/quality,q_80/format,png/resize,w_750',244725,'2024-07-21 22:47:57'),('12362--1-InputFormValue',0,12002,12362,-1,'InputFormValue',null,1616,25,787,'','������������������������������������������������������',750,'2024-07-21 22:47:57'),('12362--1-FormId',0,12002,12362,-1,'FormId',null,1616,1616,0,'125081372533536C0593','plugin-1814174369525770198',23678,'2024-07-21 22:47:57'),('12362--1-UserPost',0,12002,12362,-1,'UserPost',null,1616,1,0,'','',0,'2024-07-21 22:47:57'),('12362--1-ExchangeType',0,12002,12362,-1,'ExchangeType',null,1616,3,0,'1','8',6464,'2024-07-21 22:47:57'),('12362--1-UserName',0,12002,12362,-1,'UserName',null,1616,470,0,'**','������������������������������������������������',16257,'2024-07-21 22:47:57'),('12362--1-CreateTime',0,12002,12362,-1,'CreateTime',null,1616,1616,0,'2019-10-11 00:44:35','2024-06-27 23:07:46',12928,'2024-07-21 22:47:57'),('12362--1-GoodsTitle',0,12002,12362,-1,'GoodsTitle',null,1616,180,0,'(���������������������������)������������������������������������������������������������������������1.5L���������������������������','���������������������������������������������������������������������������������',33187,'2024-07-21 22:47:57'),('12362--1-ProjectId',0,12002,12362,-1,'ProjectId',null,1616,1,0,'15','15',12928,'2024-07-21 22:47:57'),('12362--1-CityName',0,12002,12362,-1,'CityName',null,1616,1,1616,'null','null',0,'2024-07-21 22:47:57'),('12362--1-CountyName',0,12002,12362,-1,'CountyName',null,1616,1,1616,'null','null',0,'2024-07-21 22:47:57'),('12362--1-GoodsPrices',0,12002,12362,-1,'GoodsPrices',null,1616,60,0,'1','12000',6464,'2024-07-21 22:47:57'),('12362--1-AppId',0,12002,12362,-1,'AppId',null,1616,10,0,'110151','110300',12928,'2024-07-21 22:47:57'),('12362--1-UserId',0,12002,12362,-1,'UserId',null,1616,470,0,'35098','4365971',12928,'2024-07-21 22:47:57'),('12362--1-UpdateTime',0,12002,12362,-1,'UpdateTime',null,1616,1616,0,'2019-10-11 00:44:35','2024-06-27 23:07:46',12928,'2024-07-21 22:47:57'),('12362--1-Id',0,12002,12362,-1,'Id',null,1616,1616,0,'533','238087',12928,'2024-07-21 22:47:57'),('12362--1-ProvinceCode',0,12002,12362,-1,'ProvinceCode',null,1616,1,1616,'null','null',0,'2024-07-21 22:47:57'),('12362--1-GoodsId',0,12002,12362,-1,'GoodsId',null,1616,100,0,'0','110959',12928,'2024-07-21 22:47:57'),('12362--1-UserMobile',0,12002,12362,-1,'UserMobile',null,1616,418,0,'','19997999169',16379,'2024-07-21 22:47:57'),('12362--1-CityCode',0,12002,12362,-1,'CityCode',null,1616,1,1616,'null','null',0,'2024-07-21 22:47:57'),('12362--1-CountyCode',0,12002,12362,-1,'CountyCode',null,1616,1,1616,'null','null',0,'2024-07-21 22:47:57'),('12362--1-CourierNumber',0,12002,12362,-1,'CourierNumber',null,1616,2,626,'','',0,'2024-07-21 22:47:57'),('12362--1-UserAddress',0,12002,12362,-1,'UserAddress',null,1616,4,0,'','���������������������������2���������',37,'2024-07-21 22:47:57'),('12362--1-ExchangeStates',0,12002,12362,-1,'ExchangeStates',null,1616,3,0,'1','4',12928,'2024-07-21 22:47:57'),('12362--1-ExchangeStates',0,12002,12362,-1,'ExchangeStates',null,0,0,0,NULL,NULL,0,'2024-07-21 22:48:27'),('12362--1-ExchangeStates',0,12002,12362,-1,'ExchangeStates',null,0,0,0,NULL,NULL,0,'2024-07-21 22:48:27'),('12362--1-ExchangeStates',0,12002,12362,-1,'ExchangeStates',null,0,0,0,NULL,NULL,0,'2024-07-21 22:48:37')Error msg: errCode = 3, detailMessage = tablet 11135 alive replica num 1 < quorumreplica num 2, alive backends: [11002], detail: Visible Replicas:Visible version: 1, Replicas: [replicaId=11136, BackendId=11002, version=1, dataSize=0, rowCount=0, lastFailedVersion=-1, lastSuccessVersion=1, lastFailedTimestamp=0, schemaHash=409617337, state=NORMAL], Backends: Backend [id=11002, isDecommission: false, backendStatus: [lastSuccessReportTabletsTime='2024-07-21 22:48:01', lastStreamLoadTime=-1, isQueryDisabled=false, isLoadDisabled=false], isAlive: true, lastUpdateTime: 2024-07-21 22:48:33
        at org.apache.doris.statistics.AnalysisJob.writeBuf(AnalysisJob.java:135) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.statistics.AnalysisJob.markOneTaskDone(AnalysisJob.java:87) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.statistics.AnalysisJob.appendBuf(AnalysisJob.java:75) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.statistics.OlapAnalysisTask.doExecute(OlapAnalysisTask.java:80) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.statistics.BaseAnalysisTask.executeWithRetry(BaseAnalysisTask.java:214) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.statistics.BaseAnalysisTask.execute(BaseAnalysisTask.java:199) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.statistics.AnalysisTaskWrapper.lambda$new$0(AnalysisTaskWrapper.java:43) ~[doris-fe.jar:1.2-SNAPSHOT]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_352-352]
        at org.apache.doris.statistics.AnalysisTaskWrapper.run(AnalysisTaskWrapper.java:66) ~[doris-fe.jar:1.2-SNAPSHOT]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_352-352]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_352-352]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_352-352]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_352-352]
        at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_352-352]
2024-07-21 22:48:53,303 ERROR (heartbeat mgr|36) [BDBJEJournal.write():194] catch an exception when writing to database. sleep and retry. journal id 5075
com.sleepycat.je.DiskLimitException: (JE 18.3.12) Disk usage is not within je.maxDisk or je.freeDisk limits and write operations are prohibited: maxDiskLimit=0 freeDiskLimit=5,368,709,120 adjustedMaxDiskLimit=0 maxDiskOverage=0 freeDiskShortage=728,760,320 diskFreeSpace=4,639,948,800 availableLogSize=-728,760,320 totalLogSize=5,133,216 activeLogSize=5,133,216 reservedLogSize=0 protectedLogSize=0 protectedLogSizeMap={}
        at com.sleepycat.je.Cursor.checkUpdatesAllowed(Cursor.java:5407) ~[je-18.3.14-doris-SNAPSHOT.jar:18.3.14-doris-SNAPSHOT]
        at com.sleepycat.je.Cursor.checkUpdatesAllowed(Cursor.java:5384) ~[je-18.3.14-doris-SNAPSHOT.jar:18.3.14-doris-SNAPSHOT]
        at com.sleepycat.je.Cursor.putInternal(Cursor.java:2439) ~[je-18.3.14-doris-SNAPSHOT.jar:18.3.14-doris-SNAPSHOT]
        at com.sleepycat.je.Cursor.putInternal(Cursor.java:841) ~[je-18.3.14-doris-SNAPSHOT.jar:18.3.14-doris-SNAPSHOT]
        at com.sleepycat.je.Database.put(Database.java:1635) ~[je-18.3.14-doris-SNAPSHOT.jar:18.3.14-doris-SNAPSHOT]
        at com.sleepycat.je.Database.put(Database.java:1688) ~[je-18.3.14-doris-SNAPSHOT.jar:18.3.14-doris-SNAPSHOT]
        at org.apache.doris.journal.bdbje.BDBJEJournal.write(BDBJEJournal.java:165) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.persist.EditLog.logEdit(EditLog.java:1160) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.persist.EditLog.logHeartbeat(EditLog.java:1536) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.system.HeartbeatMgr.runAfterCatalogReady(HeartbeatMgr.java:152) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.common.util.MasterDaemon.runOneCycle(MasterDaemon.java:58) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.common.util.Daemon.run(Daemon.java:116) ~[doris-fe.jar:1.2-SNAPSHOT]
2024-07-21 22:48:58,311 ERROR (heartbeat mgr|36) [BDBJEJournal.write():194] catch an exception when writing to database. sleep and retry. journal id 5075
com.sleepycat.je.DiskLimitException: (JE 18.3.12) Disk usage is not within je.maxDisk or je.freeDisk limits and write operations are prohibited: maxDiskLimit=0 freeDiskLimit=5,368,709,120 adjustedMaxDiskLimit=0 maxDiskOverage=0 freeDiskShortage=730,857,472 diskFreeSpace=4,637,851,648 availableLogSize=-730,857,472 totalLogSize=5,133,253 activeLogSize=5,133,253 reservedLogSize=0 protectedLogSize=0 protectedLogSizeMap={}
        at com.sleepycat.je.Cursor.checkUpdatesAllowed(Cursor.java:5407) ~[je-18.3.14-doris-SNAPSHOT.jar:18.3.14-doris-SNAPSHOT]
        at com.sleepycat.je.Cursor.checkUpdatesAllowed(Cursor.java:5384) ~[je-18.3.14-doris-SNAPSHOT.jar:18.3.14-doris-SNAPSHOT]
        at com.sleepycat.je.Cursor.putInternal(Cursor.java:2439) ~[je-18.3.14-doris-SNAPSHOT.jar:18.3.14-doris-SNAPSHOT]
        at com.sleepycat.je.Cursor.putInternal(Cursor.java:841) ~[je-18.3.14-doris-SNAPSHOT.jar:18.3.14-doris-SNAPSHOT]
        at com.sleepycat.je.Database.put(Database.java:1635) ~[je-18.3.14-doris-SNAPSHOT.jar:18.3.14-doris-SNAPSHOT]
        at com.sleepycat.je.Database.put(Database.java:1688) ~[je-18.3.14-doris-SNAPSHOT.jar:18.3.14-doris-SNAPSHOT]
        at org.apache.doris.journal.bdbje.BDBJEJournal.write(BDBJEJournal.java:165) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.persist.EditLog.logEdit(EditLog.java:1160) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.persist.EditLog.logHeartbeat(EditLog.java:1536) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.system.HeartbeatMgr.runAfterCatalogReady(HeartbeatMgr.java:152) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.common.util.MasterDaemon.runOneCycle(MasterDaemon.java:58) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.common.util.Daemon.run(Daemon.java:116) ~[doris-fe.jar:1.2-SNAPSHOT]
2024-07-21 22:49:03,316 ERROR (heartbeat mgr|36) [BDBJEJournal.write():194] catch an exception when writing to database. sleep and retry. journal id 5075
com.sleepycat.je.DiskLimitException: (JE 18.3.12) Disk usage is not within je.maxDisk or je.freeDisk limits and write operations are prohibited: maxDiskLimit=0 freeDiskLimit=5,368,709,120 adjustedMaxDiskLimit=0 maxDiskOverage=0 freeDiskShortage=730,857,472 diskFreeSpace=4,637,851,648 availableLogSize=-730,857,472 totalLogSize=5,133,253 activeLogSize=5,133,253 reservedLogSize=0 protectedLogSize=0 protectedLogSizeMap={}
        at com.sleepycat.je.Cursor.checkUpdatesAllowed(Cursor.java:5407) ~[je-18.3.14-doris-SNAPSHOT.jar:18.3.14-doris-SNAPSHOT]
        at com.sleepycat.je.Cursor.checkUpdatesAllowed(Cursor.java:5384) ~[je-18.3.14-doris-SNAPSHOT.jar:18.3.14-doris-SNAPSHOT]
        at com.sleepycat.je.Cursor.putInternal(Cursor.java:2439) ~[je-18.3.14-doris-SNAPSHOT.jar:18.3.14-doris-SNAPSHOT]
        at com.sleepycat.je.Cursor.putInternal(Cursor.java:841) ~[je-18.3.14-doris-SNAPSHOT.jar:18.3.14-doris-SNAPSHOT]
        at com.sleepycat.je.Database.put(Database.java:1635) ~[je-18.3.14-doris-SNAPSHOT.jar:18.3.14-doris-SNAPSHOT]
        at com.sleepycat.je.Database.put(Database.java:1688) ~[je-18.3.14-doris-SNAPSHOT.jar:18.3.14-doris-SNAPSHOT]
        at org.apache.doris.journal.bdbje.BDBJEJournal.write(BDBJEJournal.java:165) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.persist.EditLog.logEdit(EditLog.java:1160) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.persist.EditLog.logHeartbeat(EditLog.java:1536) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.system.HeartbeatMgr.runAfterCatalogReady(HeartbeatMgr.java:152) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.common.util.MasterDaemon.runOneCycle(MasterDaemon.java:58) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.common.util.Daemon.run(Daemon.java:116) ~[doris-fe.jar:1.2-SNAPSHOT]
2024-07-21 22:49:08,401 ERROR (heartbeat mgr|36) [BDBJEJournal.write():206] write bdb failed. will exit. journalId: 5075, bdb database Name: 100
2024-07-21 22:49:15,826 WARN (UNKNOWN fe_082c47cf_4c08_482b_8e43_d87638ceea23(-1)|1) [Env.notifyNewFETypeTransfer():2460] notify new FE type transfer: UNKNOWN
2024-07-21 22:54:14,408 WARN (UNKNOWN fe_082c47cf_4c08_482b_8e43_d87638ceea23(-1)|1) [Env.notifyNewFETypeTransfer():2460] notify new FE type transfer: UNKNOWN
2024-07-21 22:59:14,697 WARN (UNKNOWN fe_082c47cf_4c08_482b_8e43_d87638ceea23(-1)|1) [Env.notifyNewFETypeTransfer():2460] notify new FE type transfer: UNKNOWN
2024-07-21 23:04:13,956 WARN (UNKNOWN fe_082c47cf_4c08_482b_8e43_d87638ceea23(-1)|1) [Env.notifyNewFETypeTransfer():2460] notify new FE type transfer: UNKNOWN
2024-07-21 23:09:14,097 WARN (UNKNOWN fe_082c47cf_4c08_482b_8e43_d87638ceea23(-1)|1) [Env.notifyNewFETypeTransfer():2460] notify new FE type transfer: UNKNOWN
2024-07-21 23:14:14,172 WARN (UNKNOWN fe_082c47cf_4c08_482b_8e43_d87638ceea23(-1)|1) [Env.notifyNewFETypeTransfer():2460] notify new FE type transfer: UNKNOWN
2024-07-21 23:19:14,150 WARN (UNKNOWN fe_082c47cf_4c08_482b_8e43_d87638ceea23(-1)|1) [Env.notifyNewFETypeTransfer():2460] notify new FE type transfer: UNKNOWN
2024-07-21 23:24:14,320 WARN (UNKNOWN fe_082c47cf_4c08_482b_8e43_d87638ceea23(-1)|1) [Env.notifyNewFETypeTransfer():2460] notify new FE type transfer: UNKNOWN
2024-07-21 23:29:14,550 WARN (UNKNOWN fe_082c47cf_4c08_482b_8e43_d87638ceea23(-1)|1) [Env.notifyNewFETypeTransfer():2460] notify new FE type transfer: UNKNOWN
2024-07-21 23:39:22,505 WARN (UNKNOWN fe_082c47cf_4c08_482b_8e43_d87638ceea23(-1)|1) [Env.notifyNewFETypeTransfer():2460] notify new FE type transfer: UNKNOWN
2024-07-21 23:44:19,324 WARN (UNKNOWN fe_082c47cf_4c08_482b_8e43_d87638ceea23(-1)|1) [Env.notifyNewFETypeTransfer():2460] notify new FE type transfer: UNKNOWN
2024-07-21 23:49:19,556 WARN (UNKNOWN fe_082c47cf_4c08_482b_8e43_d87638ceea23(-1)|1) [Env.notifyNewFETypeTransfer():2460] notify new FE type transfer: UNKNOWN
2024-07-21 23:54:19,401 WARN (UNKNOWN fe_082c47cf_4c08_482b_8e43_d87638ceea23(-1)|1) [Env.notifyNewFETypeTransfer():2460] notify new FE type transfer: UNKNOWN
2024-07-21 23:59:18,844 WARN (UNKNOWN fe_082c47cf_4c08_482b_8e43_d87638ceea23(-1)|1) [Env.notifyNewFETypeTransfer():2460] notify new FE type transfer: UNKNOWN

使用doris 的be节点出现重启问题

3 Answers