flink中使用 flink-doris-connector 读取doris数据,多次运行后报异常:java.lang.OutOfMemoryError: Metaspace.

Viewed 1

在flink中使用flink-doris-connector-1.20(版本:25.1.0)读取doris数据,多次提交任务运行后,flink的Taskmanager会报异常:java.lang.OutOfMemoryError: Metaspace。

doris版本:2.1.5。

下面是flink日志中打印的GC信息:
[1472.632s][info ][gc ] GC(109) Pause Full (Metadata GC Clear Soft References) 140M->140M(545M) 245.268ms
[1472.632s][info ][gc,cpu ] GC(109) User=0.24s Sys=0.00s Real=0.24s
[1472.637s][info ][gc,start ] GC(110) Pause Full (Metadata GC Threshold)
[1472.639s][info ][gc,phases,start] GC(110) Phase 1: Mark live objects
[1472.796s][info ][gc,phases ] GC(110) Phase 1: Mark live objects 157.329ms
[1472.796s][info ][gc,phases,start] GC(110) Phase 2: Compute new object addresses
[1472.818s][info ][gc,phases ] GC(110) Phase 2: Compute new object addresses 21.391ms
[1472.818s][info ][gc,phases,start] GC(110) Phase 3: Adjust pointers
[1472.884s][info ][gc,phases ] GC(110) Phase 3: Adjust pointers 66.876ms
[1472.885s][info ][gc,phases,start] GC(110) Phase 4: Move objects
[1472.885s][info ][gc,phases ] GC(110) Phase 4: Move objects 0.032ms
[1472.885s][info ][gc,heap ] GC(110) DefNew: 784K->0K(173312K)
[1472.885s][info ][gc,heap ] GC(110) Tenured: 144255K->144304K(385024K)
[1472.885s][info ][gc,metaspace ] GC(110) Metaspace: 252676K(262144K)->252676K(262144K) NonClass: 222235K(228864K)->222235K(228864K) Class: 30440K(33280K)->30440K(33280K)
[1472.885s][info ][gc ] GC(110) Pause Full (Metadata GC Threshold) 141M->140M(545M) 248.049ms
[1472.885s][info ][gc,cpu ] GC(110) User=0.24s Sys=0.00s Real=0.24s
[1472.886s][info ][gc,start ] GC(111) Pause Full (Metadata GC Clear Soft References)
[1472.887s][info ][gc,phases,start] GC(111) Phase 1: Mark live objects
[1472.983s][info ][gc,phases ] GC(111) Phase 1: Mark live objects 95.677ms
[1472.983s][info ][gc,phases,start] GC(111) Phase 2: Compute new object addresses
[1473.003s][info ][gc,phases ] GC(111) Phase 2: Compute new object addresses 20.037ms
[1473.003s][info ][gc,phases,start] GC(111) Phase 3: Adjust pointers
[1473.067s][info ][gc,phases ] GC(111) Phase 3: Adjust pointers 64.120ms
[1473.067s][info ][gc,phases,start] GC(111) Phase 4: Move objects
[1473.080s][info ][gc,phases ] GC(111) Phase 4: Move objects 12.668ms
[1473.081s][info ][gc,heap ] GC(111) DefNew: 0K->0K(173312K)
[1473.081s][info ][gc,heap ] GC(111) Tenured: 144304K->127554K(385024K)
[1473.081s][info ][gc,metaspace ] GC(111) Metaspace: 252676K(262144K)->252676K(262144K) NonClass: 222235K(228864K)->222235K(228864K) Class: 30440K(33280K)->30440K(33280K)
[1473.081s][info ][gc ] GC(111) Pause Full (Metadata GC Clear Soft References) 140M->124M(545M) 195.097ms
[1473.081s][info ][gc,cpu ] GC(111) User=0.19s Sys=0.00s Real=0.20s
[1473.081s][info ][gc,metaspace,freelist,oom] Metaspace (data) allocation failed for size 11
[1473.095s][info ][gc,metaspace,freelist,oom]
[1473.095s][info ][gc,metaspace,freelist,oom] Usage:
[1473.095s][info ][gc,metaspace,freelist,oom] Non-class: 219.65 MB capacity, 217.03 MB ( 99%) used, 2.01 MB ( <1%) free+waste, 628.50 KB ( <1%) overhead.
[1473.095s][info ][gc,metaspace,freelist,oom] Class: 31.51 MB capacity, 29.73 MB ( 94%) used, 1.51 MB ( 5%) free+waste, 278.12 KB ( <1%) overhead.
[1473.095s][info ][gc,metaspace,freelist,oom] Both: 251.16 MB capacity, 246.75 MB ( 98%) used, 3.52 MB ( 1%) free+waste, 906.62 KB ( <1%) overhead.
[1473.095s][info ][gc,metaspace,freelist,oom]
[1473.095s][info ][gc,metaspace,freelist,oom] Virtual space:
[1473.095s][info ][gc,metaspace,freelist,oom] Non-class space: 224.00 MB reserved, 223.50 MB (>99%) committed
[1473.095s][info ][gc,metaspace,freelist,oom] Class space: 248.00 MB reserved, 32.50 MB ( 13%) committed
[1473.095s][info ][gc,metaspace,freelist,oom] Both: 472.00 MB reserved, 256.00 MB ( 54%) committed
[1473.095s][info ][gc,metaspace,freelist,oom]
[1473.095s][info ][gc,metaspace,freelist,oom] Chunk freelists:
[1473.095s][info ][gc,metaspace,freelist,oom] Non-Class: 3.83 MB
[1473.095s][info ][gc,metaspace,freelist,oom] Class: 1010.00 KB
[1473.095s][info ][gc,metaspace,freelist,oom] Both: 4.82 MB
[1473.095s][info ][gc,metaspace,freelist,oom]
[1473.095s][info ][gc,metaspace,freelist,oom] MaxMetaspaceSize: 256.00 MB
[1473.095s][info ][gc,metaspace,freelist,oom] CompressedClassSpaceSize: 248.00 MB
[1473.095s][info ][gc,metaspace,freelist,oom] Initial GC threshold: 196.00 MB
[1473.095s][info ][gc,metaspace,freelist,oom] Current GC threshold: 256.00 MB
[1473.095s][info ][gc,metaspace,freelist,oom] CDS: on
[1473.095s][info ][gc,metaspace,freelist,oom]

下面是flink Taskmanager异常的堆栈信息:
java.lang.OutOfMemoryError: Metaspace
Dumping heap to /flink-data/pretreat/heapdump1.hprof ...
Heap dump file created [186670031 bytes in 1.039 secs]
2025-11-26 10:13:58,195 ERROR org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - Fatal error occurred while executing the TaskManager. Shutting it down...
java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory error has occurred. This can mean two things: either the job requires a larger size of JVM metaspace to load classes or there is a class loading leak. In the first case 'taskmanager.memory.jvm-metaspace.size' configuration option should be increased. If the error persists (usually in cluster after several job (re-)submissions) then there is probably a class loading leak in user code or some of its dependencies which has to be investigated and fixed. The task executor has to be shutdown...
at java.base/java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
at java.base/java.lang.ClassLoader.defineClass(Unknown Source) ~[?:?]
at java.base/java.security.SecureClassLoader.defineClass(Unknown Source) ~[?:?]
at java.base/java.net.URLClassLoader.defineClass(Unknown Source) ~[?:?]
at java.base/java.net.URLClassLoader$1.run(Unknown Source) ~[?:?]
at java.base/java.net.URLClassLoader$1.run(Unknown Source) ~[?:?]
at java.base/java.security.AccessController.doPrivileged(Native Method) ~[?:?]
at java.base/java.net.URLClassLoader.findClass(Unknown Source) ~[?:?]
at org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:71) ~[flink-dist-1.20.3.jar:1.20.3]
at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:51) ~[flink-dist-1.20.3.jar:1.20.3]
at java.base/java.lang.ClassLoader.loadClass(Unknown Source) ~[?:?]
at org.apache.doris.shaded.com.fasterxml.jackson.databind.ObjectMapper.(ObjectMapper.java:677) ~[blob_p-bd49685154fa2e10da30f18982e2c807c4371c7d-976f675a781496abe86bf628c16fdd34:?]
at org.apache.doris.shaded.com.fasterxml.jackson.databind.ObjectMapper.(ObjectMapper.java:576) ~[blob_p-bd49685154fa2e10da30f18982e2c807c4371c7d-976f675a781496abe86bf628c16fdd34:?]
at org.apache.doris.flink.deserialization.converter.DorisRowConverter.(DorisRowConverter.java:66) ~[blob_p-bd49685154fa2e10da30f18982e2c807c4371c7d-976f675a781496abe86bf628c16fdd34:?]
at java.base/jdk.internal.misc.Unsafe.ensureClassInitialized0(Native Method) ~[?:?]
at java.base/jdk.internal.misc.Unsafe.ensureClassInitialized(Unknown Source) ~[?:?]
at java.base/jdk.internal.reflect.UnsafeFieldAccessorFactory.newFieldAccessor(Unknown Source) ~[?:?]
at java.base/jdk.internal.reflect.ReflectionFactory.newFieldAccessor(Unknown Source) ~[?:?]
at java.base/java.lang.reflect.Field.acquireFieldAccessor(Unknown Source) ~[?:?]
at java.base/java.lang.reflect.Field.getFieldAccessor(Unknown Source) ~[?:?]
at java.base/java.lang.reflect.Field.getLong(Unknown Source) ~[?:?]
at java.base/java.io.ObjectStreamClass.getDeclaredSUID(Unknown Source) ~[?:?]
at java.base/java.io.ObjectStreamClass$2.run(Unknown Source) ~[?:?]
at java.base/java.io.ObjectStreamClass$2.run(Unknown Source) ~[?:?]
at java.base/java.security.AccessController.doPrivileged(Native Method) ~[?:?]
at java.base/java.io.ObjectStreamClass.(Unknown Source) ~[?:?]
at java.base/java.io.ObjectStreamClass$Caches$1.computeValue(Unknown Source) ~[?:?]
at java.base/java.io.ObjectStreamClass$Caches$1.computeValue(Unknown Source) ~[?:?]
at java.base/java.io.ClassCache$1.computeValue(Unknown Source) ~[?:?]
at java.base/java.io.ClassCache$1.computeValue(Unknown Source) ~[?:?]
at java.base/java.lang.ClassValue.getFromHashMap(Unknown Source) ~[?:?]
at java.base/java.lang.ClassValue.getFromBackup(Unknown Source) ~[?:?]

0 Answers