Sources & further reading, since this walks over well-trodden ground: shayne007’s OOM troubleshooting guide and HeapHero on Java memory leaks. The numbers below are my own, reproduced locally.
If you’ve ever watched a service stay green on every health check, answer every request, and then fall over at 4am with java.lang.OutOfMemoryError in the last line of the log, you already know the frustrating part: the memory graph looked fine right up until it didn’t. Used heap climbs, drops, climbs, drops, the sawtooth every JVM draws. A leak draws that same sawtooth. So does a perfectly healthy service under load. The graph you’re staring at cannot tell you which one you have, and that’s the problem.
I wanted to see the difference with my own eyes, so I built five small programs, four that run a JVM out of memory a different way and one that deliberately survives, capped everything small enough that they die in under a minute, and read the logs on the way down. Two of them aren’t heap problems at all. OutOfMemoryError is one error class wearing at least four different failures, and the first job when your pager goes off is figuring out which one you’re actually holding.
The problem
A leak doesn’t crash. It crawls. Garbage collection runs, reclaims a little less than last time, and hands the program back a slightly higher floor to build on. Do that for long enough and the floor reaches the ceiling. By the time you get the OutOfMemoryError the JVM has been dying for minutes, sometimes hours, spending more and more of its life in GC for less and less room. And “used memory climbing” is not the signal, because a busy healthy service does exactly that too. The signal is what the collector manages to take back.
That number has a name: the live set, the bytes still reachable after a collection finishes. Peak usage barely moves between a healthy service and a leaking one. The post-GC live set is the number that gives it away.
The tell is what the collector gives back
Here’s the leak, in the one line that matters:
static final List<byte[]> ROOTS = new ArrayList<>();
static void leak() {
int block = 32 * 1024;
while (true) {
byte[] b = new byte[block];
b[0] = 1; b[block - 1] = 2; // touch it so JIT can't elide the allocation
ROOTS.add(b); // <-- the leak: nothing ever removes it
}
}
Then a second program, healthy, that allocates 32 KB blocks at the exact same rate under the exact same -Xmx256m, and simply doesn’t keep them. Same allocation pressure, same heap, one difference: retention.
I plotted the post-GC heap (the live set after each collection) against wall-clock time for both. This is the chart the memory graph won’t draw for you.
Post-GC live set: leak vs. healthy, same allocation rate, same 256 MB heap
-Xmx256m. The healthy run's live set never leaves 10 MB across the whole run, four young collections, no growth. The leak's live set climbs one Full GC at a time, 11 MB → 254 MB, until it pins against the ceiling at 33 seconds and throws. Retention is the only difference between the two lines. Measured on Temurin 21.0.11, results in benchmarks/java-oom-anatomy/results/.
Look at what the leak’s collector is doing. Early on it runs a Full GC, 13M->11M, and reclaims 2 MB. A third of the way in it runs 109M->109M and reclaims nothing at all, because nothing is garbage, it’s all reachable from ROOTS. Near the end: 254M->247M, 7 MB back out of a full heap, and it spent 12.7 ms of stop-the-world time to get it. The collector is working the whole time and the live set never comes down, and that, not the height of the sawtooth, is a leak. A healthy service’s post-GC floor is flat. A leaking one’s floor climbs like a staircase, one collection at a time.
When it finally gives up, the stack is honest about what it was doing:
java.lang.OutOfMemoryError: Java heap space
at Main.leak(Main.java:147)
at Main.main(Main.java:105)
Line 147 is ROOTS.add(b). A heap dump would say the same thing louder. I sampled a live class histogram with jcmd <pid> GC.class_histogram a moment before death:
What was on the heap when it died (top classes by retained bytes)
[B is the JVM's shorthand for byte[], and it's 252,419,016 bytes across 17,114 instances, 97% of the top-15 bytes on the heap. That's the retained blocks, the leak named in one command. In a real leak this row is your custom class, or a char[], or an Object[] backing some cache that only ever grows. You read that top row first, everything under it is noise. Measured on Temurin 21.0.11, results in benchmarks/java-oom-anatomy/results/.
The tripwire before the wall: GC overhead limit exceeded
Before a leaking JVM throws Java heap space, it often throws something that sounds scarier and means almost the same thing. Run the near-full-heap scenario under the Parallel collector and you get this instead:
java.lang.OutOfMemoryError: GC overhead limit exceeded
at Main.gcOverhead(Main.java:172)
at Main.main(Main.java:60)
This is the JVM’s own tripwire firing. HotSpot watches a rolling window, and if it spends more than about 98% of recent wall-clock time in GC while recovering less than 2% of the heap, it stops pretending it’s making progress and throws. It’s a mercy killing, the JVM refusing to spend the next hour thrashing at 1% throughput before dying anyway. I logged the collector’s behavior in the final second, split into windows:
GC overhead limit: nearly all the time in GC, almost nothing back
-XX:+UseParallelGC, results in benchmarks/java-oom-anatomy/results/.
Worth knowing: which of the two you get is a matter of timing, not a matter of two different bugs. Fill the heap a little faster and it runs out cleanly and says Java heap space; let it thrash right at the edge and the overhead heuristic trips first and it says GC overhead limit exceeded. Reproducing the overhead-limit message reliably took filling to 80% rather than 90%, and even then the harness retries a couple of times, that shape is genuinely lumpy. Don’t treat the two messages as different diagnoses. They’re the same leak or the same undersized heap, caught at slightly different moments.
The OutOfMemoryError that isn’t about the heap
Now the one that catches people, because every reflex you built above is wrong for it. You get OutOfMemoryError, you pull the heap dump, and the heap is nearly empty. Bumping -Xmx does nothing. Because it was never the heap:
java.lang.OutOfMemoryError: Metaspace
at java.base/java.lang.ClassLoader.defineClass0(Native Method)
...
at Main.main(Main.java:126)
Metaspace is where the JVM keeps class metadata, the runtime shape of every class it has loaded. It lives in native memory, outside the heap, and it has its own ceiling (-XX:MaxMetaspaceSize). You fill it not by allocating objects but by loading classes, and the classic way to leak it is to keep making new ones: a fresh classloader per request, dynamic proxies, a scripting engine, a redeploy that never lets the old classloader die. I reproduced it by spinning up thousands of throwaway classloaders, each defining one more class, and watched the heap sit still while the metadata wall came up to meet it:
Metaspace death: the heap had 91% free and the JVM still died
-XX:MaxMetaspaceSize=64m, results in benchmarks/java-oom-anatomy/results/.
The one a heap dump can’t see
Metaspace at least shows up in the JVM’s own memory pools, so a monitoring dashboard has a chance of catching it. The next one doesn’t show up anywhere you’d normally look. Direct byte buffers, the ones NIO uses under every socket and file channel and most serious network libraries, keep a small wrapper object on the heap and the actual bytes in native memory the OS handed out. Those bytes only come back when the wrapper becomes unreachable and its Cleaner runs. Hold on to the wrapper and the native memory is pinned for good:
static final List<ByteBuffer> BUFFERS = new ArrayList<>();
...
ByteBuffer b = ByteBuffer.allocateDirect(512 * 1024);
BUFFERS.add(b); // <-- retain the wrapper: the Cleaner never runs
128 buffers of 512 KB each under -XX:MaxDirectMemorySize=64m, dead in 3.1 seconds. The message is worth reading closely, because it is not the short Direct buffer memory string that most older write-ups quote. Modern JDKs hand you the arithmetic instead:
java.lang.OutOfMemoryError: Cannot reserve 524288 bytes of direct buffer memory (allocated: 66584626, limit: 67108864)
at java.base/java.nio.Bits.reserveMemory(Bits.java:178)
at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:111)
at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:360)
at Main.directBuffer(Main.java:77)
Direct buffer memory: the heap was 95% empty when it died
DirectByteBuffer objects adding up to a few kilobytes. The 63 MB that actually killed it is not in the dump, because it was never on the heap. Measured on Temurin 21.0.11, results in benchmarks/java-oom-anatomy/results/.
Two more you’ll eventually meet: unable to create new native thread, when the OS won’t hand you another thread, and Requested array size exceeds VM limit, when something computed a nonsense array length. None of these are fixed by tuning the heap, because none of them are the heap. Read the first word after the colon before you reach for a heap flag.
Reading the evidence
Put them together and the first question on the pager is never “how do I get more memory,” it’s “which OutOfMemoryError is this”:
| Message | Where it ran out | What actually fixes it |
|---|---|---|
Java heap space |
The heap | Find the growing live set, usually a collection that only ever adds. A heap dump names it. |
GC overhead limit exceeded |
The heap (caught earlier) | Same as above. It’s a leak or an undersized heap, seen mid-thrash, not a separate bug. |
Metaspace |
Class metadata, off-heap | Find the classloader that won’t die. A bigger -Xmx is wasted. |
Cannot reserve N bytes of direct buffer memory |
Native memory, off-heap | Find what’s retaining the ByteBuffer wrappers, or raise -XX:MaxDirectMemorySize. A heap dump won’t show you the bytes. |
unable to create new native thread / Requested array size exceeds VM limit |
OS limits, or a bad computed size | A thread leak, or a length someone calculated wrong. Neither is a heap problem. |
And the tell that separates a real leak from a service that’s just busy is the same in every heap case: watch the post-GC live set, the floor after each collection, not the peak. A flat floor under load is a healthy JVM doing its job. A floor that climbs collection after collection while the collector reclaims less each time is a leak, and it will reach the ceiling on its own schedule whether or not you’re watching.
The takeaway
Turn on the GC log before you need it. -Xlog:gc*:file=gc.log:time,level,tags costs almost nothing and it’s the difference between reading the death spiral and guessing at it after the fact. Add -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=... so the JVM hands you the evidence on its way out, then open the dump in something that shows a dominator tree and read the top row. And when the graph looks fine but the service keeps dying, stop looking at peak memory and look at what garbage collection gives back, because a leak and a healthy load look identical until you do.
The harness that produced all of this, four JVMs run into the ground and one held up next to them as a control, with the GC logs and histograms and stack traces captured, is on GitHub. These are laptop numbers with tiny heaps chosen to fail fast, not capacity or tuning advice, the shape is what carries over, not the seconds on the clock.
Comments