A Java service dies in the night with java.lang.OutOfMemoryError: Java heap space. Kubernetes restarts it, the graphs recover, and everyone moves on — until it happens again the next night. The only tool that actually answers why is a heap dump, and heap dump analysis has a reputation for being arcane that it does not deserve. The mechanics are simple once you have done it once.
Capture the dump at the moment of death
The single most important thing is to have the dump before you need it. A heap dump taken manually after a restart shows a healthy process; you need the snapshot from the instant the heap was full. The JVM will write one for you automatically if you ask it to at launch.
java -XX:+HeapDumpOnOutOfMemoryError \
-XX:HeapDumpPath=/var/log/dumps \
-jar app.jarSet this flag on every production JVM, today, before you have a problem. When the OOM fires, the JVM writes a .hprof file just before it dies, capturing exactly the state that killed it. Point HeapDumpPath at a volume with room — a dump is roughly the size of your max heap, so a 4GB heap writes a 4GB file, and a full disk means no dump.
In a container there is a wrinkle worth knowing: the dump is written inside the container’s filesystem, which vanishes when the pod restarts. Write it to a mounted volume, or you will get the OOM and lose the evidence in the same breath. If you cannot arrange that, you can trigger a manual dump from a running-but-struggling process with jcmd <pid> GC.heap_dump /path/dump.hprof, though catching the right moment by hand is a losing game.
Open it in a tool built for the job
A .hprof file is not something you read by hand. The Eclipse Memory Analyzer (MAT) is the standard tool and it is free; it parses the dump, builds an object graph, and — crucially — indexes it so you can ask questions without loading gigabytes into your own machine’s memory.
MAT’s headline feature is the Leak Suspects report, which it offers to generate the moment you open a dump. Let it. For a genuine leak it is right an unreasonable amount of the time, and it hands you the answer as a sentence: “one instance of SessionCache occupies 3.2 GB.” That is often the whole investigation.
Retained versus shallow: the one concept that matters
To read the report you need exactly one idea, and it is the one people skip. Shallow size is the memory an object occupies by itself. Retained size is the memory that would be freed if that object were garbage collected — the object plus everything only it keeps alive.
This distinction is the whole game. A HashMap object has a tiny shallow size — it is just a few fields and an array reference. But its retained size might be two gigabytes, because it holds the only references to two gigabytes of entries. Sort by shallow size and the map is invisible; sort by retained size and it is the obvious culprit. Always look at retained size when hunting a leak.
MAT’s dominator tree is the view that ranks objects by retained size. The object at the top is holding the most memory alive, and its children are what it is holding. Walking down from the top almost always leads to the leak within a few clicks — a cache with no eviction, a static list that only ever grows, a ThreadLocal that was never cleared.
The usual suspects
Most leaks in practice are one of a small handful of patterns, and knowing them speeds up the read enormously.
An unbounded cache is the classic: a Map used as a cache with no maximum size and no expiry, so it accumulates forever. The fix is a real cache with eviction — Caffeine, or a size-bounded LinkedHashMap — not a plain HashMap that someone called a cache.
A growing static collection is a list or map held in a static field that only ever has things added to it. Static means it lives for the life of the JVM, so anything it references can never be collected. The dominator tree points straight at it.
A ThreadLocal that outlives its request leaks on pooled threads, because the thread is reused and the value is never cleared — exactly the failure mode that also causes cross-request bugs in a multi-tenant application. If you see many instances of your value class each pinned by a thread, that is the shape.
Not every OOM is a leak
Worth saying plainly, because it saves wasted hours: sometimes the heap is full for a perfectly legitimate reason. A batch job that loads a million rows into memory at once is not leaking — it genuinely needs that memory, and the fix is to stream or paginate, not to hunt for a phantom leak. The dominator tree tells you which it is: a leak shows one object growing without bound across dumps, while a legitimate high-water mark shows the memory spread across the objects the work actually needs.
The tell is time. Take two dumps an hour apart under load. If the same object’s retained size has grown, it is a leak. If the memory is high but stable, you simply need a bigger heap or a smaller working set. Raising -Xmx on a real leak only buys you a longer interval between crashes; raising it on a legitimate high-water mark is the correct fix.
When the heap dump is empty but the process still died
A confusing case worth preparing for: the OOM fires, but the heap dump shows a heap that is nowhere near full. This means the memory that ran out was not the Java heap at all, and chasing it in the heap dump will waste hours. Java memory is not one pool; it is several, and only one of them is what a heap dump captures.
The error message actually tells you which pool, if you read past the first line. OutOfMemoryError: Metaspace means class metadata filled up — usually a classloader leak, where an application redeploys repeatedly and old class definitions are never released, common in long-lived application servers. OutOfMemoryError: Direct buffer memory points at off-heap byte buffers, which NIO and many networking libraries allocate outside the heap, invisible to a heap dump entirely. unable to create new native thread means you exhausted the OS’s thread limit, each thread costing native stack memory the heap knows nothing about.
Each of these needs a different tool. Native Memory Tracking, enabled with -XX:NativeMemoryTracking=summary, breaks down where the JVM’s non-heap memory has gone and is the right instrument when the heap looks innocent. A steadily climbing thread count points at a thread leak — an executor created per request and never shut down. And the container dimension returns here: the kernel kills the whole process for exceeding the container’s memory limit, counting all of these pools plus the heap, so a JVM sized to fit its heap inside the limit can still be killed by off-heap growth the heap dump never shows. Recognising which pool ran out, from the error line alone, is what stops you from analysing the wrong thing for an afternoon.
Prevention beats forensics
Heap dumps are the autopsy. The living-patient version is watching memory trends before they become crashes. A rolling Java Flight Recorder recording captures allocation profiles continuously at about 1% overhead, so when memory starts climbing you can see what is allocating before the OOM ever fires. And in a container, remember that the JVM and the kernel disagree about memory: the JVM can size its heap against the container limit and still get killed with exit code 137 by the kernel — a scenario worth recognising from the pod troubleshooting angle as much as the JVM one.
Do the boring thing first, though: turn on HeapDumpOnOutOfMemoryError everywhere and point it at durable storage. The next 3am OOM will leave a file that answers the question in twenty minutes instead of recurring for a month.