This one is theory, not my own benchmark. It’s my reading of two posts: Netflix’s Introducing jvmquake and Uber’s JVM Tuning for Large Scale Services. Every number below is theirs, not mine. I’m crediting them up front and drawing my own diagrams so I’m not lifting anyone’s images.
There’s a specific kind of outage that ruins your afternoon, and it’s not the crash. The crash is easy. The process dies, the supervisor restarts it, the load balancer pulls it out, and you go read the log. The bad one is the node that stays up. It answers the health check. Its port is open. Its CPU is pinned at 100%. And it is doing approximately nothing, and it will keep not doing anything until somebody notices and kills it by hand.
The problem
A JVM in a garbage collection death spiral is technically alive. The heap is nearly full, every collection reclaims almost nothing, so the collector runs again immediately, and the application threads get a sliver of time between pauses. Netflix’s description of what this does to a datastore node is the line I keep coming back to: throughput “has, typically, decreased by four orders of magnitude.” Not degraded. Four orders of magnitude. Their real example was a JVM taking repeated 20-second-plus GC pauses with almost no work between them. Meanwhile nothing in the JVM decides this is a failure, because from the JVM’s point of view nothing has failed. It hasn’t run out of memory. It found memory. It just spent twenty seconds finding it, and it’s about to do it again.
Why the guards you already have don’t fire
The instinct is that the JVM must already handle this, and it half does. There’s a family of flags for it: GCHeapFreeLimit, GCTimeLimit, OnOutOfMemoryError, ExitOnOutOfMemoryError, CrashOnOutOfMemoryError. Netflix evaluated the lot and their conclusion is worth quoting flat, because it saved me from a week of tuning them myself: they “either do not work consistently on all JVMs and garbage collectors, are hard to tune or understand, or they simply don’t work in various edge cases.”
The deeper reason is that all of them are keyed to running out of memory, and a death spiral is not running out of memory. It’s a grey failure, the space between healthy and dead where the process is technically doing its job and practically useless. Every guard in that list watches the wrong end of the curve.
The gap those flags leave open
GC time as a debt you can go bankrupt on
jvmquake’s idea is the part worth stealing even if you never run their tool. Stop asking “is the heap full.” Ask “is this process paying its GC bill,” and track that as a running balance.
It works like a leaky bucket. Hook the JVMTI callbacks GarbageCollectionStart and GarbageCollectionFinish so you can see every pause, then:
- every millisecond spent in GC adds a millisecond of debt,
- every millisecond the application actually runs pays down debt,
- debt floors at zero, it never goes negative.
That’s it. And the reason it works is a piece of arithmetic that clicks the moment you see it. A JVM that spends less than half its time in GC pays down faster than it borrows, so the balance trends to zero and stays there forever, no matter how long the process runs. A JVM that spends more than half its time in GC borrows faster than it pays, so the balance trends to infinity. There’s no tuning to get right, no threshold to guess per service. The 50% line falls out of the algorithm on its own, and a healthy process sits on the safe side of it by a mile.
Then you put a cap on the balance. Default is 30 seconds of accumulated debt, checked after each collection finishes. Cross it and the process gets killed, on the grounds that anything that has fallen 30 seconds behind is not coming back.
Two JVMs, same counter
There’s a knob for the ratio, runtime_weight. Set it to 2 and application runtime pays down debt twice as fast, which moves the break-even line from 50% throughput to 33%. That’s the whole tuning surface: one number saying how bad you’ll let it get before you call it dead.
Killing it is the easy half
Once you’ve decided the process is gone, you have a much more interesting problem: this is the only moment you will ever have a live, in-memory picture of whatever caused the spiral, and if you just SIGKILL it you have thrown that away and you will see the bug again next week.
jvmquake handles that in a way I found genuinely clever. Two options, depending on what you want out of the corpse.
The first is a heap dump, and rather than plumbing a new dump path it goes through the front door the JVM already has. It deliberately allocates giant arrays until the JVM throws a real OutOfMemoryError, which trips the -XX:+HeapDumpOnOutOfMemoryError flag you already set, and the JVM writes the heap dump itself. Induce the failure the existing machinery already knows how to document. No new code path to keep working.
The second is a full core dump, which is strictly better for diagnosis and strictly worse for your disk. Send SIGABRT, the kernel writes the core. Netflix’s problem was that these cores are enormous and the box doesn’t have room, so they don’t land them on disk at all: a script compresses and pipes the core straight to S3, and systemd restarts the process while the upload is still going. They report they “reliably upload 16GB core dumps in less than two minutes.” The node is back in rotation before the evidence has finished uploading.
That’s the part I’d argue matters more than the detector. If the kill leaves you nothing to read, you’ve bought back a few minutes of availability and learned nothing, and the same bug is on the calendar for next week. Netflix says jvmquake “mitigated dozens of incidents, each time in mere minutes,” and separately that the core dumps let them chase down real bugs in Cassandra and Elasticsearch offline. Those are two different wins and only the second one compounds.
The other direction: not dying in the first place
Uber’s post is the mirror image. Same enemy, opposite end. Instead of “how do I detect a JVM that’s already lost,” it’s “what actually moves GC pauses on a heap the size of a small car.”
Their trigger point is a number worth writing down: GC tuning becomes worth your time when pauses consistently exceed 100ms. Above that you’re paying for it in throughput and reliability. Below it, go do something else.
The result I like best is the counter-intuitive one. On the HDFS NameNode they raised the heap from 120GB to 160GB, which should help, and ParNew pause times went up about 35%. More memory, worse pauses. The reason is that they’d grown the total heap without growing the young generation, so the old generation got bigger while the young gen stayed at 7.4GB, and the young collection has to scan old-gen references to find what’s still pointing into the young space. Bigger old gen, more scanning, longer “young” pauses. The fix was to raise young gen to 16GB alongside the heap and set -XX:ParGCCardsPerStrideChunk=32k to chunk that scan better.
HDFS NameNode: -Xmx160g -Xmn7.4g vs -Xmx160g -Xmn16g -XX:ParGCCardsPerStrideChunk=32k
Uber’s rule of thumb from that: young gen is typically 20 to 50% of total heap, more if your service allocates heavily, and when you grow the heap you grow the young gen with it. And size the total heap about 20% above the maximum live footprint you see in verbose GC logs, so a full GC that reclaims almost nothing doesn’t immediately trigger the next one. That cascade, full GCs freeing too little and instantly re-firing, is exactly the death spiral from the top of this post, seen from the tuning side. On their Presto coordinator the fix for it was unglamorous: add 10% more heap and the cascade stopped.
The GC problem that wasn’t a GC problem
The Hive Metastore story is the one I’d tell a junior engineer, because it’s the trap. API latencies went from under 100ms to 2 to 4 seconds. GC logs looked awful: 2,258 collections averaging ~177ms, heap sawtoothing violently. Every instinct says tune the collector.
The actual cause was a metrics collector daemon with its backoff set to 1 millisecond instead of 1 second. A thousand mbeans calls a second, generating garbage at roughly 400 Mbps. Nobody’s collector survives that. They fixed the one wrong constant and collections dropped from 2,258 to 143.
Hive Metastore: GC events, before and after fixing a 1ms backoff
I’ve done the version of this where you spend two days on GC flags for a problem that was one line of application code, and the tell is always the same: the allocation rate is absurd for what the service is supposed to be doing. Check that the traffic into the heap makes sense before you tune what’s cleaning up after it.
Their other findings in the same spirit. On the Presto coordinator, string deduplication was costing 6.59% of runtime in GC pauses; turning it off dropped that to 3% and weekly errors fell from 2.5% to 0.73%. And of all the parameters they swept, only one or two per service actually moved anything. TLABSize, ConcGCThreads and friends did essentially nothing. Sweeping twenty flags is how you convince yourself you’re working.
They also ran Azul’s C4 collector against CMS for the large-heap case, and it’s a real result with a real price: ~17ms RPC queue latency versus ~24ms, and pauses that stay flat even at 650GB heaps rather than growing with the heap. The price is about 150GB of extra off-heap memory for a 200GB heap. That’s the honest shape of the modern pauseless collectors, ZGC and Shenandoah included. You’re buying flat pauses with memory and CPU.
The takeaway
Two halves of the same problem, and you want both.
Detection first, because it’s cheap and you probably have nothing. Your health checks almost certainly cannot tell a working JVM from one that’s spending 95% of its wall clock in GC, and neither can ExitOnOutOfMemoryError, because that failure never reaches an out-of-memory condition. The debt counter is the fix and the idea travels: GC time borrows, application time repays, floor at zero, kill above a cap. Under 50% GC time it self-corrects, over 50% it runs away, and you didn’t have to pick a threshold for that.
Then tuning, in this order. Check the allocation rate before the collector, because the Metastore bug was a wrong constant and the flags would never have found it. Size the young generation as a fraction of the heap, not as a fixed number you forget to revisit when the heap grows, or you’ll get Uber’s 35% pause regression from an upgrade that was supposed to help. Leave the heap ~20% of headroom above live footprint so a full GC that frees nothing doesn’t immediately call the next one. And expect one or two flags to matter, not twenty.
The last thing, the one worth building before you need it: whatever kills the sick process must leave a body. A restart with no heap dump and no core is the same incident again next week, and you’ll have learned nothing except how to restart it faster.
Comments