The first time I saw it, the collector pod was in a restart loop. Not crashing on a bad config, not failing a health check, just quietly climbing in memory for about half a minute and then getting OOMKilled, coming back, and doing it again. kubectl get pods showed a restart count going up like a clock. The config had shipped weeks earlier and worked fine in staging, where the traffic was a trickle. Production was not a trickle.
The processor doing this was tail_sampling. We had turned it on to be smart about cost, keep the error traces and the slow ones, throw away the boring 200s that all look the same. That part worked. The part nobody mentioned is that a tail sampler has to hold whole traces in memory long enough to decide, and “long enough” times “how many arrive” is a number you can compute, and that number was bigger than the memory limit.
The problem
Head sampling decides at the first span: roll the dice when the trace starts, keep it or drop it, done. Tail sampling can’t do that. To keep the slow traces you have to wait until the trace is finished to know it was slow, so the collector buffers every span of every trace in a ring buffer and starts a per-trace timer called decision_wait. Only when that timer fires does it run your policies and decide. So the traces sitting in memory at any moment are roughly the ones that arrived in the last decision_wait seconds, which is rate × decision_wait, times some headroom. At 1,000 traces/sec with a 60s wait that is 60,000 traces resident, all the time. At ~10 to 50KB a trace, that is real memory, and when it crosses the container limit the kernel kills the collector. Worse, when the ring buffer fills before your traffic slows down, the processor evicts the oldest traces to make room, and the oldest traces are the ones whose decision hasn’t fired yet, which can be exactly the error traces you built this thing to keep.
I wanted to watch all three of those happen, so I ran a contrib collector locally, pointed a load generator at it, and read the numbers back off the collector’s own metrics endpoint.
What the processor is actually holding
The config is small. This is the whole processor:
processors:
tail_sampling:
decision_wait: ${env:DECISION_WAIT}
num_traces: ${env:NUM_TRACES}
expected_new_traces_per_sec: ${env:EXPECTED_NEW_TPS}
policies:
- name: keep-errors
type: status_code
status_code:
status_codes: [ERROR]
- name: keep-slow
type: latency
latency:
threshold_ms: ${env:LATENCY_THRESHOLD_MS}
Two policies: keep a trace if any span has status ERROR, or if the trace took longer than the latency threshold. decision_wait is how long each trace waits before those policies run. num_traces is the size of the ring buffer, the hard cap on how many traces can be in memory at once. The docs suggest sizing it as rate Ă— decision_wait Ă— 2, and that Ă— 2 is the whole story: it is not a safety margin you can ignore, it is telling you how much RAM the processor is going to want.
The collector exposes its internals on a Prometheus endpoint, so I didn’t have to guess. The metric that mattered most was otelcol_processor_tail_sampling_sampling_traces_on_memory, the live count of traces in the buffer, plus ..._sampling_trace_dropped_too_early for evictions and ..._global_count_traces_sampled for what survived. The names carry a sampling_ infix in this build, which I only found by curling the endpoint on the running collector, so don’t trust a metric name from a blog, including this one, check yours.
Memory tracks decision_wait, exactly like the formula says
First experiment: fix the rate at 1,000 traces/sec, give the ring buffer enough room, and turn the decision_wait dial from 5s up to 60s. Then watch the collector’s resident memory. If the formula is real, memory should climb with the wait.
It does.
Collector RSS vs decision_wait (rate fixed at 1,000 traces/sec)
Nothing about the traffic changed between those four bars. Same rate, same trace size, same everything. The only thing I moved was how long each trace sits in the buffer before the collector decides, and the memory followed it up to 576 MB. If your container has a 512 MB limit, which is a completely normal limit, a 60s decision_wait at 1,000 traces/sec doesn’t fit, and it never did. It fit in staging because staging was doing 20 traces/sec.
The part that stings: it drops the oldest first
Memory pressure is one failure. Here is the other one, and it’s the one that would have cost me a debugging night if I hadn’t gone looking.
Set decision_wait long, 30s, so no trace gets a decision during the test. Inject 500 error traces first. Then flood the collector with normal traces, more than the ring buffer can hold, all before that 30s timer fires for anyone. The error traces are now the oldest things in the buffer. What happens to them?
Error traces kept, out of 500 sent (buffer overflowed before any decision fired)
Zero. With the small ring, all 500 error traces were gone before the collector ever ran the policy that was supposed to keep them. The dropped_too_early counter reported 6,500 traces evicted before their decision. This is the quiet failure, and it is so much worse than the OOM, because the OOM at least tells you. This one doesn’t. The collector is up, it’s green, it’s exporting traces, your storage bill went down, and the whole time it is throwing away the errors first because they are the oldest things in a buffer that is too small. You would trust that pipeline. You would build alerts on it. And on the day something actually breaks, the trace you go looking for was evicted three hours ago to make room for a flood of healthy 200s.
The fix is the same knob that fixes the memory, num_traces, which is the thing I got wrong about this processor for a while. I thought decision_wait was the memory dial and num_traces was just a safety cap. It’s the other way around. On this build a trace’s memory isn’t released the instant its decision fires, it stays in the ring until num_traces eviction reclaims the slot, so num_traces is the real memory knob, and the rate × wait × 2 rule is really about sizing that ring.
And then it just dies
To make sure the OOM wasn’t hypothetical, I gave the container a 400 MB limit, set decision_wait to 90s, and pushed 3,000 traces/sec with num_traces sized to actually fill. The projected working set is well past 400 MB, so this should die.
It died in 67 seconds. RSS climbed to about 357 MB against the 400 MB cap, the kernel OOMKilled the container, and it came back with a restart count of 1. Left running, that is the restart loop from the top of this post, on a timer, forever, until someone lowers decision_wait or raises the limit or shrinks the buffer. (One honest wrinkle: after the restart the container’s State.OOMKilled flag resets to false, so the durable evidence of the kill is the restart count going up under the cap while memory was climbing, not the flag.)
So is it worth it
Yes, and it’s not close, which is the frustrating part, because the thing that OOMs you is also genuinely good at its job. Last experiment: a realistic mix, 1% error traces and 1% slow ones, the policy keeping both, a ring big enough to decide correctly. How much traffic actually made it to storage?
Spans stored: tail sampling keeping only errors and slow traces
98% less to store, and you kept the traces you’d actually open. That is a real bill, cut hard, for a policy that took six lines of YAML. The catch is that the 98% and the OOM are the same mechanism. You save that much precisely because the collector held everything long enough to be picky, and holding everything long enough is the thing that costs the memory.
The takeaway
Tail sampling is worth it, and it will crash you if you treat num_traces as an afterthought. Before you ship it, do the arithmetic the docs are quietly handing you: take your real peak rate, multiply by decision_wait, multiply by 2, and that is roughly how many traces the collector will hold. Multiply that by 10 to 50KB and compare it to your container limit. If it doesn’t fit, it will OOMKill on a loop the first time production traffic shows up, and long before that, if num_traces is set too small, it will quietly evict your oldest traces to stay under the cap, which are the error traces you turned this on to keep. None of it is hidden, the × 2 in the formula was the warning all along. Size the buffer, cap the rate you feed it, and give it the memory the arithmetic says it needs.
The harness, the collector config, and every number in the charts are on GitHub. These are laptop numbers on a single collector, meant to show the mechanism, not to size your production fleet, your trace sizes and rates are your own.
Comments