The first analytics pipeline I helped run sent one event per request. A button click, a page view, an export started: each one its own little JSON body, its own TLS handshake amortized across nothing, its own line in the bill. It worked. It also cost more than the feature it was measuring, and when someone asked why the ingestion bill kept climbing faster than traffic, the honest answer was that we were paying to ship the same field names a few billion times a day.
So we did the obvious thing. We batched. Group a few hundred events, compress the batch, send one request. And the first time I measured it I expected the compression ratio to just keep climbing as the batches got bigger: more data, more redundancy, more for the compressor to chew on. It climbed for a while. Then it stopped at 4.7x and would not move, no matter how big the batch got.
That plateau is the whole post. It turns out the thing that makes an analytics event useful is the same thing that makes it incompressible, and once you see why, you stop trying to beat it and start designing around it.
The problem
An analytics event is mostly boilerplate. Here’s a compact one, the kind a click generates:
{"event":"button_clicked","user_id":"u_9f3ac21b7e004d18","session_id":"s_4a1c9e2f7b03","ts":1721030400123,"props":{"page":"/design/edit","referrer":"/home","device":"desktop","country":"US","ab_variant":"B","duration_ms":842,"position":3}}
Compact-encoded, mine averaged 292.7 bytes. Look at what’s actually in there. The field names (event, user_id, session_id, props, page, device) repeat on every single event. The enum values repeat too: there are maybe a dozen event names, three device types, eight country codes. If you’re sending these one at a time, you are re-transmitting that dictionary billions of times, and a compressor that only ever sees one event has nothing to compare it against.
That’s the naive cost. And you can watch compression do almost nothing about it when the batch size is one:
Compressed bytes per event, by batch size (zstd level 3)
At batch size one, zstd got the event from 292.6 down to 227.2 bytes. That’s a ratio of 1.29x, and it’s basically the compressor shrugging: a little intra-event redundancy, nothing structural. Batch a thousand of them together and the same events cost 62.2 bytes each. Nothing about the events changed. The only thing that changed is that the compressor could finally see that "session_id":"s_" shows up on all of them.
Why batching is really a compression trick
I used to think of batching as an amortization thing: fewer requests, fewer handshakes, less per-message framing overhead. That’s real, but it’s the small win. The big win is that batching is what feeds the compressor. A dictionary compressor like zstd works by finding a byte sequence it has seen before and replacing the repeat with a short back-reference. One event gives it no history. A batch gives it a few hundred near-identical rows, and every field name after the first is a back-reference.
So the ratio climbs with batch size, steeply at first:
Compression ratio vs batch size (zstd level 3)
From batch 1 to batch 10 the ratio more than doubles, 1.29x to 3.15x. From 10 to 100 it climbs again to 4.62x. And then it just… stops. Batch 100 is 4.62x, batch 250 is 4.75x, batch 1000 is 4.72x. Ten times the batch buys you two percent more ratio. The standard deviation at batch 1000 was ±0.009, so that flatness is real, not noise. I ran fifty batches of a thousand and they all landed on 4.7x.
The floor is the identifiers
Here’s the part that took me a minute. Why 4.7x and not 10x, not 20x? The boilerplate is extremely repetitive, you’d think it would crush.
Answer: because the compressible part isn’t the whole event. Every event carries three fields that are, by design, high-entropy: a user_id, a session_id, and (in the real pipeline) a per-event event_id for deduplication. Those are random. That’s the point of them; an identifier that compressed well would be an identifier that collided. zstd can strip the repeated schema down to almost nothing, but it cannot do a thing about sixteen bytes of random hex on every row. So the batch converges to a floor: roughly the size of the incompressible identity payload, plus a rounding error of structure.
That 62 bytes per event at batch 1000 is basically the identifiers, encoded. The schema, everything that made the raw event 292 bytes, has been compressed into the noise. You’re paying to ship the thing that makes each event unique, and nothing more. Which is the correct amount to be paying. The lesson I took: when your compression ratio plateaus, you’re not looking at a compressor limit, you’re looking at your data’s entropy floor. Stop tuning the compressor. Go look at what in the payload is actually random, and ask whether it needs to be that big.
So which compressor, and at what level
Once you know the ratio is capped by entropy, the level knob gets a lot less exciting. I put five options through the same batch of 500 events, no compression, gzip at its default 6, and zstd at 3, 9, and 19:
Codec shootout at batch size 500
Compression ratio
Compress time per batch (ms)
zstd-3 came out at 4.43x in 0.29 ms per batch, 509 MB/s of input. gzip-6 got a slightly better 4.71x but took 1.37 ms, about a fifth the throughput. And zstd-19, the one you reach for when you think ratio is everything, got 5.81x (the best on the board) for 31.05 ms per batch. That’s a hundred times the compress cost of zstd-3 to move the ratio from 4.4 to 5.8. On a pipeline doing billions of events, thirty-one milliseconds of CPU per batch of five hundred is not a compression setting, it’s a capacity problem.
The decompress side barely moved across all of them, 0.06 to 0.18 ms, which matters because your consumers decompress far more often than your producers compress. zstd-3 is the boring correct answer here: most of the ratio, a fraction of the cost, and it decompresses fast. (Ignore the none row’s throughput number in the raw CSV, it’s dividing by a near-zero time and reads as two million MB/s, which just means “instant”.)
The bill you didn’t expect: duplicates
There’s a second half to batching that nobody warns you about, and it’s not a compression problem at all. Once you batch and compress and ship over a network, you have to decide what happens when a send is ambiguous: the server committed the batch but the 200 OK got lost on the way back. The producer doesn’t know it succeeded, so it retries. And now that batch is in the pipeline twice.
The honest guarantee a system like this can offer is at-least-once: if you got a 200, every event is delivered to every consumer at least once. Not exactly once. At least once. Which means duplicates are not a bug, they’re a documented output, and the duplicate rate just tracks how often sends go ambiguous:
Duplicate rate at the consumer vs producer retry probability
At a 1% retry rate I got a 1.05% duplicate rate, 423 extra events on 40,000. At 5%, it was 5.08%. It tracks the retry rate because that’s all it is: the events that rode a retried batch, delivered twice. And this is exactly why every event needs that random event_id: the same field that put a floor under our compression ratio is the field that makes dedup possible. I dedup’d each run by event_id and got back exactly 40,000 unique events, every time, at every retry rate.
So the identifiers cut both ways. They’re the bytes that won’t compress, and they’re the bytes a consumer uses to throw away the duplicate copy of a click. Same field, both jobs.
The takeaway
Batching analytics events is a compression play before it’s a throughput play: a single event barely compresses because the compressor has no history, and the ratio only shows up once you give it a few hundred near-identical rows to back-reference against. But it converges to a floor, and the floor is your data’s entropy: the random per-event identifiers that don’t compress no matter how big the batch gets. For my events that floor was 4.7x and about 62 bytes per event, and once you’re there, chasing a higher zstd level costs you 100x the CPU for single-digit percent gains. Use zstd-3, size your batches into the hundreds, and stop.
And remember what those incompressible bytes are for. At-least-once delivery means retries, retries mean duplicates, and the duplicate rate just tracks how flaky your network is, about 1% duplicates at a 1% retry rate. The only reason a consumer can clean that up is the per-event id, which is also the exact thing capping your compression ratio. You’re paying in bytes for the identifier, and getting dedup in return. Budget for both.
The harness is on GitHub: event generator, the four experiments, and the CSVs. These are laptop numbers meant to show the shape of the thing, not capacity planning for your pipeline; run it against your own event schema, because your entropy floor is set by your identifiers, not mine.
Comments