We Forwarded Telemetry to Our Customers. Then One of Them Hit 600 Requests a Second.

At a previous job, a customer asked if we could send them their telemetry. They wanted the raw events our product generated about their usage, pushed somewhere they controlled, so their own team could do whatever they liked with it.

It seemed like a reasonable ask. We already had the events. Sending them somewhere was a queue and a few lines of code. We said yes.

Then a second customer asked. Their endpoint was different, their auth was different, and they had their own opinion about what the payload should look like. We said yes again, because turning down a paying customer over something that sounds this small is hard to justify.

The part nobody budgets for

Writing the first version wasn't the problem. Living with it was.

Telemetry is spiky. When a customer's usage surges, the stream of events about that usage surges with it. During peaks, a single client pushed our outbound delivery to around 600 requests a second. We had several clients like that, and their peaks didn't take turns.

So the forwarding system had to autoscale. It had to keep up when one customer's traffic jumped, and it had to stay upright when that customer's endpoint got slow or started throwing errors under the load we were sending it. Every new customer who wanted the feed meant somebody looking at capacity again.

That somebody was an engineer, often me. Every hour spent there was an hour not spent on the product.

It wasn't our product

This is what bothered me most. The telemetry feed wasn't why anyone bought from us. Nobody picked us because our event forwarding was great. It mattered during the sale, and after that nobody thought about it until it broke.

But it ate engineering time like a core feature. Maybe more than some core features, because it dragged in infrastructure we otherwise didn't think about much: outbound networking, retry policy, per-customer credentials, noticing when a customer's endpoint had been down for hours and the backlog kept growing.

If you've run a SaaS product for a while you probably know this kind of work. It rarely comes up in planning, and it comes up constantly on call.

What I actually wanted

I didn't want to stop offering the feed. A customer asking for their own data is a good sign. It usually means they have a data or security team, they're serious, and they plan to stick around. Audit logs into their SIEM, usage events into their warehouse, raw files into their S3 bucket. These requests are reasonable, and they get more frequent as you sell to bigger companies.

What I wanted was to hand the events to something and stop thinking about them. That something would need to:

• absorb a spike from one customer without me adding capacity by hand
• keep one slow endpoint from delaying everybody else
• retry the failures worth retrying and give up on the ones that aren't
• tell me exactly what happened to any given event
• support the next destination a customer asks for without me writing a new integration

Most tooling I'd seen in this space was built for pulling data in, or for sending webhooks and nothing else. The vendor who wants to push their own events out to wherever each customer wants them, buckets and warehouses included, didn't have much.

So I built it

That's Pushrail. You send events to one API. Each of your customers sets up where they want their data to go, either through your dashboard or through a portal embedded in your own product. Pushrail delivers to webhooks, object storage, warehouses, message queues, and a couple of databases. It handles retries, per-destination rate limits and health tracking, and keeps a delivery log you can replay from.

What I want is for the answer to "can you send our data to GCS?" to be "sure, set it up here," and not "let me check with the platform team."

I'm going to write about the specific problems from that old system, because each one turned into its own small rabbit hole. How to keep one customer's spike from slowing everyone else down. What enterprise buyers mean when they ask for audit log export. Why sending files to a customer's bucket sounds trivial and isn't. If you're the one who gets pulled off product work every time a customer feed backs up, those are for you.