Outbound Webhook Architecture: A Reference Design

Every team that builds outbound webhooks ends up with roughly the same architecture: a producer, a queue, a worker pool, a delivery adapter, and a receiver. The differences are in how each component is built. This is a reference architecture for what each piece does and why.

The high-level shape

[Your service] → [Ingest API] → [Queue] → [Worker pool] → [Adapter] → [Receiver]
                                    ↓                         ↓
                              [Persistence]             [Attempt log]
                                    ↓                         ↓
                              [Dead-letter queue]      [Customer dashboard]

Each arrow is a hop where things can go wrong, and each box is a component with its own scaling and reliability characteristics.

Component 1: ingest API

The ingest API is the entry point. Your service POSTs events to it. The ingest API's job:

• Validate the event shape against a schema.
• Assign a system-generated event ID.
• Check the idempotency key against recent events.
• Persist the event durably.
• Enqueue delivery jobs for matching destinations.
• Acknowledge to the producer.

Critical property: fast. The producer is waiting. Aim for <100ms ack latency. Anything slower and your service's hot path gets coupled to your webhook platform's throughput.

Component 2: persistence

Events need to be stored durably before the ack returns to the producer. If a worker crashes mid-delivery, the event needs to be retryable, which means it must exist somewhere durable.

Common choice: Postgres for control-plane data (events, destinations, attempts, customers) plus Redis for hot queue state. Postgres gives durability and queryability; Redis gives latency.

Don't store events only in the queue. Queue persistence guarantees are weaker than database persistence, and you usually want to query the event history for compliance, replay, and observability anyway.

Component 3: queue + worker pool

The queue separates ingest (fast, latency-bound) from delivery (slower, IO-bound). Workers pull from the queue and deliver.

Queue choice: anything with at-least-once delivery and reasonable ack semantics works. BullMQ, SQS, Kafka, all viable. The integration matters more than the brand.

Worker pool: scale horizontally. Each worker should be stateless and idempotent (it can be killed mid-delivery and the work resumes on another worker). Per-worker concurrency limits prevent any single worker from overwhelming a particular receiver.

Component 4: delivery adapter

The adapter is per-destination-type. For a webhook, the adapter:

• Loads the destination config (URL, signing secret, retry policy).
• Computes the signature.
• POSTs the request with the right headers.
• Captures the response.
• Classifies the outcome (success, transient_failure, permanent_failure).
• Returns the result to the worker for queue acknowledgment.

For non-webhook destinations (S3, BigQuery, etc.), the adapter looks different (batching, file-format conversion, native SDK calls), but the contract is the same: take an event, deliver it, classify the outcome.

This is the dimension where complexity grows. Each new destination type is a new adapter with its own auth, batching, and error-handling logic. The right boundary: each adapter is a small, focused module with the same input (canonical event + destination config) and output (attempt outcome). The worker doesn't care which adapter it's running.

Component 5: attempt log

Every adapter run produces an attempt record: event ID, destination, timestamp, outcome, status code, latency, response excerpt. The log is the observability surface, it's what answers "did event X get delivered?" without diving into application logs.

Attempt records have high write volume (one per attempt, often multiple per event due to retries). Plan for write-heavy storage. Postgres with appropriate indexes works at smaller scales; consider ClickHouse or similar for high volume.

Retention is per the customer's plan tier. After retention, attempts get rolled up to summary stats and the detailed records are purged.

Component 6: dead-letter queue

When an attempt is classified as permanent failure, or when transient retries exhaust, the delivery lands in the DLQ. The DLQ is a separate queue (or a marker on the original event) that allows the customer to inspect, fix the underlying problem, and replay.

The DLQ is also where you alert from: a destination accumulating DLQ entries at an elevated rate is a signal of either a degraded receiver or a misconfigured destination.

Cross-cutting: routing

Before delivery, the platform decides which destinations a given event should go to. Routing rules check the event against per-destination filters: event type matches, payload field equals, customer tier is X.

The routing layer is logically between ingest and queue-enqueue. Each event produces N delivery jobs, one per matching destination.

Cross-cutting: transforms

If the destination needs a different payload shape (BigQuery columns vs webhook JSON, for example), a transform runs between routing and delivery. Transforms are per-destination, versioned (so replays use the same transform that was applied to the original delivery), and lightweight.

Where teams underestimate

Three places where DIY implementations almost always fall short:

• Retry policy correctness. The exponential-backoff-with-jitter pattern is well-documented; most implementations have at least one bug (no jitter, infinite retries, retrying 4xx).
• Replay tooling. Re-running deliveries seems straightforward until you account for transform versioning, signature regeneration, ordering, and idempotency.
• Observability. The per-attempt log gets built; the customer-facing surface and operator triage tooling rarely do.

The Reliability cluster covers each in depth.

Using a platform vs building it

This architecture is a few months of focused engineering to build well. For most B2B SaaS, outbound delivery isn't a differentiator, it's plumbing the product needs. The build-vs-buy math usually favors a platform.

Pushrail implements the reference architecture above. The event delivery layer is the public-facing view of what's running underneath every delivery.