Send events to Apache Kafka
Native Kafka producer with SASL/TLS, deliver events into a customer's topic with configurable partitioning.
What is Apache Kafka?
Apache Kafka is the standard durable distributed log. Many data teams already operate Kafka clusters; routing your product events there is often the lowest-friction way to plug your data into their internal architecture. Pushrail's Kafka adapter is a native producer, no proxy, no REST shim, so latency stays low and ordering guarantees match what the customer's downstream consumers expect.
Why deliver events to Apache Kafka
- The customer's data team already operates Kafka, no new infrastructure to learn.
- Streaming consumers (Flink, Kafka Streams, Spark Streaming, custom) plug in natively.
- Ordering within a partition is preserved, so per-customer or per-resource ordering is configurable.
- Headers carry envelope fields, so consumers can filter without parsing the payload.
How Pushrail delivers events to Apache Kafka
The Kafka adapter is a native producer that publishes one record per event to the configured topic. Partition keys are configurable, default is `customerExternalId` so all events for a given customer land in the same partition and preserve order. Envelope fields (`event_type`, `occurred_at`, `customer_external_id`, `idempotency_key`) are written as record headers; the canonical JSON event is the record value.
Auth and credentials
Authentication options match what Kafka brokers expose: SASL/PLAIN, SASL/SCRAM-SHA-256, SASL/SCRAM-SHA-512, or SASL/OAUTHBEARER, all over TLS. mTLS with client certificates is also supported. Credentials and certs are stored encrypted at rest; rotations are dashboard-driven. The customer grants a dedicated principal `WRITE` on the destination topic and no other ACLs.
Batching, retries, and replay
The producer uses Kafka's native batching, by default a 100 ms linger and 1 MB batch size, tuned to throughput-leaning. `acks=all` is the default so a record is acknowledged only after all in-sync replicas have it. Transient errors (NotLeaderForPartition, broker disconnects) retry with the native producer retry loop. Permanent errors (TopicAuthorizationFailed, UnknownTopic) land in the DLQ. Replay re-publishes records with the same key and headers; consumers using `idempotency_key` from the headers can dedup if needed.
Example payload
Pushrail accepts the canonical event shape on POST /v1/events. Below is the ingestion request your service makes.
{
"eventType": "order.completed",
"occurredAt": "2026-05-26T14:21:08.493Z",
"source": "billing-service",
"customerExternalId": "acct_8K2zRq",
"idempotencyKey": "order_38a91f-completed",
"correlationId": "req_4f30b2",
"payload": {
"orderId": "ord_38a91f",
"amount": 12900,
"currency": "USD",
"items": [
{ "sku": "PR-PRO-MONTHLY", "qty": 1, "price": 12900 }
]
},
"metadata": {
"tier": "pro",
"region": "us-east-1"
}
}Example configuration
The fields your customer fills in to point Pushrail at their Apache Kafka setup.
{
"type": "KAFKA",
"name": "Customer Kafka cluster",
"brokers": ["kafka-0.acme.example:9094", "kafka-1.acme.example:9094"],
"topic": "pushrail.events.v1",
"partitionKey": "customerExternalId",
"auth": {
"mode": "SASL_SCRAM_SHA_512",
"username": "pushrail-writer",
"password": "••••••••",
"tls": true
},
"producer": {
"lingerMs": 100,
"batchSizeBytes": 1048576,
"acks": "all"
}
}Common use cases
- Plug into a customer's existing Kafka-backed data platform.
- Feed real-time stream processing (Flink, Kafka Streams) on the customer's side.
- Power customer-built features that need sub-second event reaction time.
- Bridge your product into a customer's event-driven architecture without exposing your own bus.
Related destinations
Ordered shard delivery into a customer-owned Kinesis Data Stream, configurable partition keys, native AWS SDK.
Topic delivery with ordering keys and dead-letter topics, native to GCP.
Deliver events into a customer's EventBridge bus, rule-driven routing inside AWS.
Frequently asked questions
Is this a native Kafka producer or a REST shim?
A native producer, no proxy and no REST shim, so latency stays low and ordering guarantees match what the customer's downstream consumers expect. It publishes one record per event to the configured topic.
How is ordering and partitioning handled?
Partition keys are configurable; the default is customerExternalId, so all events for a given customer land in the same partition and preserve order. Envelope fields are written as record headers so consumers can filter without parsing the payload.
What authentication does the Kafka adapter support?
SASL/PLAIN, SASL/SCRAM-SHA-256, SASL/SCRAM-SHA-512, or SASL/OAUTHBEARER over TLS, plus mTLS with client certificates. Credentials and certs are stored encrypted at rest and rotated from the dashboard. The customer grants a dedicated principal WRITE on the destination topic and no other ACLs.
What happens if delivery to a broker fails?
With acks=all, a record is acknowledged only after all in-sync replicas have it. Transient errors like NotLeaderForPartition or broker disconnects retry with exponential backoff; permanent errors like TopicAuthorizationFailed land in the dead-letter queue. Every attempt is recorded in the delivery logs. Replay re-publishes records with the same key and headers, so consumers can dedup on the idempotency_key header.