Amplitude to Kafka: How to Stream Your Data
Stream Amplitude events into Kafka with Airbyte. The 4 GB export limit, tuning Request Time Range, the EU region setting, and sizing topics for bursts.
Streaming Amplitude to Kafka sounds like it should be a real-time pipeline and is not one, and being clear about that up front will save an argument later. Amplitude's Export API hands you events in bulk over a time range. Airbyte reads that range, produces records, and the Kafka destination writes them as messages. What arrives in your topic is a batch wearing streaming clothes.
That is still useful. Kafka gives you fan-out to multiple consumers, replay from an offset, and a buffer in front of systems that cannot take a firehose. Just size the topic for spiky bulk arrivals rather than a steady trickle, because a sync covering 24 hours of a high-traffic product lands as one large burst, not as an even flow.
Request Time Range is the setting that matters
The Events stream reads Amplitude's Export API, and that API caps each export at 4 GB. Exceed it and the request errors. Get close to it and the request times out instead. Both failures have the same fix: reduce the Request Time Range, which sets how many hours each Events request covers. It defaults to 24.
For a low-volume product the default is fine and you will never think about it. For a consumer app doing millions of events a day, 24 hours of events is well past 4 GB and the first sync simply fails. Drop it to six hours, or one, and the same data moves in more, smaller requests. There is no formula published for this, so the practical approach is to start conservatively for the backfill and raise it once steady state is smaller.
The other streams come from the Dashboard REST API, which uses a cost-based budget rather than a request count, with an hourly allowance, a five minute burst allowance and a cap on concurrent requests. You do not need to manage this: the connector tracks what each request costs and throttles itself to stay inside the limits. It is worth knowing it exists so that a slow sync reads as expected behaviour rather than a fault.
Get the region right
Configuration is an API key, a secret key, a replication start date and a data region. The region setting defaults to the standard server, and if your Amplitude project sits in the EU data centre you must select the EU residency server instead. Leaving it wrong does not produce a helpful message, it produces a connection that cannot find your data.
Leaving the start date blank replicates everything, which on a mature Amplitude project is a very large first sync into a Kafka topic. Set it to the earliest date anyone will actually query.
What consumers receive
The Kafka destination writes JSON, with no Avro option, so a schema registry cannot validate what Airbyte produces. Each message is keyed by the UUID Airbyte assigns the record, and the value wraps your event data in an envelope alongside that identifier, the extraction timestamp and the stream name.
For analytics events this matters less than it does for entity data. Nobody needs per-user ordering guarantees on a page view, and a random key spreads load evenly across partitions, which is what you want for throughput. But consumers do have to unwrap the envelope, and topic sizing has to assume the burst rather than the average. Set partition count and retention on the assumption that one sync delivers a day of events at once.
Frequently asked questions
Why does my Amplitude Events sync fail or time out?
The Export API caps each export at 4 GB and large requests can time out. Reduce the Request Time Range so each request covers fewer hours.
Is this a real-time pipeline?
No. Amplitude's Export API delivers events in bulk over a time range, so the topic receives periodic bursts. Kafka gives you fan-out and replay, not low latency, in this design.
My Amplitude project is in the EU. What changes?
Set the data region to the EU residency server rather than leaving the standard server default. The connector otherwise queries the wrong data centre.
Do I need to handle Amplitude's rate limits myself?
No. The connector tracks the cost of each Dashboard API request and throttles automatically to stay inside the hourly and burst budgets.
Get your Amplitude data into Kafka
Set the region, set a real start date, and treat Request Time Range as the first thing you tune rather than the last. Then size the topic for bursts and tell your consumers the payload is wrapped. The rate limiting takes care of itself, which is more than most analytics connectors offer.
For the same destination fed by a warehouse rather than an analytics API, see BigQuery to Kafka. For another marketing analytics source and its own export constraints, see AppsFlyer to BigQuery.
Integrate with 600+ apps using Airbyte
Move data from 600+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.