Amplitude to Kafka: How to Stream Your Data
Stream Amplitude data into Kafka with Airbyte. The 4 GB export limit, why this is not a real-time stream, and the contract to give your consumers.
Putting Amplitude data onto Kafka gives several systems one feed to subscribe to rather than each integrating with Amplitude separately. A personalisation service, a warehouse loader and an alerting job can all read the same topic instead of three teams managing three sets of API credentials.
One expectation to set immediately: this is not a real-time event stream. Airbyte syncs on a schedule rather than tailing Amplitude, so the topic receives batches at your sync interval. This guide covers the managed path, and the export limit that decides whether your first sync completes.
Amplitude to Kafka at a glance:
Why move data from Amplitude to Kafka?
Two situations account for most of these pipelines.
The first is fan-out. Several systems want behavioural data, and giving each its own Amplitude integration means several sets of credentials, several schedules and several things to fix when something changes. One pipeline into a topic they subscribe to is fewer moving parts.
The second is replay. A consumer that breaks can be fixed and re-run from an offset rather than losing whatever arrived while it was down. If you have exactly one consumer and it is a warehouse, skip the bus and load there directly.
What do you need before you start?
Two credentials and two settings that quietly decide whether the first sync succeeds:
An Amplitude API key and secret key. Both found in your Amplitude project settings, and they are per project rather than per account, so a multi-project setup means a source per project. The Amplitude source documentation lists every field.
The correct data region. The setting defaults to the standard server, and if your Amplitude project sits in the EU data centre you must select the EU residency server instead. Getting this wrong produces a connection that cannot find your data rather than a helpful message.
A deliberate start date. Leaving it blank replicates everything, which on a mature project is an enormous first sync against an API with a hard size ceiling per request.
Topics created in advance. The destination fails on an unknown topic unless the broker creates topics automatically, which is discouraged in production where partition counts and retention should be chosen deliberately.
If your Kafka cluster restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.
How do you build an Amplitude to Kafka pipeline in Airbyte?
Step 1: Estimate your daily event volume
Find out roughly how many events your project records per day, because that number determines your Request Time Range before you touch anything else. The Export API caps each export at 4 GB, and a high-traffic consumer product generates well past that in twenty-four hours.
Step 2: Configure the Amplitude source
Click Sources in the left navigation, then New Source, and select Amplitude, following adding a source. Supply the API key, secret key, data region and start date, then set the Request Time Range, which controls how many hours of events each request covers and defaults to 24.
Step 3: Configure the Kafka destination
Click Destinations, then New Destination, and select Kafka, following adding a destination. Supply the bootstrap servers, security protocol and credentials, and a topic pattern. One topic per stream is usually right, since a consumer interested in events has no use for cohort definitions.
Step 4: Create the connection and backfill carefully
Click Connections, then New connection, select your streams and an incremental sync mode where available. Run the backfill with a short Request Time Range even if steady state would tolerate the default, because the historical period is where the size ceiling bites and a failure six hours in costs you those six hours.
Size your topic retention against how far behind a consumer might realistically fall, and remember that a backfill of historical events will land on the topic in a burst rather than spread across time.
Why does the events stream fail or time out?
Because the Export API caps each export at 4 GB. Exceed it and the request errors outright. Get close to it and the request times out instead. Both failures have the same fix, which is reducing the Request Time Range so each request covers fewer hours.
For a low-volume product the 24 hour default is fine and you will never think about it. For a consumer app doing millions of events a day, a day of events is well past the ceiling and the first sync simply fails. Drop it to six hours, or one, and the same data moves in more, smaller requests.
The other streams come from the Dashboard REST API, which uses a cost-based budget. You do not manage that: the connector tracks what each request costs and throttles itself to stay inside the limits, so a slow sync there is expected behaviour rather than a fault.
What contract should you give your consumers?
The destination writes JSON and only JSON, so consumers cannot validate against a schema registry. Each message value carries the record identifier, the extraction timestamp, the stream name and your event data as a nested structure, so consumers unwrap the envelope to reach the fields they want.
Messages are keyed by a UUID Airbyte generates rather than the user or device identifier, so events for the same user can land in different partitions and be consumed out of order. For a consumer aggregating counts that is harmless. For anything building per-user state, repartition on the user identifier first.
Be explicit about timing too. Airbyte syncs on a schedule rather than streaming, so the topic receives batches at your sync interval and an event happening now appears at the next run. Tell consuming teams that plainly, because the combination of Amplitude and Kafka leads people to assume something this pipeline does not offer.
Frequently asked questions
Is this a real-time event stream?
No. Syncs run on a schedule rather than continuously, so the topic receives batches at your sync interval. Use it for durable fan-out rather than for triggering things.
Why does my Amplitude events sync fail or time out?
The Export API caps each export at 4 GB and large requests can time out. Reduce the Request Time Range so each request covers fewer hours.
My Amplitude project is in the EU. What changes?
Set the data region to the EU residency server rather than leaving the standard server default, otherwise the connector queries the wrong data centre.
Are events for the same user ordered?
Not by default. Messages are keyed by an Airbyte UUID rather than the user identifier, so repartition on the user if a consumer builds per-user state.
Can I do this without writing code?
The Airbyte setup is entirely UI-driven. Consumers need to unwrap the JSON envelope, and any repartitioning is code in whatever stream processing you already run.
Get your Amplitude data into Kafka
Estimate your event volume and set the Request Time Range against it, get the data region right, create your topics deliberately, and write down the contract for consuming teams: batches rather than a stream, a wrapped payload, and no per-user ordering without a repartition.
Airbyte's connector catalog includes 600+ pre-built connectors, so behavioural data can reach a bus and an analytical engine at once. For the analytical version, see Amplitude to ClickHouse, and for another product analytics source feeding the same destination, PostHog to ClickHouse.
Integrate with 700+ apps using Airbyte
Move data from 700+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.