Mixpanel to Kafka: How to Move Your Data
Move Mixpanel into Kafka with Airbyte. Why 60 requests an hour shapes everything, why duplicates are by design, and why compaction cannot remove them.
Moving Mixpanel into Kafka lets several systems consume product events without each holding credentials and competing for the same limited API. Mixpanel is generous with analysis and stingy with requests, so a single pipeline publishing to topics is a considerably better arrangement than four services polling independently.
This guide covers the managed path with Airbyte. Two things shape the build: sixty requests an hour governs everything you can do, and incremental syncs return duplicate records by design rather than by accident.
Mixpanel to Kafka at a glance:
Why move data from Mixpanel to Kafka?
Two situations account for most of these pipelines.
The first is the rate limit itself. Sixty requests an hour does not divide comfortably between a warehouse loader, a personalisation service and somebody's experiment, and every additional consumer makes the contention worse. One pipeline reading Mixpanel and publishing to topics converts a shared scarce resource into an abundant one.
The second is feeding services rather than analysts, where something reacts to product events instead of charting them. The usual caution applies: with one consumer at the other end a bus is overhead, and a direct pipeline is simpler. If a warehouse is the only destination, Mixpanel to BigQuery gets you there with fewer parts to operate.
What do you need before you start?
Four things, and the second one silently changes your numbers if you get it wrong:
Mixpanel credentials and your project. A service account or project secret, along with the project identifier and region. The Mixpanel source documentation covers the options and what each one can reach.
Your project timezone, checked rather than assumed. Mixpanel defaults to US/Pacific, and if your reporting works in another timezone then every daily boundary is in the wrong place. Nothing errors, the numbers are simply shifted, and reconciling them later is miserable.
Topics, created before the first sync. The destination writes to topics that already exist. Set partition counts and retention deliberately, since event volume is usually the largest thing you will put on a bus.
Consumers that can tolerate duplicates. Not a nice-to-have. Incremental syncs return overlapping records by design, so a consumer that counts naively will overcount, and that has to be designed for rather than patched afterwards.
If your organisation restricts access by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.
How do you build a Mixpanel to Kafka pipeline in Airbyte?
Step 1: Confirm the project timezone before anything else
Open the project settings and look at the timezone rather than assuming it matches your business. The default is US/Pacific, which is wrong for most of the world, and because the connector slices data by day, a mismatch puts every daily boundary several hours out. The figures will be internally consistent and disagree with everything else your organisation reports, which is the hardest kind of discrepancy to trace.
Step 2: Configure the Mixpanel source
Click Sources in the left navigation, then New Source, and select Mixpanel, following adding a source. Supply your credentials, project and timezone, then consider the date slicing window, which defaults to thirty days. Smaller windows use more requests against a tight budget; larger ones use more memory per request. Neither extreme is comfortable.
Step 3: Configure the Kafka destination
Click Destinations, then New Destination, and select Kafka, following adding a destination. Supply the bootstrap servers, security protocol and topic configuration. Messages are JSON with no Avro option, and each one wraps the record alongside its identifier, the extraction timestamp and the stream name, so consumers read through an envelope rather than finding Mixpanel's own shape at the top level.
Step 4: Create the connection and schedule against the budget
Click Connections, then New connection, select your streams and a sync mode. Frequent syncs are not available to you in any meaningful sense, because sixty requests an hour is shared across everything this source does. Choose a schedule that leaves headroom for the occasional retry rather than one that consumes the entire allowance on a good day.
Changing the attribution window moves the start date backwards and triggers a re-sync, so treat it as a deliberate operation rather than an adjustment to make casually on a busy afternoon.
Why does sixty requests an hour decide everything?
Because it applies to both API families and it is not generous. One request a minute, shared across every stream you selected and every retry the connector needs, is the entire budget, and no configuration anywhere increases it. Everything else in this pipeline is downstream of that number.
The backfill feels it first. A year of history sliced into thirty-day windows is a manageable number of requests; the same year sliced more finely to keep memory down is not, and the whole exercise stretches into days. This is the tension in the date slicing window: smaller slices are gentler on memory and hungrier for a resource you barely have.
It is also the strongest argument for the bus. If three services each polled Mixpanel directly, they would divide sixty requests between them and interfere with each other's backfills. Publishing once and letting consumers read from topics removes that contention entirely, which is a better justification for Kafka than most pipelines can offer.
Why will your consumers see the same event twice?
Because incremental syncs return duplicates by design. The API filters by day rather than by a finer timestamp, so each sync re-reads a boundary period and republishes events already sent. This is documented behaviour rather than a fault, and it means the topic contains overlap as a matter of course.
Kafka offers no help here. Log compaction keeps the latest message per key, and messages are keyed by an Airbyte-generated UUID, so every copy of an event carries a different key and compaction has nothing to collapse. The duplicates persist for the full retention period regardless of how the topic is configured.
So deduplication is the consumer's job, and it needs Mixpanel's own event identifier from inside the message rather than the Kafka key. Any consumer producing a count, a total or a rate has to deduplicate before aggregating, and a consumer merely reacting to events needs to be idempotent. Say this to consuming teams at the start, because a service that silently overcounts is considerably worse than one that fails, and nobody discovers it from the topic itself.
Frequently asked questions
Can I sync more often than the rate limit allows?
No. Sixty requests an hour covers both API families and every stream you have selected, and there is no configuration that raises it. Schedule with headroom for retries.
Why are the same events appearing more than once?
By design. The incremental filter works at day granularity, so boundary periods are re-read and republished. Consumers must deduplicate on Mixpanel's event identifier.
Will log compaction clean the topic up?
No, because messages are keyed by a generated UUID and duplicates therefore carry different keys. Compaction has nothing to match, so plan retention accordingly.
Our daily figures do not match Mixpanel's. Why?
Check the project timezone, which defaults to US/Pacific. A mismatch shifts every day boundary and produces figures that are consistent with themselves and wrong against everything else.
Can I do this without writing code?
The pipeline, yes. The consumers are yours, and every one of them needs deduplication logic, which is real work rather than configuration.
Get your Mixpanel data into Kafka
Check the project timezone first, because that mistake is silent and contaminates every daily figure. Schedule against sixty requests an hour with headroom for retries, and treat the backfill as a slow exercise rather than a task. Above all, tell consuming teams that duplicates are expected and that neither Kafka nor the pipeline will remove them, so deduplication on Mixpanel's event identifier belongs in every consumer that counts anything.
Airbyte's connector catalog includes 600+ pre-built connectors, so a rate-limited API can serve many consumers from one pipeline. For the same source into a column store, see Mixpanel to ClickHouse, and for a comparable product analytics tool onto the same bus, Amplitude to Kafka.
Integrate with 700+ apps using Airbyte
Move data from 700+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.