Posthog to Kafka: How to Move Your Data

Putting PostHog data onto Kafka gives several systems one feed to subscribe to rather than each integrating with PostHog separately. A personalisation service, a warehouse loader and an alerting job can all read the same topic instead of three teams sharing one rate limit.

One expectation to set immediately: this is not a real-time event stream. Airbyte syncs on a schedule rather than tailing PostHog, so the topic receives batches at your sync interval. This guide covers the managed path, and the setting that decides whether your first sync completes.

PostHog to Kafka at a glance:

Why move data from PostHog to Kafka?

Two situations account for most of these pipelines.

The first is fan-out, and it matters more here than on most sources. PostHog applies its rate limits across the whole organisation, so every team integrating separately competes for the same budget. One pipeline into a topic several consumers read turns that contention into a single well-behaved client.

The second is replay. A consumer that breaks can be fixed and re-run from an offset rather than going back to PostHog and spending your shared rate limit on a backfill. If you have exactly one consumer and it is a warehouse, skip the bus and load there directly.

What do you need before you start?

Short list, with one item that is a conversation rather than a credential:

A PostHog personal API key and a start date. If you self-host PostHog, change the base URL from the default to your instance. Self-hosted instances are not subject to PostHog Cloud's rate limits, though your own infrastructure becomes the constraint instead. The PostHog source documentation lists every field.

An agreed share of the rate limit. PostHog applies its limits across the organisation rather than per key, so your sync competes with every other tool and script on the account. Find out what else talks to PostHog before choosing a schedule.

Topics created in advance. The destination fails on an unknown topic unless the broker creates topics automatically, which is discouraged in production where partition counts and retention should be chosen deliberately.

An honest estimate of event volume. PostHog recommends batch exports for large or regular exports of events. If you are moving hundreds of millions of events, use those. If you are moving a few million plus cohorts, flags and persons, this connector is the right tool.

If your Kafka cluster restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.

How do you build a PostHog to Kafka pipeline in Airbyte?

Step 1: Decide which streams your consumers need

Only events is incremental. Projects, annotations, cohorts, feature flags, insights and persons are re-read in full on every sync. For the configuration objects that is cheap; persons is the exception, and on a consumer product with a large identified user base it is genuinely large and re-read every time.

Step 2: Configure the PostHog source

Click Sources in the left navigation, then New Source, and select PostHog, following adding a source. Supply the API key, start date and base URL. Set events_time_step deliberately: it controls how many days of events each request covers, and it is the difference between a first sync that completes and one that dies partway through.

Step 3: Configure the Kafka destination

Click Destinations, then New Destination, and select Kafka, following adding a destination. Supply the bootstrap servers, security protocol and credentials, and a topic pattern. One topic per stream is the natural arrangement, since a consumer interested in events has no use for feature flag definitions.

Step 4: Create the connection and backfill carefully

Click Connections, then New connection, select your streams and an incremental sync mode on events. Run the backfill with a short events_time_step even if steady state would tolerate more, because the historical period is where the shared rate limit bites hardest.

Size your topic retention against how far behind a consumer might realistically fall, and remember that a backfill of historical events lands on the topic in a burst rather than spread across time.

Why is fan-out the point here?

Because PostHog applies its limits at the organisation level rather than per API key. The analytics endpoints are capped at 240 requests per minute and 1,200 per hour, and every tool, script and personal key on the account draws from the same budget.

So three teams each polling PostHog directly are not three independent integrations, they are three claims on one allowance, and each one can throttle the others. Putting a single pipeline in front of a topic replaces that with one client whose consumption you can schedule and reason about.

Many teams end up running both approaches: PostHog batch exports for the event firehose, and this connector for cohorts, feature flags, insights and persons onto the bus. That is a sensible split rather than a compromise, because the two mechanisms are good at different things.

What contract should you give your consumers?

The destination writes JSON and only JSON, so consumers cannot validate against a schema registry. Each message value carries the record identifier, the extraction timestamp, the stream name and your event data as a nested structure, so consumers unwrap the envelope to reach the fields they want.

Messages are keyed by a UUID Airbyte generates rather than the person or distinct identifier, so events for the same user can land in different partitions and be consumed out of order. For a consumer aggregating counts that is harmless. For anything building per-user state, repartition on the person identifier first.

Be explicit about timing and about the full-refresh streams. The topic receives batches at your sync interval rather than continuously, and the non-incremental streams republish their entire contents on every run, so a consumer reading persons should expect the whole set repeatedly rather than a change feed.

Frequently asked questions

Is this a real-time event stream?

No. Syncs run on a schedule rather than continuously, so the topic receives batches at your sync interval. Use it for durable fan-out rather than for triggering things.

Does the PostHog rate limit apply per API key?

No. It is applied across your entire organisation, which is the main argument for one pipeline feeding several consumers rather than several direct integrations.

My backfill keeps failing partway through. What should I change?

Reduce events_time_step so each request covers fewer days. Smaller slices keep individual requests within what the API returns and let the sync checkpoint more often.

Why does the persons topic keep receiving everything?

Only events syncs incrementally. The other streams are re-read in full each time, so they republish their whole contents on every run rather than acting as a change feed.

Can I do this without writing code?

The Airbyte setup is entirely UI-driven. Consumers need to unwrap the JSON envelope, and any repartitioning is code in whatever stream processing you already run.

Get your PostHog data into Kafka

Tune events_time_step before anything else, agree your share of the organisation's rate limit, create your topics deliberately, and write down the contract: batches rather than a stream, a wrapped payload, no per-user ordering without a repartition, and full republication on the non-incremental streams.

Airbyte's connector catalog includes 600+ pre-built connectors, so product events can reach a bus and an analytical engine at once. For the analytical version, see PostHog to ClickHouse, and for another product analytics source feeding the same destination, Amplitude to Kafka.

Start syncing now →