PostgreSQL to Kafka: How to Stream Your Data

Search for PostgreSQL to Kafka and most of what comes back is Debezium. That is not an accident. Debezium was built for this exact path, it holds a replication slot open and emits events as they commit, and its output carries the before and after images that stream processing usually expects.

Airbyte will also move Postgres into Kafka, with no code and no Connect cluster to run. There are real reasons to choose it, but you should choose it deliberately rather than by default, because the destination has a particular shape.

Should you use Airbyte or Debezium?

Debezium is a streaming process. It emits events as transactions commit, with latency measured in milliseconds, and its message format is what Kafka Connect ecosystems are built around.

Airbyte is a batch scheduler. It runs a sync, reads what changed since the last one, writes it, and stops. Latency is whatever your schedule interval is.

Airbyte wins when Kafka is one destination among several and you would rather not run a separate ingestion stack per target. It wins when the team owning the pipeline is a data team rather than a platform team, because the operational surface is a schedule instead of a Connect cluster. And it wins when your latency requirement is minutes, which covers more use cases than people usually admit. Debezium wins when you need per-transaction fidelity, sub-second latency, or the change-event shape by default.

What happens to deleted rows?

This is where an assumption carried over from Debezium causes trouble. The Kafka destination supports full refresh append and incremental append only. Neither overwrite nor deduped modes are available.

So if you run CDC into this destination, deletes will not remove anything. The deletion arrives as a record and the topic appends it like any other message. Your consumer has to interpret it. That is normal for a log-shaped destination, but it surprises people who expect the destination to maintain state on their behalf.

One version note is specific to this pairing. Airbyte's documentation records that Open Source users running the Postgres source with the Kafka destination need platform version v0.40.0-alpha or newer and Kafka connector 0.1.10 or newer. Check both before debugging anything else.

How do you set up the Postgres source?

Decide between CDC and cursor-based incremental first, because the prerequisites are different.

Cursor mode needs a monotonically increasing column, usually an updated timestamp or a serial ID, and a read-only user with SELECT on the tables you want. It is simpler, and it misses deletes along with any update that does not touch the cursor.

CDC mode reads the write-ahead log. You will need wal_level set to logical, a replication slot using the pgoutput plugin, a publication covering your tables, and a user with the REPLICATION attribute. Changing the WAL level requires a restart.

Now the part that matters operationally. A replication slot guarantees the consumer will not miss anything, and it delivers that guarantee by retaining WAL segments until they are read. If Airbyte stops reading, because a connection is paused, a sync is failing repeatedly, or a credential expired, the slot keeps holding WAL and disk on your primary grows. If it fills, the database stops accepting writes. Alert on replication slot lag on day one, and drop slots you are no longer consuming.

How should you route topics?

The topic_pattern field controls where messages land. A literal value sends every table to one topic. The {namespace} and {stream} variables give each table its own.

For Postgres, one topic per table is usually what you want, which argues for the dynamic pattern. That in turn means new tables produce new topics, and a sync fails when the topic does not exist and automatic topic creation is off at the broker. Airbyte's documentation recommends leaving it off in production and creating topics deliberately, so adding a table to a publication becomes a two-step change: the publication and the topic.

Remember the record envelope when you write consumers. The row itself sits inside _airbyte_data, alongside an assigned ID, an emitted timestamp, and the stream name.

Which producer settings matter?

The connector exposes most of the standard Kafka producer configuration. Enable idempotence so producer retries do not create duplicates. Set acks to all if you would rather fail a sync than lose a message. Raise batch size and linger together for throughput on large syncs. And check the maximum request size against your widest row, because a Postgres table carrying JSONB columns can produce large messages.

Frequently asked questions

Will deletes propagate to Kafka?

A CDC delete arrives as a record on the topic. Nothing is removed, because the destination has no deduped or overwrite mode. Your consumer decides what a delete means.

Do I need CDC, or is a cursor enough?

If you never delete rows and every update touches your cursor column, a cursor is enough and considerably simpler. Otherwise CDC is the only option that will not silently drift from the source.

Why is disk filling up on my Postgres primary?

An inactive replication slot holds WAL segments indefinitely. If a CDC connection is paused or failing, the slot keeps growing. Monitor slot lag and drop slots you are not consuming.

Can I run this on Airbyte Cloud?

The Kafka destination shows as available on Core and PyAirbyte, not on Standard, Plus, Pro, or Enterprise Flex. Plan on a self-managed deployment.

Get your PostgreSQL data into Kafka

Pick the tool that matches your latency requirement, alert on slot lag before anything else, and design your consumers around an append-only topic. If your real target is a queryable store, see our guide to PostgreSQL to Snowflake. Streaming from a different source into the same topics? See n8n to Kafka.

Start syncing now →