Kafka to PostgreSQL: How to Move Your Data

Moving Kafka into PostgreSQL is a pipeline with a size mismatch you have to design around. Kafka topics are unbounded streams, frequently carrying millions of events a day. Airbyte recommends the Postgres destination for volumes of roughly 10GB or less, or for testing.

That does not make the pipeline wrong, it makes the scoping decision the whole job. This guide covers the managed path with Airbyte, and it treats the question of which topics belong here as the important one, alongside a consumption model that is not what most people expect from a Kafka consumer.

Kafka to PostgreSQL at a glance:

Why move data from Kafka to PostgreSQL?

Two situations account for most of these pipelines, and both involve a narrow slice of what Kafka carries.

The first is giving an application queryable state. A service needs to answer questions about events it did not produce, and querying Kafka is not how you do that. Landing a specific topic in Postgres gives it a table to read with indexes and joins, without writing a bespoke consumer that maintains its own store.

The second is retention beyond the topic. Kafka keeps messages for days rather than years, so a low-volume but important topic, a configuration change log or an audit trail, is being deleted on a rolling basis. If you want to analyse a high-volume topic instead, a warehouse or ClickHouse is the destination and this pipeline will struggle.

What do you need before you start?

One of these can rule the pipeline out and one determines whether it stays healthy:

A broker security setup the connector supports. You choose a protocol covering how the connector talks to brokers and authenticates. Community reports indicate the Kafka connectors cannot authenticate against brokers secured with SASL_SSL and SCRAM, so verify that against current documentation if that is your cluster.

Topic throughput figures. How many messages per minute each candidate topic produces, and how large the messages are. That number decides whether Postgres is a sensible destination at all, and it is the check most often skipped. The Kafka source documentation covers the connector's settings.

A dedicated consumer group ID. Kafka tracks read position per consumer group, so give Airbyte its own rather than sharing with an existing application. Sharing means the two compete for partitions and neither sees everything.

A Postgres user with write permission. On the target schema. If you already run a Postgres destination on an older connector version, read the migration guide before upgrading, because downstream models referencing raw tables will need updating.

If your brokers restrict inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.

How do you build a Kafka to PostgreSQL pipeline in Airbyte?

Step 1: Choose topics against the volume guidance

Take your throughput figures and work out what each topic accumulates in Postgres over a year. A topic producing a few thousand small messages a day is entirely comfortable. One producing millions is not, and no configuration changes that. Pick the topics that suit the destination rather than trying to make the destination suit the topics.

Step 2: Configure the Kafka source

Click Sources in the left navigation, then New Source, and select Kafka, following adding a source. Supply the bootstrap servers, protocol and credentials, the message format, a topic pattern where each matching topic becomes a stream, and your group ID. Use the test topic field to confirm the connector can actually consume before pointing it at production topics.

Step 3: Configure the PostgreSQL destination

Click Destinations, then New Destination, and select Postgres, following adding a destination. From version 3.0.0 the Postgres destination uses Direct Load architecture, writing records straight to final tables with no intermediate raw tables, and adds its own metadata columns including a JSONB column recording per-row typing changes.

Step 4: Set polling time and frequency together

Click Connections, then New connection, select your topics and a sync mode. Polling time is how long, in milliseconds, the connector polls for messages per sync, and it should be set alongside the schedule rather than separately, because a short poll on a frequent schedule and a long poll on an infrequent one behave very differently against the same topic.

Then plan retention on the Postgres side, which is the step people forget. A topic that appends forever will grow a table forever, and unlike Kafka there is nothing expiring old rows for you. Decide up front whether this table is trimmed, partitioned by date or allowed to grow.

Is PostgreSQL the right destination for your topic?

Airbyte is direct about this: Postgres is an excellent relational database but it is not a data warehouse, and Postgres destinations are recommended for small volumes of roughly 10GB or less, or for testing. Beyond that, expect slowdowns on large writes and watch memory, CPU and IOPS during syncs.

Kafka exists precisely because systems produce more events than a single database wants to hold, so the mismatch is structural rather than accidental. If somebody has asked you to land the main event topic in Postgres, the honest answer is usually that the topic is the wrong one rather than the destination being wrong.

What works well is a low-volume topic an application needs to query, or a filtered subset produced upstream into its own topic. That second option is worth suggesting: a stream processor writing a narrowed topic gives you a source sized for the destination, and moves the filtering to where it belongs.

What happens if the connector cannot keep up?

Each sync consumes for the configured polling duration and then stops, committing its position in the consumer group. If messages arrive faster than your syncs consume them, the gap between your committed offset and the head of the topic grows on every run.

Lag on its own is survivable because Kafka retains messages and you can catch up. What is not survivable is lag exceeding retention. Once the broker deletes messages you have not read they are gone, and nothing in the pipeline reports it: syncs keep succeeding and simply resume from the oldest message still present.

So monitor consumer lag for Airbyte's group ID using your normal Kafka tooling and alert on it. That is the check that tells you this pipeline is healthy, and sync success in Airbyte does not. On a low-volume topic of the kind this destination suits, lag should be near zero, which makes any growth a clear signal rather than something to interpret.

Frequently asked questions

How much Kafka data can I move into PostgreSQL?

Airbyte recommends Postgres destinations for roughly 10GB or less. Choose low-volume topics, or have a stream processor write a filtered subset to its own topic and sync that.

Is this a real-time pipeline?

No. The connector polls for a configured duration on a schedule rather than consuming continuously, so the destination receives batches at your sync interval.

How do I know if I am losing messages?

Monitor consumer lag for Airbyte's group ID in your Kafka tooling. If lag exceeds topic retention, messages are deleted before being read and syncs still report success.

Will the Postgres table grow forever?

Yes, unless you manage it. Kafka expires old messages and Postgres does not, so decide whether the table is trimmed, partitioned by date, or allowed to accumulate.

Can I do this without writing code?

Yes. The setup above is entirely UI-driven. If your topics carry JSON you will want a view to extract fields from the payload, which is a short piece of SQL.

Get your Kafka data into PostgreSQL

Measure throughput before choosing topics, because that decides whether Postgres suits them at all. Then give Airbyte its own consumer group, set polling time and schedule as one decision, plan retention on the destination side, and alert on consumer lag rather than trusting sync status.

Airbyte's connector catalog includes 600+ pre-built connectors, so a high-volume topic can go to a warehouse while a narrow one lands here. For that pairing, see Kafka to BigQuery, and for the same destination fed by a database, MySQL to PostgreSQL.

Start syncing now →