Amazon Redshift to Kafka: How to Move Your Data
Stream Amazon Redshift into Kafka with Airbyte. Why this is not change data capture, publishing modelled tables, and the contract to give your consumers.
Putting Amazon Redshift onto Kafka gives several systems one feed to subscribe to rather than each querying the warehouse directly. A personalisation service, a reverse ETL job and an alerting consumer can all read the same topic instead of three teams competing for the same cluster slots.
One expectation to set immediately, because warehouse to bus invites the wrong assumption: this is not change data capture. Airbyte reads Redshift on a schedule using a cursor, so the topic receives batches of rows that appeared since last time rather than a stream of database changes.
Amazon Redshift to Kafka at a glance:
Why move data from Amazon Redshift to Kafka?
Two situations account for most of these pipelines, and both are about protecting the warehouse.
The first is fan-out to operational systems. Models built in the warehouse, such as a customer segment or a propensity score, are wanted by several services, and each one polling Redshift directly is a concurrency slot taken from the analysts the cluster exists for. One pipeline onto a topic replaces all of them.
The second is replay. A consumer that breaks can be fixed and re-run from an offset rather than re-querying the warehouse. If you have exactly one consumer, connect it to Redshift directly and skip the bus, because a topic you maintain for a single reader is overhead without benefit.
What do you need before you start?
Most of the preparation is on the Redshift side, because that is where the cost lands:
A read-only Redshift user. With select on the schemas you intend to publish. Create one for the pipeline rather than reusing an analyst account, so its queries are identifiable in your monitoring when somebody asks what is consuming the cluster.
A cursor column per table. The source is cursor-based, so each table needs a column that only increases and is set on every write. Warehouse tables often carry an audit column recording when a row was loaded, and that is usually the most reliable choice.
Topics created in advance. The destination fails on an unknown topic unless the broker creates topics automatically, which is discouraged in production where partition counts and retention should be chosen deliberately.
Knowledge of your Redshift workload. Specifically when the cluster is quiet and how your workload management is configured, because the sync is another concurrent query competing for the same slots as everything else.
If either platform restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list on both sides before you begin.
How do you build a Redshift to Kafka pipeline in Airbyte?
Step 1: Publish models rather than raw tables
Decide which tables belong on the bus, and prefer your modelled outputs to raw landing tables. The value of a warehouse feeding a topic is that consumers receive something already reconciled and defined, and publishing raw tables asks every consumer to redo work your transformation layer has already done.
Step 2: Configure the Redshift source
Click Sources in the left navigation, then New Source, and select Redshift, following adding a source. Supply the host, port, database, schema and read-only credentials. The connector reads over JDBC, so it appears in your cluster's query monitoring like any other client.
Step 3: Configure the Kafka destination
Click Destinations, then New Destination, and select Kafka, following adding a destination. Supply the bootstrap servers, security protocol and credentials, and a topic pattern. One topic per table is the natural arrangement, since a consumer wanting customer segments has no use for order line items.
Step 4: Nominate cursors and schedule off-peak
Click Connections, then New connection, select your tables, nominate a cursor field for each and choose an incremental sync mode. Schedule it after your overnight transformation jobs finish, both because that is when the models are current and because it keeps the read away from business hours.
Size your topic retention against how far behind a consumer might realistically fall, and remember that a backfill lands in a burst rather than spread across time.
Why is this not change data capture?
Because the Redshift source reads with a cursor rather than from a log. A query filtered on a timestamp finds rows that appeared or advanced since last time, and it cannot observe anything that vanished, so deletions never reach your consumers.
On a warehouse source that is less troubling than it sounds, because warehouse tables are usually rebuilt by transformation jobs rather than edited in place. If your models rewrite a table each night with a fresh load timestamp, the cursor captures the new version and the concept of a deletion barely applies.
What matters is telling your consumers. A team building on a topic labelled with a warehouse name will reasonably assume they are receiving changes, and the honest description is periodic batches of new or updated rows with no removals. That sentence in your documentation prevents a category of downstream bug.
What contract should you give your consumers?
The destination writes JSON and only JSON, so consumers cannot validate against a schema registry. Each message value carries the record identifier, the extraction timestamp, the stream name and your row as a nested structure, so consumers unwrap the envelope to reach the columns they want.
Messages are keyed by a UUID Airbyte generates rather than anything from your data, so rows for the same entity can land in different partitions and be consumed out of order. For a consumer loading a lookup table that is harmless. For anything maintaining state per customer, repartition on a business key first.
Be explicit about timing too. The topic receives batches at your sync interval, which on a warehouse source is typically once a night after the models finish, so a consumer expecting continuous updates will be reading yesterday's picture all day. Say so rather than letting them discover it.
Frequently asked questions
Is this change data capture from Redshift?
No. The source is cursor-based, so the topic receives periodic batches of new or updated rows and deletions are never captured.
Why is my sync slowing down other Redshift queries?
Because it occupies a concurrency slot like any other client. Schedule it off-peak, and consider giving it its own workload management queue with a modest allocation.
Which column should I use as the cursor?
An audit column your loading process maintains, rather than a business timestamp that might be back-dated. It needs to increase on every write.
Are rows for one customer ordered?
Not by default. Messages are keyed by an Airbyte UUID rather than a business key, so repartition on one if a consumer maintains state in sequence.
Can I do this without writing code?
The Airbyte setup is entirely UI-driven. Consumers need to unwrap the JSON envelope, and any repartitioning is code in whatever stream processing you already run.
Get your Redshift data into Kafka
Publish modelled tables rather than raw ones so consumers inherit your definitions, find reliable cursor columns, schedule after your overnight models, and write down the contract: periodic batches rather than change capture, no deletions, a wrapped payload, and no per-entity ordering without a repartition.
Airbyte's connector catalog includes 600+ pre-built connectors, so warehouse models can reach operational systems as easily as dashboards. For the same source into another warehouse, see Amazon Redshift to Snowflake, and for a database feeding the same bus, SFTP to Kafka.
Integrate with 700+ apps using Airbyte
Move data from 700+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.