IBM Db2 to Kafka: How to Move Your Data

Streaming Db2 to Kafka publishes changes from an enterprise database onto topics that modern services can consume, which is the usual first step in unpicking a mainframe-era system without rewriting it. Airbyte can build this pipeline without code, but two constraints sit at either end and both are worth knowing before you design anything.

The Kafka destination is a Marketplace-tier connector available on Airbyte Core and PyAirbyte but not on the Standard, Plus, Pro, or Enterprise Flex plans. And the standard Db2 source has no change data capture. Together those shape what this pipeline can honestly deliver.

What can the Db2 source actually capture?

The standard IBM Db2 source supports full refresh and cursor-based incremental sync, built on the IBM Data Server Driver for JDBC and SQLJ, a pure-Java Type 4 driver. There is no change data capture.

For a Kafka pipeline that matters more than usual, because the point of putting a database on a topic is usually to broadcast changes. With a cursor you broadcast insertions and any update that touches the cursor column. Deletes never appear, and neither does an update to a column your cursor does not track. Consumers building state from the topic will drift from the database, slowly and silently.

The Db2 Enterprise connector does capture INSERT, UPDATE, and DELETE, but through database triggers writing to tracking tables in an _ab_cdc schema rather than by reading a log. The connector reads those tables during each sync and removes processed rows older than the current checkpoint. Setting it up needs a privileged database user, triggers add write overhead to your source tables, and dropping a source table means recreating its triggers and tracking table by hand.

It is also explicitly not compatible with IBM InfoSphere Change Data Capture. If InfoSphere already runs in your environment, this builds a second, parallel mechanism rather than reading from the existing one.

What lands on the topic?

Each Airbyte stream maps to a Kafka topic. The message key is the UUID Airbyte assigned the record, and the value is an envelope with four fields: _airbyte_ab_id, _airbyte_emitted_at, _airbyte_stream, and _airbyte_data holding your row as a JSON blob. Consumers must unwrap that data field, which is worth writing into the consumer contract before anyone builds against it.

The Kafka destination writes JSON only, with Avro described as a future addition, so consumers expecting Avro against a schema registry need a translation step. It supports full refresh append and incremental append and neither overwrite nor deduped modes, which is an honest reflection of what a topic is rather than a shortcoming.

Topic naming and broker configuration

The topic_pattern field takes a literal name or a template using {namespace} and {stream} variables. For a database source, one topic per table is usually right, which argues for the dynamic form.

That brings a dependency. A new table produces a new topic, and the sync fails if the topic does not exist and automatic topic creation is off at the broker. Airbyte's documentation recommends leaving it off in production and pre-creating topics deliberately, so adding a Db2 table to your replication set becomes a two-step change. Note also that a naming transformation is applied to the target topic, replacing some special characters, so check the resulting name rather than assuming your pattern survives verbatim. Db2 schema and table names often carry conventions that will not pass through unchanged.

The connector exposes most of the standard Kafka producer configuration. Enable idempotence so producer retries do not create duplicates, set acks to all if losing a message is worse than failing a sync, and check the maximum request size against your widest Db2 row. Some SSL configurations are noted as not yet available, which is worth verifying against your cluster's requirements early, since enterprise Kafka deployments are rarely unsecured.

Setting up Db2 access

Create a dedicated read-only user and role with access to the schemas you plan to replicate, granting the role database connect access and assigning it to the Airbyte user. On a Db2 system this is usually mandatory rather than merely recommended, since permission control and auditing tend to be governed centrally.

Expect network arrangements. Db2 instances are seldom publicly reachable, and the quickest way to confirm Airbyte can reach yours is to test the connection in the UI rather than reasoning from the network diagram.

Frequently asked questions

Will deletes appear on the topic?

Not with the standard connector, which is cursor-based. The Enterprise connector captures deletes via triggers, and they arrive as records on the topic. Nothing is removed either way, since the destination is append-only.

Can I run this on a paid Airbyte Cloud plan?

The Kafka destination shows as available on Core and PyAirbyte only, not on Standard, Plus, Pro, or Enterprise Flex. Plan on a self-managed deployment.

Can consumers read Avro with a schema registry?

Not directly. The connector writes JSON only, with Avro noted as a future addition, so you would need a translation step between the topic and those consumers.

Why did my sync fail after adding a table?

A new stream resolves to a topic that does not exist yet. Pre-create it, or enable automatic topic creation at the broker, which the documentation advises against in production.

Get your Db2 data into Kafka

Confirm which connector you have, design consumers around append-only JSON, and pre-create your topics. If your real target is a queryable store rather than a log, see our guide to Oracle to ClickHouse. For streaming from a different source into the same topics, see n8n to Kafka.

Start syncing now →