IBM Db2 to Databricks: How to Move Your Data

Getting IBM Db2 data into Databricks is usually done first with a nightly extract: an export to flat files, dropped on object storage, loaded into the lakehouse by a job somebody maintains. It works, it runs once a day whether anything changed or not, and it has no idea which rows were deleted.

This guide covers the managed path: replicating Db2 tables into Unity Catalog with Airbyte. The interesting decision arrives before you touch the lakehouse, because there are two Db2 source connectors and the choice between them is not the usual trade between convenience and capability. It is a trade between change data capture and encryption in transit.

IBM Db2 to Databricks at a glance:

Why move data from IBM Db2 to Databricks?

Two situations account for most of these pipelines.

The first is reach. Db2 often holds the oldest and most authoritative data in an organisation, and the people who want to analyse it are using Spark, notebooks and SQL warehouses rather than anything that speaks to Db2 directly. Replicating into Unity Catalog puts that data where the tools already are, and puts governance around it in the same place as everything else.

The second is joining. Db2 is one system among several, and the value comes from combining it with data that lives elsewhere. That framing matters for the choices below, because a table nobody will join is a table you can sync less often, and possibly without the CDC machinery at all.

What do you need before you start?

More of this list involves other people than is usual for an Airbyte pipeline:

A decision on encryption. The Enterprise Db2 connector lists SSL support and SSH tunnel support as unavailable in its feature table. The community connector supports SSL via a PEM file but replicates with a cursor and cannot capture deletes. Db2 usually holds data with a compliance regime attached, so settle this with your security team first.

A DBA to provision CDC. Triggers and tracking tables must exist in advance. The connector does not create them when you click through the UI, and this is the single most common reason a Db2 CDC project takes longer than estimated.

A dedicated read-only user. With access to the relevant tables. Optional, and strongly recommended, for the usual reasons: revocation without collateral damage, and an obvious owner when someone examines what is querying the database.

Unity Catalog access. On the Databricks side each stream becomes a table in Unity Catalog, and the connector needs permission to create Volumes for Avro staging.

One thing to bring your DBA rather than spring on them: triggers will be added to production tables, a new schema will hold tracking tables, those tables grow if syncs stop, and the connector deletes rows from them as it processes. All reasonable, and all better as a designed change than a discovered one.

How do you build a Db2 to Databricks pipeline in Airbyte?

Step 1: Provision CDC on the Db2 side

Have your DBA create the triggers and tracking tables for the tables you intend to replicate, and create the read-only user. Start with a small set of tables rather than the whole schema, so you can watch what the triggers do to write latency on something non-critical before extending.

Step 2: Configure the Db2 source

Click Sources in the left navigation, then New Source, and select Db2, following adding a source. Supply the host, port, database and credentials. If you are using CDC, select Read Changes using Change Data Capture as the cursor method and set the initial load timeout, which defaults to eight hours and bounds the snapshot phase before the connector switches to incremental mode.

Step 3: Configure the Databricks destination

Click Destinations, then New Destination, and select Databricks, following adding a destination. Supply your workspace details, the catalog and schema, and authentication. Namespaces are supported on the source side, so your Db2 schema structure can carry across rather than collapsing into a single flat namespace, which matters on an instance with many schemas.

Step 4: Create the connection and choose deletion semantics

Click Connections, then New connection, pick your source and destination, and select tables and a sync mode per stream. With CDC, deleted rows arrive flagged rather than vanishing, so append and deduped gives a table mirroring current Db2 state while plain append preserves every version including the deletion. Db2 systems frequently sit under audit requirements, and there plain append with a current-state view on top is the safer default, because you can derive the present from the history and never the reverse.

Estimate the snapshot duration before running it. Eight hours is generous for most workloads and not generous for a Db2 instance that has been accumulating data since before the lakehouse existed. Raise the timeout, or bring tables across in tranches so each initial load finishes comfortably inside the window.

How does Db2 change data capture actually work?

Not the way you are probably expecting. Most CDC implementations read the database's transaction log, which is why their overhead on the source is small. The Enterprise Db2 connector uses triggers instead. For each table you replicate, a tracking table is created in a dedicated schema, and triggers on the source table write a row into it whenever something is inserted, updated or deleted.

During a sync the connector reads the tracking tables and deletes the change rows it has processed. The design is sound and it produces genuine delete capture, which a cursor cannot. But the cost lands on every write to your production tables. A trigger firing on each insert, update and delete adds work to transactions that previously did not have it, and on a high-throughput Db2 system that is a performance conversation rather than a footnote.

There is a second consequence. The tracking tables grow between syncs, since rows accumulate until a sync removes them. If a connection is paused, or fails repeatedly over a weekend, those tables keep growing on your source database. Monitor them the way you would monitor any queue a consumer might stop draining.

Can you live without encryption in transit?

For many Db2 shops this decides the whole project. If your traffic would cross a network you do not control, no SSL and no SSH tunnel is not acceptable, and no amount of enthusiasm for CDC changes that.

Where it can work is a deployment where the network path is already private and controlled: self-managed Airbyte inside the same data centre or private network as the Db2 instance, with nothing traversing anything shared. That is a real architecture and plenty of Db2 environments look exactly like it. It is a decision to make deliberately, with the people who own that judgement, rather than one to discover in a security review after the pipeline is built.

If the answer is no, the community connector with SSL and cursor-based replication remains available. You give up delete capture, which on an append-heavy Db2 workload such as transactions or events may cost you nothing at all. Check whether the tables you need are ones where rows are ever deleted before assuming CDC is required.

Frequently asked questions

Does Airbyte read the Db2 transaction log for CDC?

No. The Enterprise Db2 connector implements CDC using triggers and tracking tables rather than log reading, which is why the overhead lands on writes to your source tables.

Can I use SSL with Db2 change data capture?

The Enterprise connector's feature table lists SSL and SSH tunnel support as not available. The community connector supports SSL but replicates with a cursor and cannot capture deletes.

Who sets up the triggers and tracking tables?

Your DBA, in advance of the first sync. The connector does not provision them, and this is usually the longest lead time in the project.

What happens if syncs stop for a while?

Change rows accumulate in the tracking tables on your Db2 instance until a sync processes and removes them. Monitor their size the way you would any undrained queue.

Can I do this without writing code?

The Airbyte setup is entirely UI-driven. The exception on this pipeline is the Db2 side, where the triggers and tracking tables have to be provisioned by a DBA before you begin.

Get your Db2 data into Databricks

Settle the encryption question first, because it determines which connector you can use and therefore whether deletes are available at all. If CDC is viable, involve your DBA early, size the initial load timeout against reality, and watch what triggers do to write latency on a small set of tables before extending to the whole schema.

Airbyte's connector catalog includes 600+ pre-built connectors, so the same Db2 source can feed other destinations later without rebuilding anything. For the same lakehouse fed by a source with log-based capture and a much lighter setup, see PostgreSQL to Databricks, and for a columnar source landing in the same place, ClickHouse to Databricks.

Start syncing now →