IBM Db2 to Amazon S3 with AWS Glue: How to Move Your Data

Moving IBM Db2 into Amazon S3 with AWS Glue takes data from a system that usually predates everything else in the business and puts it in an open table format the whole AWS estate can query. Athena, Glue jobs, EMR and Redshift Spectrum all read the same tables, and nothing is locked inside a warehouse.

One thing to establish first: Airbyte's current route here is the S3 Data Lake connector, which writes the Iceberg table format to S3 using a supported catalog, of which AWS Glue is one. So you are writing Iceberg tables that Glue registers rather than loose files with a catalogue bolted on.

IBM Db2 to Amazon S3 with AWS Glue at a glance:

Why move data from IBM Db2 to S3 with AWS Glue?

Two situations account for most of these pipelines.

The first is access. Db2 instances are typically the system of record for something important, tightly controlled, and not somewhere you hand out connections freely. A lake copy gives analysts and data scientists the data without adding load or access to the source, which is frequently the entire business case.

The second is keeping your options open. Data in Iceberg on S3, registered in Glue, is readable by every AWS analytical service and by engines outside AWS that speak Iceberg. For data that will outlive several generations of query engine, which Db2 data usually does, that matters.

What do you need before you start?

The AWS side has two prerequisites and the Db2 side has a decision:

An S3 bucket and a bucket policy. Created through the IAM console, granting the connector what it needs on that bucket. The S3 Data Lake destination documentation includes the policy to paste in with your bucket name substituted.

A Glue database. AWS Glue is one of the supported Iceberg catalogs and the one you want if the point is querying from Athena and the rest of the AWS estate. REST, Nessie and Polaris are also available if your organisation standardised elsewhere.

A read-only Db2 user and a decision about deletes. Create an account for the pipeline rather than reusing an application one, and establish what change capture your Db2 connector supports before designing around it, because that determines whether a deleted row ever leaves your lake.

A look at your primary key types. Specifically whether any are numeric, because of how the destination handles them in Iceberg. This is the detail most likely to produce a surprising result and it is covered below.

If your organisation restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list on the Db2 side before you begin.

How do you build a Db2 to S3 pipeline in Airbyte?

Step 1: Set up the bucket and the Glue catalog

Create the S3 bucket and apply the bucket policy, then confirm the Glue database you want tables registered in. Doing this first means the connection test tells you something useful rather than failing on permissions you have not granted, and gives whoever owns your AWS account a chance to review before tables appear.

Step 2: Configure the Db2 source

Click Sources in the left navigation, then New Source, and select IBM Db2, following adding a source. Supply the host, port, database and the read-only credentials, along with any encryption settings your instance requires. Nominate cursor fields where you are using incremental sync.

Step 3: Configure the S3 Data Lake destination

Click Destinations, then New Destination, and select S3 Data Lake, following adding a destination. Supply your bucket, region and credentials, then select AWS Glue as the catalog and give it the database name. The connector writes Iceberg tables to the bucket and registers them in Glue, which is what makes them queryable from Athena without further work.

Step 4: Create the connection and verify from Athena

Click Connections, then New connection, select your tables and a sync mode per stream. If you use a deduplicating mode, the connector translates your primary keys into Iceberg identifier columns and performs upserts as an equality delete followed by an insert, which is Iceberg's merge-on-read approach.

After the first sync, query a table from Athena to confirm the Glue registration worked end to end. That check takes a minute and validates the whole chain rather than just the Airbyte half.

Why do numeric primary keys sort strangely?

This is the detail most likely to catch you out, and it comes from Iceberg rather than from Airbyte. When a primary key field has the Airbyte Number type, the connector stores it as an Iceberg string rather than a double, because Iceberg identifier fields do not support the double type and deduplication correctness depends on the key being a valid identifier column.

The trade-off is that ordering and comparison on those columns become lexicographic rather than numeric. So the value 9 sorts after 10, and a range filter on the key behaves in a way that looks broken to anyone who has not been told. Deduplication stays correct, which is what the choice protects.

This matters more on Db2 than on most sources, because legacy schemas lean heavily on numeric keys, often generated sequentially over decades. Check which of your tables use one before building, and where the key is used analytically rather than only for deduplication, cast it explicitly in your downstream views.

What happens to your table and column names?

They may be modified. Like most Airbyte destinations, this one adjusts identifiers for compatibility, and when using AWS Glue it changes any non-alphanumeric character in a table name or namespace to an underscore, for Athena's benefit.

Db2 schemas are often old enough to carry naming conventions nobody would choose today, so this is worth checking rather than assuming. It is not a problem in itself, and it becomes one if somebody writes Athena queries against the names they saw in Db2 rather than the names that exist in Glue.

So check the registered names after the first sync and use those in any documentation you hand to analysts. It matters more here than on most destinations because several query engines will be reading these tables independently, each of them from the catalogue rather than from your notes.

Frequently asked questions

Does this write plain files or actual tables?

Iceberg tables. The S3 Data Lake connector is Airbyte's official support for the Iceberg protocol on S3, registered through a catalog such as AWS Glue so engines like Athena can query them directly.

Which catalogs are supported?

AWS Glue, REST, Nessie and Polaris. Glue is the natural choice when the goal is querying from Athena and the wider AWS estate.

Why does my numeric ID column sort incorrectly?

Number-typed primary keys are stored as Iceberg strings, because identifier fields do not support the double type. Ordering is therefore lexicographic, so 9 sorts after 10. Cast explicitly in downstream views.

Why did my table names change?

When using AWS Glue, the connector replaces non-alphanumeric characters in table names and namespaces with underscores for Athena compatibility. Check the registered names after the first sync.

Can I do this without writing code?

The Airbyte setup is entirely UI-driven. The AWS side needs a bucket policy, which is JSON the documentation provides, and a Glue database.

Get your Db2 data into S3

Set up the bucket and Glue database first, establish what a deletion should mean given your Db2 connector's capabilities, check which tables use numeric primary keys, and verify the registered table names from Athena after the first sync rather than trusting the names you started with.

Airbyte's connector catalog includes 600+ pre-built connectors, so the same Db2 source can feed a warehouse alongside the lake. For that pairing, see IBM Db2 to BigQuery, and for a relational source landing in the same lake, PostgreSQL to Amazon S3 with AWS Glue.

Start syncing now →