Oracle Database to Amazon S3 with AWS Glue: How to Move Your Data

Moving Oracle into Amazon S3 with AWS Glue puts relational data into an open table format that several engines can read. Oracle is an excellent database and an expensive place to keep history nobody queries daily, and an Iceberg table in your own bucket costs storage rather than licensing.

This guide covers the managed path with Airbyte. Two things shape the build, and both are quiet rather than obvious: LogMiner ignores identifiers beyond a certain length without telling you, and numeric primary keys do not stay numeric once they become Iceberg identifier fields.

Oracle Database to Amazon S3 with AWS Glue at a glance:

Why move data from Oracle Database to Amazon S3 with AWS Glue?

Two situations account for most of these pipelines.

The first is cost and history. Oracle holds years of transactional records that nobody queries weekly but everybody insists on keeping, and storage priced as part of a database licence is an expensive way to keep them. Iceberg tables on S3 hold that history cheaply while staying genuinely queryable, which is what separates this from an export to a folder of files.

The second is opening the data to more than one engine. Once tables are registered in Glue, Athena, Spark, EMR and Redshift Spectrum can all read them without another copy. The pairing is a poor fit if you simply want a warehouse and have no appetite for managing a catalog, table maintenance and compaction, because Iceberg gives you flexibility in exchange for operational work; Oracle Database to BigQuery asks far less of you.

What do you need before you start?

Four things, and two of them are audits rather than credentials:

An Oracle user, and a decision about change capture. The standard connector is cursor-based and never sees deletes. LogMiner capture sits in the enterprise connector and requires supplemental logging plus a long list of grants. The Oracle source documentation sets out what each path needs.

An audit of your identifier lengths. If you are using LogMiner, anything over 30 characters is ignored rather than rejected. Older Oracle schemas are full of descriptive names that comfortably exceed that, so list the tables and columns you intend to replicate and measure them before you trust the results.

An S3 bucket, a Glue catalog and the permissions joining them. The pipeline writes data files to the bucket and registers tables in Glue, so the credentials need both. Decide your namespace naming now, because Glue will rewrite anything it dislikes.

A list of your primary keys and their types. Numeric keys behave differently here than anywhere else you have sent this data, and knowing which tables have them tells you where to expect it.

If your database restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.

How do you build an Oracle Database to Amazon S3 with AWS Glue pipeline in Airbyte?

Step 1: Audit your names and keys before configuring anything

Two lists, twenty minutes. First, every table and column name you plan to replicate, with its length, so you know which ones LogMiner will quietly skip. Second, every primary key and its Oracle type, so you know which tables will have numeric keys stored as strings at the other end. Both problems are cheap to plan around and unpleasant to discover from a reconciliation that does not balance.

Step 2: Configure the Oracle source

Click Sources in the left navigation, then New Source, and select Oracle, following adding a source. Supply the host, port, service name or SID, and credentials. Leave out any table using user-defined types such as VARRAY if you are relying on LogMiner, since those cannot replicate through change capture and are better handled as a separate full refresh.

Step 3: Configure the S3 Data Lake destination

Click Destinations, then New Destination, and select the S3 Data Lake, following adding a destination. Choose AWS Glue as the catalog and supply the bucket, region and credentials. Keep namespace and table names alphanumeric with underscores from the start, because Glue converts anything else and a name you did not choose is a name nobody will guess.

Step 4: Create the connection and understand how upserts land

Click Connections, then New connection, select your streams and a sync mode. An upsert in Iceberg is an equality delete followed by an insert rather than an update in place, which is worth knowing because frequent small syncs produce many delete files and gradually slow reads until compaction runs.

Plan table maintenance now rather than later. Compaction and snapshot expiry are yours to schedule, and an Iceberg table left entirely alone for a year is slower and larger than it needs to be.

What does LogMiner quietly leave behind?

Anything named with more than 30 characters. LogMiner does not reject these or warn about them; it ignores them. A table whose name runs long simply does not produce change records, and the sync completes successfully having replicated everything else, which is the worst possible combination for noticing.

Long names are not rare in Oracle estates. Schemas built by people who valued clarity over brevity, or generated by an application that concatenates a module prefix onto a business noun, produce exactly this. The limit catches established systems more often than new ones, which means it catches the systems whose data you most want.

User-defined types are the other gap. Columns using constructs such as VARRAY cannot replicate through change capture at all, so a table containing one needs a different approach entirely, usually a full refresh on its own schedule or a view that flattens the structure into ordinary columns. Identify both categories during the audit and decide deliberately, rather than assuming the absence of errors means the absence of problems.

Why does 9 sort after 10 in your Iceberg table?

Because Iceberg identifier fields do not support the double type, so a numeric primary key is stored as a string. Strings sort lexicographically, which puts 10 before 9 and 100 before 20, and every Oracle sequence-generated key you have ever relied on being ordered now is not.

Nothing is lost and nothing errors, which is precisely why this bites. The values are all there and each one is correct. It is ordering and range comparison that break, so a query asking for identifiers above a threshold, or taking the highest key to find the newest record, returns confident nonsense.

Deal with it by not ordering on the key. Use a timestamp column for anything chronological, which is more honest anyway since a sequence only approximates insertion order. If a consumer genuinely needs numeric comparison, cast in the query or expose a view that casts once, and make sure whoever writes against these tables knows before they write rather than after their results look odd.

Frequently asked questions

Why is one table producing no changes?

Check the length of its name. LogMiner ignores identifiers over 30 characters silently, so the sync succeeds while that table contributes nothing at all.

Why did my table name change in Glue?

Glue replaces non-alphanumeric characters in table and namespace names with underscores. Choose names that survive that conversion so the catalog matches what you expected to see.

Can I keep numeric keys numeric?

Not as identifier fields, which do not support the double type and therefore store them as strings. Order on a timestamp instead, or cast explicitly where numeric comparison is genuinely required.

Why are reads getting slower over time?

Upserts are equality deletes plus inserts, so frequent syncs accumulate delete files. Schedule compaction and snapshot expiry, since neither happens on its own.

Can I do this without writing code?

The pipeline, yes. Table maintenance is not, and a lake nobody maintains degrades quietly, so budget for compaction jobs alongside the pipeline itself.

Get your Oracle Database data into Amazon S3 with AWS Glue

Audit names and keys first, because both failure modes here are silent. Long identifiers vanish from LogMiner without error, and numeric keys become strings that sort in an order nobody expects. Keep names alphanumeric so Glue leaves them alone, order on timestamps rather than keys, and schedule compaction from the beginning so the table stays as fast as it was on day one.

Airbyte's connector catalog includes 600+ pre-built connectors, so relational history can live in open formats rather than licensed storage. For the same source into a lakehouse, see Oracle Database to Databricks, and for another relational source into the same destination, PostgreSQL to Amazon S3 with AWS Glue.

Start syncing now →