MongoDB to Amazon S3 with AWS Glue: How to Move Your Data

Moving MongoDB into Amazon S3 with AWS Glue gives application data a home that tolerates change. Documents evolve as the product does, which is the whole appeal of MongoDB and the reason a strict warehouse fights you every time a developer adds a field.

This guide covers the managed path with Airbyte. Two things shape the build: Iceberg needs a declared schema and identifier fields that MongoDB does not naturally provide, and once you have satisfied that, a lake handles evolving documents better than anywhere else you could send them.

MongoDB to Amazon S3 with AWS Glue at a glance:

Why move data from MongoDB to Amazon S3 with AWS Glue?

Two situations account for most of these pipelines.

The first is volume and cost. Application databases accumulate enormous collections of events, sessions and records that are interesting in aggregate and rarely queried individually, and keeping all of that in a warehouse is an expensive way to store something nobody reads daily. Iceberg on S3 costs storage and stays queryable, which is the combination this problem wants.

The second is feeding transformation and modelling work with several engines rather than one. The counterargument is operational: a lake means catalogs, compaction and snapshot expiry to look after. If your documents are stable and your team would rather not run table maintenance, MongoDB to Snowflake is simpler and perfectly capable.

What do you need before you start?

Four things, and two of them are decisions Iceberg forces on a schemaless source:

A replica set and a read user. Change capture works through change streams, which exist only on a replica set, and it is what makes deletes reach the lake. The MongoDB source documentation covers the permissions involved.

A schema mode decision. Schema-enforced samples documents and produces typed columns. Schemaless passes each document across whole, which Iceberg will store and which leaves you with one opaque value where a table should be.

A view on what your identifier field will be. Iceberg identifier fields are stored as strings, which is harmless for a MongoDB object identifier and consequential if you nominate a numeric field instead.

An S3 bucket and a Glue catalog. Credentials need to write files to one and register tables in the other. Keep collection names alphanumeric where you can, since Glue converts anything else into underscores.

If your database restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.

How do you build a MongoDB to Amazon S3 with AWS Glue pipeline in Airbyte?

Step 1: Settle schema mode and your identifier field

Choose schema-enforced, then decide which field identifies a record. The MongoDB object identifier is the natural choice and behaves well because it is already a string. If you nominate a numeric field from your application instead, it will be stored as a string and sort lexicographically, which puts 10 before 9 and breaks any range comparison somebody writes against it. Deciding now is a conversation; deciding later is a rebuild.

Step 2: Configure the MongoDB source

Click Sources in the left navigation, then New Source, and select MongoDB, following adding a source. Supply the connection string, database and credentials, and select schema-enforced mode. Review what the sampling inferred, since fields present on only a fraction of documents may be missing from it, and note which ones so you can watch for them later.

Step 3: Configure the S3 Data Lake destination

Click Destinations, then New Destination, and select the S3 Data Lake, following adding a destination. Choose AWS Glue as the catalog and supply the bucket, region and credentials. Other catalogs exist, including REST, Nessie and Polaris, and Glue is the sensible default when the rest of your estate is already in AWS.

Step 4: Create the connection and use change capture

Click Connections, then New connection, select your collections and a sync mode. Change capture handles deletes, which matters if anything here is subject to a deletion request. Change streams read from the oplog, so a pipeline paused longer than your oplog window cannot resume where it stopped and needs a fresh snapshot.

Schedule compaction and snapshot expiry, since an Iceberg table left alone accumulates small files and delete files that gradually slow every read.

What does Iceberg need that MongoDB does not provide?

A declared schema and a stable identifier, neither of which a document store maintains for you. MongoDB is content for two documents in one collection to look nothing alike, and Iceberg needs to know what the columns are before it can write a file. That gap is bridged by schema-enforced mode, which samples documents and infers the structure.

Schemaless mode technically works and defeats the purpose. Each document arrives whole, so the table holds one opaque field, and the columnar layout, predicate pushdown and column pruning that make a lake worth querying have nothing to operate on. You have paid for the format and kept none of its benefits.

The identifier is the second requirement and the one with a trap in it. Iceberg identifier fields do not support the double type, so a numeric key is stored as a string and sorts lexicographically, which quietly breaks ordering and range comparisons while every value remains correct. MongoDB's own object identifier avoids this entirely by already being a string, which is one more reason to prefer it over an application-generated number.

Why does a lake suit evolving documents better than a warehouse?

Because Iceberg supports schema evolution as a normal operation rather than an event. A developer adding a field to a document next quarter is routine in MongoDB, and in a strict columnar warehouse it becomes a change request, a migration and a conversation. Here the table gains a column and older files remain valid, with nothing rewritten and nothing broken.

That tolerance is the strongest argument for this particular pairing. A schemaless source and a format that expects its schema to change are well matched, whereas sending the same collection somewhere that wants stable columns means fighting the source's nature every time the product moves. If your documents genuinely churn, this is the destination that minds least.

Two caveats keep it honest. The connector still infers schema by sampling, so a genuinely new field appears only once enough documents carry it, which means evolution is discovered rather than announced and is worth reviewing periodically. And tolerance is not an excuse to skip maintenance: compaction and snapshot expiry are still yours, and a table absorbing constant change accumulates files faster than a quiet one does.

Frequently asked questions

Should I use schemaless mode for a schemaless source?

No. It produces one opaque field per document, which removes the columnar benefits that make an Iceberg table worth querying. Use schema-enforced.

Which field should identify a record?

MongoDB's object identifier, which is already a string and therefore avoids the lexicographic sorting problem that a numeric application key would introduce.

What happens when developers add a field?

Iceberg handles schema evolution, so the table can gain a column without existing files being rewritten. Sampling has to see the field often enough first, so review the schema periodically.

Do deletes reach the lake?

With change capture, yes, which needs a replica set. That matters where deletion requests have to propagate rather than leaving records in the lake indefinitely.

Can I do this without writing code?

The pipeline, yes. Compaction and snapshot expiry are jobs you schedule yourself, and a lake without them degrades quietly rather than failing.

Get your MongoDB data into Amazon S3 with AWS Glue

Use schema-enforced mode, since a single opaque field wastes the format entirely, and prefer MongoDB's own object identifier so you never meet the lexicographic sorting problem. Use change capture so deletes propagate. Then enjoy the part that makes this pairing worth choosing: documents that evolve are a normal event here rather than a migration, provided you review the inferred schema occasionally and keep compaction running.

Airbyte's connector catalog includes 600+ pre-built connectors, so application data can live in open formats without a bespoke export. For the same source into a lakehouse, see MongoDB to Databricks, and for a relational source into the same destination, IBM Db2 to Amazon S3 with AWS Glue.

Start syncing now →