DynamoDB to MongoDB: How to Move Your Data

Moving DynamoDB into MongoDB is a migration between two schemaless stores, which sounds like it should be the easy case and is not. Both hold documents, both tolerate items that do not match each other, and both perform well on lookups. What does not transfer is the thing that made DynamoDB fast: its key design.

This guide covers the managed path with Airbyte. Two things shape the build: the connector works out your schema by sampling items rather than reading a definition, and DynamoDB's access pattern has to be redesigned as MongoDB indexes rather than copied across.

DynamoDB to MongoDB at a glance:

Why move data from DynamoDB to MongoDB?

Two situations account for most of these pipelines.

The first is escaping an access pattern. DynamoDB rewards you enormously for queries that match its key design and punishes everything else, so a table that was modelled two years ago around one question becomes awkward when the product starts asking different ones. MongoDB's secondary indexes are far more forgiving of a query somebody thought of late.

The second is consolidation, pulling an application's data out of a managed service and into the document store the rest of the estate already runs on. Neither reason is analytical, and that is the boundary worth respecting: both of these databases are poor places to aggregate. If the destination is meant to answer reporting questions, DynamoDB to PostgreSQL will serve you better than swapping one document store for another.

What do you need before you start?

Four things, and two of them exist because this source has no schema to read:

AWS credentials, or a deliberate absence of them. Supply an access key and secret, or omit both and let the default credentials chain resolve them from the environment. The chain is the tidier option where it is available, since it keeps long-lived keys out of a configuration screen. The DynamoDB source documentation covers both.

A cursor field that meets the rules. It has to be a top-level attribute, a string or an integer, formatted as ISO 8601 or an epoch timestamp, and present on every item. Blank is not tolerated. Plenty of tables have no attribute that qualifies, which is a thing to discover now rather than halfway through configuring a connection.

A list of attributes that sampling might miss. Ask the application team which attributes appear on only some items, particularly ones added recently or set by a rare code path. These are exactly what a sample of a thousand items can fail to notice.

A MongoDB cluster and a fresh index plan. Fresh is the operative word. Your DynamoDB partition and sort keys describe how that database was organised, not how MongoDB will be, and the pipeline creates no indexes at all.

If your MongoDB cluster restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.

How do you build a DynamoDB to MongoDB pipeline in Airbyte?

Step 1: Find a cursor field, or accept full refresh

Check your items for a top-level timestamp that every one of them carries, in ISO 8601 or epoch form, that the application reliably updates on write. If it exists, incremental works. If it does not, you are running full refresh on every sync, which is fine for a small table and expensive for a large one, and either way it is better to know before you design the schedule than to discover it from a bill.

Step 2: Configure the DynamoDB source

Click Sources in the left navigation, then New Source, and select DynamoDB, following adding a source. Supply the region and credentials, or leave the keys empty for the default chain. Then read the discovered schema properly rather than clicking past it, because it is a guess drawn from a sample and it is the only description of your data anything downstream will have.

Step 3: Configure the MongoDB destination

Click Destinations, then New Destination, and select MongoDB, following adding a destination. Supply the connection string, database and credentials. Documents arrive with metadata fields the destination adds, so anything reading them should ask for the fields it wants by name rather than treating a document as exactly what DynamoDB held.

Step 4: Create the connection and watch the read capacity

Click Connections, then New connection, select your streams and a sync mode. Remember that reading DynamoDB consumes read capacity, and a full refresh of a large table on a provisioned table competes with the application using it. Schedule accordingly, and check the first sync's effect on your table metrics before settling on a frequency.

Then build your MongoDB indexes. Nothing in the pipeline does this, and a collection without them serves every query by scanning, which is not the experience anybody expects after migrating away from a database famous for speed.

How does the connector know what your items look like?

It guesses, by scanning at least a thousand items and describing what it finds. DynamoDB holds no schema to consult, so there is no alternative, but the result is a description of a sample rather than a definition of your data. Everything downstream then treats that guess as authoritative.

Two things slip through. Sparse attributes, present on a small fraction of items, may not appear in the sample at all, so an attribute your application sets in one uncommon case is simply absent from the schema. And the sample assumes consistent types, so an attribute that is a number on most items and a string on a few hundred old ones looks unambiguous when it is not.

The defence is human rather than technical. Ask the people who write to this table what the unusual items look like, and check the discovered schema against that list rather than against your expectations. A table that has been through three product iterations almost always contains a generation of items nobody remembers, and finding them now is considerably cheaper than explaining later why a field is missing for exactly the customers who complained.

Why doesn't your DynamoDB key design carry over?

Because a partition key and a sort key are not an index, they are how DynamoDB physically distributes and orders your data. They arrive in MongoDB as ordinary fields on ordinary documents, carrying none of the properties that made them fast. A collection that looks identical to the table performs nothing like it until you say otherwise.

This is an opportunity as much as a chore, and it is often the reason for the migration. DynamoDB's key choice is effectively permanent, which is why teams end up with composite keys encoding three concepts into one string. MongoDB lets you index the fields your queries actually use, so the sensible move is to model the indexes on the questions you ask today rather than reproducing a structure designed for the questions you asked in 2023.

One consequence of the migration deserves saying plainly: deletes do not arrive. The connector does not read DynamoDB Streams, so an item removed from the table stays in MongoDB indefinitely. If the source deletes anything, whether for a user request or a tidy-up job, you need your own mechanism for propagating that, and the collection is otherwise an append-only copy that slowly diverges from the truth.

Frequently asked questions

Will deleted items be removed from MongoDB?

No. DynamoDB Streams are not read, so there is no way for the pipeline to observe a deletion. Plan your own process if removals need to propagate, particularly where deletion is a compliance obligation.

Why is an attribute missing from the schema?

Because the schema comes from a sample of at least a thousand items and sparse attributes can fall outside it. Check the discovered schema against what the application actually writes rather than trusting the sample.

My table has no suitable cursor. What now?

Full refresh, or add a top-level timestamp to the items going forward. The cursor must be top level, a string or integer, ISO 8601 or epoch, and never blank, which rules out a lot of real tables.

Should I recreate my partition and sort keys as indexes?

Only where they match a query you still run. They encoded a DynamoDB access pattern, and MongoDB's flexibility is the usual reason for moving, so index the queries you have now rather than the ones the old key design forced on you.

Can I do this without writing code?

The pipeline, yes, and it is a short setup. Creating indexes and handling deletes are yours, and neither is optional if the collection is going to serve an application.

Get your DynamoDB data into MongoDB

Check for a qualifying cursor before anything else, since its absence turns every sync into a full table read. Treat the discovered schema as a sample rather than a specification and verify it with the people who write to the table. Then design MongoDB indexes around the queries you actually have, rather than transplanting a key design you were probably trying to escape, and decide how deletes will be handled before the collection is old enough for the divergence to matter.

Airbyte's connector catalog includes 600+ pre-built connectors, so application data can move between stores without a bespoke migration script. For the same source into an analytical database, see DynamoDB to ClickHouse, and for another schemaless source into the same destination, Firebase Realtime Database to MongoDB.

Start syncing now →