PostgreSQL to DynamoDB: How to Move Your Data

Moving PostgreSQL into DynamoDB is worth being clear about first, because it is not an analytics pipeline. DynamoDB is built for fetching an item by its key very quickly at any scale, so this exists to serve an application: a read model a service can look up per request, or a staged migration away from a relational store.

This guide covers the managed path with Airbyte. One behaviour shapes everything downstream, and it surprises people who expect their columns to become attributes: your row arrives as a single JSON blob rather than as individual fields.

PostgreSQL to DynamoDB at a glance:

Why move data from PostgreSQL to DynamoDB?

Two situations account for most of these pipelines, and neither is reporting.

The first is scale on the read path. A service handling very high request volumes wants a store that answers by key in single-digit milliseconds without a connection pool to manage, and DynamoDB does that where a single Postgres instance eventually will not.

The second is a migration in progress, where an application is moving onto DynamoDB and the pipeline keeps it current while traffic shifts. If what you actually want is analysis, send PostgreSQL to a warehouse or lakehouse instead, because DynamoDB is a poor place to aggregate anything.

What do you need before you start?

One prerequisite affects your production database and one affects what the destination can ever do:

Logical replication on PostgreSQL, plus monitoring. A replication slot and a publication, and an alert on slot lag. An unconsumed slot means PostgreSQL retains write-ahead log it cannot discard, which on a busy database fills the disk of the system serving your application.

A decision about your key design. DynamoDB items are addressed by a partition key and optionally a sort key, and those are what your application will look up by. Work out that access pattern before creating the table, because it is not something you change later.

Raised write capacity for the initial load. The default provisioned capacity is low, and a bulk load of a Postgres table will exhaust it and be throttled. Raise it before the first sync and lower it afterwards, or use on-demand capacity.

AWS credentials or an IAM role. With write access to the target table. On self-managed Airbyte running in AWS, a role attached to the workload avoids long-lived keys in the configuration.

If your organisation restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list on the PostgreSQL side before you begin.

How do you build a PostgreSQL to DynamoDB pipeline in Airbyte?

Step 1: Design the access pattern first

Write down exactly how your application will fetch items, then design the partition and sort keys to serve that. DynamoDB rewards knowing the access pattern in advance and punishes discovering it later, and because of how the data lands you cannot fall back on indexing an arbitrary field afterwards.

Step 2: Configure the PostgreSQL source

Click Sources in the left navigation, then New Source, and select Postgres, following adding a source. Supply the host, port, database, schema and credentials, and choose logical replication as the replication method with your slot and publication names, so deletions reach the destination rather than leaving orphaned items.

Step 3: Configure the DynamoDB destination

Click Destinations, then New Destination, and select DynamoDB, following adding a destination. Supply the region, table name prefix and credentials. Confirm the capacity settings on the target table now rather than during the first load, when throttling will present as a slow sync rather than a clear error.

Step 4: Choose a sync mode that is safe for a live table

Click Connections, then New connection, select your tables and a sync mode. Overwrite is destructive, so if an application is reading the table while a sync runs it will see an incomplete picture. Incremental is the safe default here, and where you do need a full reload, the pattern below is the way to do it.

Load into a new table, verify it, then repoint the application by changing the table name it reads. That way a rebuild never leaves live traffic looking at a half-populated table.

Why do your columns not become attributes?

Because the destination writes each record as a JSON document rather than mapping fields onto DynamoDB attributes individually. So a Postgres row with twenty columns does not become an item with twenty attributes, it becomes an item holding one structure containing all of them.

The consequence that bites is indexing. A global secondary index is built on an attribute, and a field buried inside a JSON document is not an attribute, so you cannot create one on it. If your application needs to query by anything other than the key, that requirement has to be satisfied by the key design rather than added afterwards.

This is why step 1 comes first. On most destinations you can model your way out of a surprise after loading; here the shape of what lands constrains what the table can ever do, so the access pattern has to be settled before the first sync rather than discovered from it.

What happens if the pipeline stops?

This deserves its own section because the consequence lands on your production database rather than on the pipeline. A logical replication slot marks the oldest position a consumer still needs, and PostgreSQL will not discard write-ahead log segments beyond it.

So if the connection is paused for a sprint, fails repeatedly over a holiday, or somebody deletes the Airbyte connection without dropping the slot, WAL accumulates. On a busy database that consumes disk quickly, and the failure mode is an outage on the system serving your users rather than a stale table.

Monitor replication slot lag alongside your other database metrics and alert on it, treating it as a database health signal rather than a pipeline one. And make dropping the slot part of your decommissioning checklist, because an abandoned slot is the version nobody is watching for.

Frequently asked questions

Can I build a secondary index on a column from Postgres?

No. Records land as a JSON document rather than individual attributes, and a global secondary index needs an attribute. Satisfy the access pattern through your key design instead.

Why is my initial load so slow?

Almost certainly write capacity throttling. The default provisioned capacity is low for a bulk load, so raise it before the first sync or use on-demand capacity.

Is overwrite mode safe against a live table?

No. It is destructive, so an application reading during a sync sees an incomplete table. Load into a new table and repoint the application once it is verified.

Why is my PostgreSQL disk filling up?

An unconsumed logical replication slot. PostgreSQL retains write-ahead log beyond the slot's position, so monitor slot lag and drop slots you no longer use.

Can I do this without writing code?

The Airbyte setup is entirely UI-driven. Your application code will need to read the JSON structure rather than individual attributes, which is worth knowing before anyone starts building against it.

Get your PostgreSQL data into DynamoDB

Settle the access pattern before the first sync, because the JSON shape means you cannot index your way out of a mistake afterwards. Raise write capacity for the initial load, load into a new table and repoint rather than overwriting in place, and monitor the replication slot as a database health metric.

Airbyte's connector catalog includes 600+ pre-built connectors, so the same PostgreSQL source can feed a warehouse alongside this one. For a document store with a richer query model, see PostgreSQL to MongoDB, and for the same destination fed by files, Parquet File to DynamoDB.

Start syncing now →