Azure Table Storage to DynamoDB: How to Move Your Data

Moving Azure Table Storage into DynamoDB is usually a migration rather than an analysis project. Both are key-value stores built for fast lookups at scale, so this is about changing cloud rather than changing what the data does, and the good news is that the two share a key model almost exactly.

This guide covers the managed path with Airbyte. Two things shape the build: the source hands every row across as a single generic property, and the destination cannot index anything inside a record, which together make your key design the only access path you will have.

Azure Table Storage to DynamoDB at a glance:

Why move data from Azure Table Storage to DynamoDB?

Two situations account for most of these pipelines.

The first is consolidation onto AWS. An application moving cloud needs its key-value store to move with it, and the two products are close enough conceptually that the data model survives the journey largely intact, which is not true of most migrations.

The second is putting the data beside AWS services that will consume it. What this pairing does not do is make the data analysable, since neither store is any good at aggregation and the contents arrive unindexable. If somebody wants to query what is inside these rows rather than fetch them by key, Azure Table Storage to Snowflake gives you typed columns instead.

What do you need before you start?

Four things, and the last one cannot be changed once the table exists:

A storage account key, not a shared access signature. Shared access signatures are not supported here, which surprises organisations whose policy is to avoid account keys precisely because they grant so much. Raise it with whoever governs Azure credentials early. The Azure Table Storage source documentation sets out the configuration.

AWS credentials and a view on write capacity. Default write capacity is low and a bulk load will exceed it. Throttling does not announce itself as an error; the sync simply appears inexplicably slow while everything reports as healthy.

A list of the queries your application makes. Since nothing inside a record can be indexed, every access route has to go through the keys. Knowing what the application asks for is what tells you whether the obvious key design will actually serve it.

A decided partition and sort key. This is the irreversible one. DynamoDB's access pattern is fixed when the table is created, and changing your mind means a new table and another load.

If your storage account restricts traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.

How do you build an Azure Table Storage to DynamoDB pipeline in Airbyte?

Step 1: Map Azure's keys onto DynamoDB's keys

Azure Table Storage identifies a row by a partition key and a row key, and DynamoDB identifies an item by a partition key and a sort key. That correspondence is the most useful thing about this pairing, and the default of carrying it across directly is usually right. Confirm it against your application's queries rather than assuming, because this is the one decision here you cannot revisit.

Step 2: Configure the Azure Table Storage source

Click Sources in the left navigation, then New Source, and select Azure Table Storage, following adding a source. Supply the storage account name and access key. The discovered schema will look identical for every table you select, which is expected rather than a fault, for reasons covered below.

Step 3: Configure the DynamoDB destination

Click Destinations, then New Destination, and select DynamoDB, following adding a destination. Supply the region, credentials and table configuration. Raise write capacity before the first load if the table is provisioned rather than on-demand, and lower it again once the backfill has finished.

Step 4: Create the connection and never overwrite in place

Click Connections, then New connection, select your streams and a sync mode. Overwrite is destructive here, so where an application is already reading the table, load into a new one and repoint it once the load completes. A failed overwrite against a live table leaves consumers with nothing rather than with stale data.

If this is a migration rather than an ongoing copy, plan the cutover as a separate exercise, since a pipeline that keeps both stores in step is a different thing from one that moves the data once.

Why does every stream look identical?

Because the connector uses one generic schema for everything. Whatever properties a row holds arrive inside a single data property, and that shape is the same for every table in the account. Azure Table Storage does not enforce a schema across rows, so there is nothing consistent for the connector to describe, and this is the honest response to that.

No type mappings are applied either, so nothing arrives typed. A number in Azure is not delivered as a number for anything downstream to rely on, which matters a great deal if your destination were a warehouse and matters much less here, since DynamoDB was never going to compute anything with it anyway.

The practical consequence is that whatever reads these records does the interpreting. Your application already knows what a row contains, which is why this pairing works for a migration and fails for analysis. Record what each table's rows actually hold somewhere outside the data, because the schema will not tell anybody and the person maintaining this in two years will have no other source.

Why are your keys the only access path you get?

Because the record lands in DynamoDB as a single JSON blob rather than as a set of attributes. The fields are all present and readable once an item is retrieved, and DynamoDB cannot see them as structure, which means a global secondary index cannot be built on anything inside. Every query has to go through the partition and sort key.

That compounds with the source, because Azure already handed you a single generic property and the destination wraps it again. Anybody planning to index on a status field or a timestamp inside the row is planning something that cannot be implemented against this table, and it is much better to discover that now than after the migration.

Which is why the key correspondence matters so much. Azure's partition key and row key carry across naturally, and if your application was already fetching by those, everything continues to work. If it was relying on Azure Table Storage query filters against row properties, those have no equivalent here, and you will need either a composite sort key encoding what you filter on or a second table keyed differently. Work that out before the table exists rather than after.

Frequently asked questions

Can I authenticate with a shared access signature?

No. Only the storage account key is supported, which is worth raising with whoever governs your Azure credentials before the project depends on it.

Why does the schema look the same for every table?

The connector uses one generic schema with all row properties inside a single data property. Azure Table Storage does not enforce a schema across rows, so there is nothing consistent to describe.

Can I index a field inside the record?

No. Records arrive as a blob, so a global secondary index has no real attribute to build on. Plan every access route around the partition and sort key.

Why is the first load so slow?

Probably write throttling, since default capacity is low and bulk loads exceed it. It presents as slowness rather than failure, so raise capacity for the backfill and lower it afterwards.

Can I do this without writing code?

The pipeline, yes, and the configuration is short. The consuming application is yours, and it will need to parse the record itself since DynamoDB cannot do that for it.

Get your Azure Table Storage data into DynamoDB

Confirm you can use an account key, since shared access signatures are not an option. Map Azure's partition and row key onto DynamoDB's partition and sort key, which is the natural correspondence and the one decision you cannot undo. Accept that nothing inside a record is indexable, so any filter your application relied on in Azure needs a key design here instead. Raise write capacity for the load, and never overwrite a table something is reading.

Airbyte's connector catalog includes 600+ pre-built connectors, so application data can change cloud without a bespoke migration script. For the same source into an operational database, see Azure Table Storage to PostgreSQL, and for another schemaless source into the same destination, Firebase Realtime Database to DynamoDB.

Start syncing now →