Azure Table Storage to PostgreSQL: How to Move Your Data

Moving Azure Table Storage into PostgreSQL puts non-relational rows somewhere you can join and aggregate them. Table Storage is built for fast key-based access at scale and answers almost nothing else, so anything shaped like how many, grouped by, or joined to needs the data elsewhere.

There is a mismatch to plan for, though. Table Storage has no schema, and there is no efficient way to read one, so the connector does not try: every stream lands with the same generic shape, a single data property holding all the properties of each row. A relational destination is the least forgiving place for that to arrive.

Azure Table Storage to PostgreSQL at a glance:

Why move data from Azure Table Storage to PostgreSQL?

Two situations account for most of these pipelines.

The first is querying by something other than the key. Joins, aggregations and ad hoc filters are what Postgres does well and Table Storage does not, and an internal tool or reporting layer that already speaks Postgres gets those for free once the data is there.

The second is joining application data to relational data that already lives in that database. If you are doing large-scale analytics across a very big table, a lakehouse is the better destination, both for volume and because it handles the semi-structured landing shape more gracefully.

What do you need before you start?

One of these will disappoint a security-minded colleague and one determines whether the result is usable:

Your storage account name and access key. Found in the Azure portal under your storage account. Airbyte recommends creating a restricted key specifically for its access so you control what it can reach. The Azure Table Storage source documentation covers where to find them.

Acceptance that shared access key authentication is not supported. The connector does not support it yet, so if your organisation's policy is to issue scoped, time-limited shared access signatures rather than account keys, have that conversation before you build rather than after.

Knowledge of what your rows actually contain. Since no schema is read and no type mappings are applied, nothing in the pipeline will tell you. Ask whoever writes to these tables what the properties are and what types they hold, because that conversation becomes your casting layer.

A PostgreSQL user with write access. On the target schema. If you already run a Postgres destination on an older connector version, read the migration guide before upgrading, because downstream models referencing raw tables will need updating.

If your organisation restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list on the PostgreSQL side before you begin.

How do you build an Azure Table Storage to PostgreSQL pipeline in Airbyte?

Step 1: Document what each table holds

Sit down with whoever owns the application writing to these tables and list the properties each one uses, including optional ones and any that changed type over the years. Nothing in this pipeline discovers that for you, so this list is the specification for your casting layer and the only documentation the dataset will ever have.

Step 2: Configure the Azure Table Storage source

Click Sources in the left navigation, then New Source, and select Azure Table Storage, following adding a source. Supply the storage account name and the restricted access key you created, then run the connection test.

Step 3: Configure the PostgreSQL destination

Click Destinations, then New Destination, and select Postgres, following adding a destination. From version 3.0.0 the Postgres destination uses Direct Load architecture, writing records straight to final tables with no intermediate raw tables, and adds its own metadata columns including a JSONB column recording per-row typing changes.

Step 4: Select tables and a sync mode

Click Connections, then New connection, select the tables from step 1 and choose a sync mode for each. Both full refresh and incremental are available. Rate limits are unlikely to constrain you here, so schedule against how often the data actually changes.

Run one table first and look at the result before extending. The generic schema means every table looks alike, so confirming that the data object contains what you expect is worth doing once per table shape rather than assuming.

Why is every stream the same shape?

Because Table Storage does not have a schema to read, and there is no efficient way to derive one. Rather than sampling rows and guessing, the connector uses a data property holding all the properties for any given row, which produces the same generic schema for every stream.

That is an honest design rather than a limitation, and it avoids a real failure mode: a connector that sampled rows to infer a schema would miss properties present on only a fraction of them. Nothing is dropped here, because nothing is inferred.

The cost lands on you, and it lands harder on a relational destination than on a lakehouse. Postgres wants typed columns, and no explicit data type mappings are applied, so a number written by your application is not guaranteed to arrive as a number you can aggregate. The landed table is a staging table, not something to point a query at.

How do you turn a data object into real columns?

With a view, and the good news is that Postgres is well equipped for it. Its JSON operators let you extract properties out of the data object and cast them in the same expression, so a modelled view sitting over the landed table is short to write and does the whole job.

Anchor that view on Table Storage's own keys. Every entity carries a partition key grouping related entities and a row key identifying it within that partition, and together they are the natural unique identifier. The partition key is usually meaningful too, since applications choose it to group entities read together, which makes it worth promoting to a proper column.

Decide per column how a bad value should behave. A property your application always writes as a number should fail loudly if it ever is not, because that indicates a change upstream you want to hear about. An optional free-text field should not. Keeping the landed table underneath means either way you can reprocess once you know.

Frequently asked questions

Why do all my tables have the same schema?

Because Table Storage has no schema to read efficiently, so the connector uses a generic one where a data property holds all properties for each row. Extract and cast it in a view.

Can I use a shared access signature instead of the account key?

Not currently. Shared access key authentication is not supported by this connector yet, so plan on a restricted storage account key.

Are data types converted for me?

No. The connector applies no explicit data type mappings, so casting values into the types Postgres should hold is work for your downstream view.

Is PostgreSQL big enough for this data?

For a focused set of tables, comfortably. Airbyte recommends relational destinations for roughly 10GB or less, and Table Storage is often chosen precisely because data grows past that, so check before committing.

Can I do this without writing code?

The Airbyte setup is entirely UI-driven. Extracting the data object into typed columns is SQL, and on this destination it is closer to a requirement than a nicety.

Get your Azure Table Storage data into PostgreSQL

Raise the authentication constraint early, document what each table holds because nothing else will, and treat the landed table as staging. Then write the view that extracts and casts, promote the partition key to a real column, and check your volume against the guidance before committing.

Airbyte's connector catalog includes 600+ pre-built connectors, so application data can reach a database and a lakehouse at once. For the version that handles the generic shape more gracefully, see Azure Table Storage to Databricks, and for another key-value store landing in Postgres, DynamoDB to PostgreSQL.

Start syncing now →