Azureblobstorage to PostgreSQL: How to Move Your Data

Moving Azure Blob Storage into PostgreSQL turns files somebody drops in a container into rows an application can query. Blob storage holds things; it does not join, filter or aggregate them, so every question about what is in those files currently requires downloading and parsing them first.

This guide covers the managed path with Airbyte. Two things shape the build: one error message covers two completely different problems, and the PostgreSQL destination changed how it writes in version 3.0.0.

Azureblobstorage to PostgreSQL at a glance:

Why move data from Azureblobstorage to PostgreSQL?

Two situations account for most of these pipelines.

The first is that a container has become an integration point. A partner uploads a file every night, an internal system exports to blob storage because that was easiest, and now several things need the contents as data rather than as files. A database is where that becomes queryable, joinable and available to an application.

The second is giving an application a working copy of reference data that arrives as files. The limit is volume. Guidance for this destination is around ten gigabytes, and containers have a way of accumulating years of daily exports without anybody noticing, so if you are loading the whole history rather than recent files, Azureblobstorage to BigQuery handles that scale more comfortably.

What do you need before you start?

Four things, and the second catches almost everybody once:

Credentials, by one of two routes. Either a storage account key, which is simple and shares one secret across everything, or a service principal with a tenant, client and secret, which is tidier for anything long-lived. The Azure Blob Storage source documentation covers both.

The Storage Blob Data Reader role, if you use a service principal. Owner and Contributor let you manage the storage account and do not let you read what is inside it. Those are separate planes in Azure, and an administrator granting Owner in good faith has granted nothing useful to this pipeline.

A glob pattern you have verified matches something. Include the subfolder path, and remember that a double asterisk matches recursively. Check it against the container before configuring, because a glob matching nothing produces the same error as bad credentials.

A PostgreSQL database, and knowledge of which connector version you are on. Version 3.0.0 changed the write path substantially. A new pipeline gets the current behaviour; an existing one being upgraded has a migration to read first.

If your database restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.

How do you build an Azureblobstorage to PostgreSQL pipeline in Airbyte?

Step 1: Prove the role and the glob separately

Because one error message covers both, establish each independently before you configure anything. Confirm the service principal holds Storage Blob Data Reader on the container, then list the blobs your glob should match using the Azure portal or command line. Two minutes each, and between them they account for the overwhelming majority of failed first syncs on this connector.

Step 2: Configure the Azure Blob Storage source

Click Sources in the left navigation, then New Source, and select Azure Blob Storage, following adding a source. Supply the account name, credentials, container and glob, then choose your delivery method: replicate records parses the files into rows, while copy raw files moves them across untouched. For a database destination, parsed records is almost always what you want.

Step 3: Configure the PostgreSQL destination

Click Destinations, then New Destination, and select PostgreSQL, following adding a destination. Supply the host, port, database and credentials. From version 3.0.0 the connector uses Direct Load, so tables are written without intermediate raw tables, and if you are upgrading an existing connection rather than starting fresh, read the Postgres migration guide first.

Step 4: Create the connection and split streams by file shape

Click Connections, then New connection, select your streams and a sync mode. Incremental works from file creation or modification time, which suits a container receiving new files rather than edited ones. If the container holds files of different shapes, give each shape its own glob and its own stream rather than hoping they reconcile.

Watch the total volume against the ten gigabyte guidance, especially if your glob reaches back through years of daily uploads rather than recent ones.

Why do correct credentials still fail?

Two reasons, and one message covering both. The first is the Azure role model: Owner and Contributor are management-plane roles that let somebody administer the storage account without granting any ability to read the blobs inside it. Data access needs Storage Blob Data Reader specifically, which surprises people who reasonably assume Owner outranks everything.

The second is that a glob matching no files produces the same credentials error. Nothing distinguishes it, so somebody with a correctly permissioned service principal and a slightly wrong path spends an afternoon regenerating secrets that were never the problem, which is a remarkably effective way to lose confidence in a pipeline that is almost working.

This is why proving each independently is worth the effort. Confirm the role assignment in Azure, then confirm the glob returns files, and you have eliminated both causes before the connector has a chance to blur them together. The most common mistakes are forgetting the subfolder path in the pattern and assuming a single asterisk descends into subdirectories when recursion needs a double one.

What changed in the PostgreSQL destination at version 3.0.0?

The write path. Direct Load removed the intermediate raw tables the connector previously maintained, so data lands in your tables without a staging layer sitting alongside them. Less clutter in the database, one fewer thing to explain to whoever audits the schema, and a simpler mental model of what the pipeline is doing.

It also adds metadata columns, including a JSONB column recording per-row changes in typing. That column is genuinely useful when a file arrives with a value that did not match the expected type, because it tells you which rows were affected rather than leaving you to infer it, and it is worth knowing about before somebody asks what the extra column is for.

For a new pipeline none of this needs action, since you get the current behaviour by default. For an existing connection being upgraded, the migration guide matters and should be read before the upgrade rather than during the incident it causes. Either way, tell whoever owns the database that the metadata columns belong to the pipeline and should be left alone, and expose views selecting the columns applications actually need.

Frequently asked questions

My service principal is an Owner. Why can it not read the files?

Owner is a management-plane role covering administration of the account, not access to its contents. Assign Storage Blob Data Reader for data access.

I am certain my credentials are right and it still says otherwise.

Check the glob. A pattern matching no files raises the same error as a credential problem, and a missing subfolder path is the usual cause.

Should I replicate records or copy raw files?

Replicate records when the destination is a database, since that parses the files into rows. Copying raw files suits archival, which a PostgreSQL table is not.

What is the extra JSONB column?

Metadata added from version 3.0.0, recording typing changes per row. Leave it in place and select around it in views rather than dropping it.

Can I do this without writing code?

Yes, the setup is entirely UI-driven. The Azure role assignment is portal configuration, and views over the loaded tables are optional but make life easier for anything reading them.

Get your Azureblobstorage data into PostgreSQL

Prove the role and the glob separately before configuring anything, because one error message hides two unrelated causes and people routinely fix the wrong one. Choose replicate records for a database destination, split differently shaped files into separate streams, and keep an eye on total volume against the ten gigabyte guidance. If you are upgrading rather than starting fresh, read the migration guide before the upgrade rather than after it.

Airbyte's connector catalog includes 600+ pre-built connectors, so files arriving in cloud storage can become queryable data without a parsing script. For the other Azure storage service into the same destination, see Azure Table Storage to PostgreSQL, and for another file-based source into the same destination, SFTP to PostgreSQL.

Start syncing now →