Azureblobstorage to BigQuery: How to Move Your Data
Move Azure Blob Storage into BigQuery with Airbyte. Why Owner does not grant blob access, the Storage Blob Data Reader role, and choosing the delivery method.
Moving Azure Blob Storage into BigQuery makes files queryable that are otherwise invisible to everybody. Partner extracts, exports from another system and output from a job all land in a container and sit there, useful to nobody, until something loads them somewhere analysts can reach.
This guide covers the managed path with Airbyte. The thing that costs people the most time is an Azure permissions distinction: an identity can own the storage account and still be unable to read a single blob, and the resulting error does not explain why.
Azure Blob Storage to BigQuery at a glance:
Why move data from Azure Blob Storage to BigQuery?
Two situations account for most of these pipelines.
The first is a cross-cloud reality. Plenty of organisations have systems producing files into Azure while their analytical estate runs on Google Cloud, and this pipeline is the bridge rather than a choice anybody made deliberately.
The second is making a file feed joinable. A partner's product list or a vendor extract is only useful next to your own data, and a warehouse is where that join happens. If your analysis already runs on Azure Synapse or Fabric, moving the files elsewhere is work you may not need.
What do you need before you start?
The Azure side has a specific role requirement that is easy to get wrong:
Credentials, by one of two methods. Either a storage account key, which is simple and shares one secret for the whole account, or a service principal using client credentials with a tenant ID, client ID and client secret. The Azure Blob Storage source documentation covers both.
The Storage Blob Data Reader role, if you use a service principal. Assigned to that principal on the storage account. This is the specific role to search for when adding the assignment, and it is not implied by the broader roles people usually already hold.
Your storage account name and container name. Plus a decision about delivery method: replicate records parses files into rows, while copy raw files transfers them unparsed. For a warehouse destination you want the first.
A BigQuery service account and dataset. With the BigQuery User and BigQuery Data Editor roles. Set the dataset location to match whatever you intend to join against, since it cannot be changed after creation.
Finally, find out whether files are ever rewritten in place, because incremental sync works from file modification time and that habit decides what gets re-read.
How do you build an Azure Blob Storage to BigQuery pipeline in Airbyte?
Step 1: Assign the data plane role
If you are using a service principal, go to the storage account's access control, add a role assignment, search for Storage Blob Data Reader by name and assign it to the principal. Doing this before configuring Airbyte means the connection test tells you something useful rather than failing on a permission nobody mentioned.
Step 2: Configure the Azure Blob Storage source
Click Sources in the left navigation, then New Source, and select Azure Blob Storage, following adding a source. Choose the delivery method, enter the storage account and container, then supply either the account key or the tenant, client and secret values. Define your streams with a name of your choosing and a glob pattern selecting the files.
Step 3: Configure the BigQuery destination
Click Destinations, then New Destination, and select BigQuery, following adding a destination. Supply the project, dataset and service account key. Consider Cloud Storage staging rather than the default if your first load covers a large backlog of files.
Step 4: Create the connection and schedule against arrivals
Click Connections, then New connection, select your streams and an incremental sync mode so new files are picked up without re-reading the ones already processed. Set the frequency against how often files actually arrive rather than how quickly anyone would like to see them.
Check the resulting table after the first sync, particularly if your files are CSV, since that format carries no type information and the connector has to infer what each column contains.
Why do correct credentials still fail?
Because Azure separates managing a storage account from reading what is inside it. Owner and Contributor let an identity administer the account, change its settings and read its keys, and neither grants permission to read blob contents through the data plane.
That is why Storage Blob Data Reader exists as a separate role, and why a service principal your Azure administrator considers well provisioned can still fail the connection test. The error says to check the credentials provide permission to read files, which is accurate and reads like a credentials problem rather than a role assignment one.
So check the role assignment before regenerating any secrets. And note that the same error appears when the glob pattern matches nothing, since the connector cannot distinguish a container it cannot read from one where your pattern found no files. Verify both before concluding the credentials are wrong.
Should you use an account key or a service principal?
The account key is quicker and blunter. It is a single secret granting full access to everything in the storage account, it cannot be scoped to one container, and rotating it affects every other tool using the same key. For a quick proof of concept that is fine.
A service principal takes longer to set up and is the better answer for anything lasting. It gets exactly the Storage Blob Data Reader role and nothing else, its access is visible in the portal as a role assignment somebody can audit, and revoking it does not disturb any other integration.
The practical advice is to start with whichever gets you moving and plan the service principal before the pipeline becomes permanent. Swapping later is a configuration change rather than a rebuild, so this is one of the few decisions here that is genuinely reversible.
Frequently asked questions
My service principal has Owner and still cannot read files. Why?
Owner manages the account rather than granting data access. Assign Storage Blob Data Reader on the storage account specifically.
The error mentions credentials but mine are right. What else could it be?
A glob pattern matching no files produces the same message, because the connector cannot tell that apart from being unable to read. Check the pattern and the role assignment before changing secrets.
Which delivery method should I use?
Replicate records, for a warehouse. Copy raw files transfers bytes without parsing them, which suits archival rather than analysis.
Is an account key good enough?
For a proof of concept, yes. For anything lasting, a service principal scoped to Storage Blob Data Reader is auditable and revocable without affecting other tools.
Can I do this without writing code?
Yes. The setup above is UI-driven on both sides, and the Azure role assignment is portal configuration.
Get your Azure Blob Storage data into BigQuery
Assign Storage Blob Data Reader before anything else, because that is where the time goes and the error message will not tell you. Choose replicate records, write a specific glob pattern, ask whether files are rewritten in place, and plan the service principal before the pipeline becomes permanent.
Airbyte's connector catalog includes 600+ pre-built connectors, so file feeds can reach a warehouse and a lake at once. For a similar file source, see SFTP to PostgreSQL, and for another Azure source in a warehouse, Azure Table Storage to Databricks.
Integrate with 700+ apps using Airbyte
Move data from 700+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.