Amazon S3 to BigQuery: How to Move Your Data
Move Amazon S3 files into BigQuery with Airbyte. What cross-cloud egress actually costs, the IAM permission that fails silently, and scoping glob patterns.
Moving Amazon S3 into BigQuery turns files sitting in a bucket into typed tables anybody can query. Partner extracts, exports from another system and output from jobs all land in object storage and stay invisible to your analysts until something loads them.
This is a cross-cloud pipeline, which makes it different from the other two destinations in this series. Data leaves AWS and arrives in Google Cloud, and that boundary has a cost attached that nobody mentions until the bill does.
Amazon S3 to BigQuery at a glance:
Why move data from Amazon S3 to BigQuery?
Two situations account for most of these pipelines.
The first is a cross-cloud reality rather than a preference. A partner or an upstream system writes to S3 because that is what they use, and your analytical estate runs on Google Cloud because that is what you use. The pipeline is the bridge between two decisions nobody made together.
The second is joining those files to data already in BigQuery, such as matching a vendor's product list against your own catalogue. If you control where the files are written and nothing else depends on S3, writing them to Cloud Storage instead removes this pipeline and its egress cost entirely.
What do you need before you start?
Credentials on both clouds, and a policy detail that accounts for most failed first attempts:
An IAM policy covering both the bucket and its objects. You need read and list permissions, and the resource list must name both the bucket itself and everything under it. Object-level permissions alone are not sufficient to authenticate. The S3 source documentation includes the policy to copy.
A glob pattern per stream. There are no predefined streams here, because the streams are whatever your bucket contains. Patterns use wcmatch syntax, are relative to the bucket root, and can be combined with a pipe, with a double asterisk matching every file.
An estimate of how much data will cross the boundary. Egress is charged per byte leaving AWS, so a glob pattern that sweeps in more than you need is not merely untidy here. It has a running cost that compounds every sync.
A BigQuery service account and a decided dataset location. With the BigQuery User and BigQuery Data Editor roles. Queries can only reference datasets in the same physical location and that cannot be changed after creation, so match it to whatever you intend to join against.
Finally, know whether files are ever rewritten in place, because incremental sync works from file modification time and that habit decides what gets re-read, and therefore what you pay to move again.
How do you build an Amazon S3 to BigQuery pipeline in Airbyte?
Step 1: Write the IAM policy and scope the pattern
Create the policy with both resource entries, the bucket ARN and the path beneath it. Then decide which files you actually need, because on a cross-cloud pipeline the glob pattern is a cost control as much as a correctness one.
Step 2: Configure the S3 source
Click Sources in the left navigation, then New Source, and select S3, following adding a source. Choose your authentication method, enter the bucket name, then define a stream for each file format with a name, a format and a glob pattern selecting only files of that type.
Step 3: Configure the BigQuery destination
Click Destinations, then New Destination, and select BigQuery, following adding a destination. Supply the project, dataset and service account key. Batched standard inserts is the default and is ample for most file feeds, with Cloud Storage staging worth considering for a large initial backlog.
Step 4: Create the connection and schedule against arrivals
Click Connections, then New connection, select your streams and an incremental sync mode. Incremental matters more here than usual, because re-reading a file is not just wasted time but egress you pay for a second time.
Check the resulting table after the first sync, particularly with CSV, since that format carries no type information and the connector has to infer what each column contains.
What does crossing clouds actually cost?
More than people expect, and on a bill nobody is watching. Reading from S3 incurs egress charges that land on your AWS account rather than your Airbyte one, so the cost of this pipeline appears in a place the person who built it may never look.
That changes three decisions. Your glob pattern should select only the files you need rather than everything in the bucket. Incremental sync becomes close to mandatory, since re-reading is paid for twice. And a daily schedule on a feed that arrives weekly is five unnecessary reads a week.
It is also worth asking whether the files need to be in S3 at all. If your own system writes them and nothing else consumes them from there, writing to Cloud Storage instead removes the boundary, and the GCS source works the same way. That is not always possible with a partner feed, and it is worth ten minutes of thought before you build.
Why do correct credentials still fail?
Because S3 treats listing a bucket and reading an object as separate permissions, and the connector needs both. A policy granting access to everything under the bucket path looks complete and leaves the connector unable to discover which files exist, which is the step that happens first.
Airbyte's documentation flags this directly, because object-level permissions alone are not sufficient to authenticate. The resource list in your policy needs two entries: the bucket ARN on its own for listing, and the same ARN with a wildcard path for reading the objects inside it.
The error that results reads like a credentials problem, so the instinct is to regenerate the access key, which changes nothing. Check the policy has both entries before touching any secret, and check the glob pattern too, since a pattern matching no files can produce a similar unhelpful result.
Frequently asked questions
Who pays the egress charges?
You do, on your AWS bill rather than your Airbyte one. Scope your glob pattern tightly, use incremental sync, and match the schedule to how often files actually arrive.
My access key is correct but the connection fails. What now?
Check the IAM policy names both the bucket ARN and the path beneath it. Object-level permissions alone are not sufficient to authenticate, and the resulting error looks like a credentials problem.
Should I move the files to Cloud Storage instead?
If you control where they are written and nothing else reads them from S3, yes. That removes the cloud boundary and its egress cost, and the GCS source works the same way.
Which streams does the S3 source provide?
None predefined. The streams are whatever you define from your bucket's contents, each with a format and a glob pattern selecting its files.
Can I do this without writing code?
Yes. The setup above is UI-driven apart from the IAM policy, which the documentation provides for you to paste. You may want BigQuery views casting CSV columns.
Get your Amazon S3 data into BigQuery
Ask first whether the files could be written to Cloud Storage instead, because that removes the boundary and its cost. If not, write the IAM policy with both resource entries, scope the glob pattern tightly since it is now a cost control, use incremental sync, and create the dataset in the right location.
Airbyte's connector catalog includes 700+ pre-built connectors, so file feeds can reach a warehouse alongside the systems they describe. For the same source in another warehouse, see Amazon S3 to Snowflake, and for object storage on another cloud, Azure Blob Storage to BigQuery.
Integrate with 700+ apps using Airbyte
Move data from 700+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.