Amazon S3 to Snowflake: How to Move Your Data

Moving Amazon S3 into Snowflake turns files sitting in a bucket into typed tables anybody can query. Partner extracts, exports from another system and output from jobs all land in object storage and stay invisible to your analysts until something loads them.

Snowflake can also read S3 directly through an external stage, so the honest first question is whether you need a pipeline at all. Beyond that, one IAM detail accounts for most failed first attempts, and it produces an error that reads like bad credentials.

Amazon S3 to Snowflake at a glance:

Why move data from Amazon S3 to Snowflake?

Two situations account for most of these pipelines.

The first is making a file feed usable without asking every analyst to think about buckets. A loaded table is typed once, governed like everything else in your warehouse, and queried without anybody needing storage credentials or knowing which prefix this month's extract lands under.

The second is joining those files to warehouse data, such as matching a vendor's product list against your own catalogue. If the files are already in a format Snowflake reads well and nobody needs a permanent copy, an external stage may serve you better, which the section below covers honestly.

What do you need before you start?

The IAM policy is the item worth getting right first, because it is where most first attempts fail:

An IAM policy covering both the bucket and its objects. You need read and list permissions, and the resource list must name both the bucket itself and everything under it. Object-level permissions alone are not sufficient to authenticate. The S3 source documentation includes the policy to copy.

A decision about how to authenticate. An IAM user with an access key and secret, an IAM role identified by its ARN, or no authentication at all for a genuinely public bucket. A role is the better answer for anything lasting, since there is no long-lived key to rotate or leak.

A glob pattern per stream. There are no predefined streams here, because the streams are whatever your bucket contains. Patterns use wcmatch syntax, are relative to the bucket root, and can be combined with a pipe, with a double asterisk matching every file.

Snowflake objects and a role. A warehouse, database, schema and a role able to create tables in that schema. Size the warehouse for the initial backlog rather than the steady state, since the first sync is usually far heavier than anything after it.

Finally, know whether files are ever rewritten in place, because incremental sync works from file modification time and that habit decides what gets re-read.

How do you build an Amazon S3 to Snowflake pipeline in Airbyte?

Step 1: Write the IAM policy properly

Create the policy with both resource entries, the bucket ARN and the path beneath it, then attach it. Doing this before you touch Airbyte means the connection test tells you something useful rather than failing on a permission nobody mentioned, which is the most common way an hour disappears here.

Step 2: Configure the S3 source

Click Sources in the left navigation, then New Source, and select S3, following adding a source. Choose your authentication method, enter the bucket name, then define a stream for each file format with a name, a format and a glob pattern selecting only files of that type.

Step 3: Configure the Snowflake destination

Click Destinations, then New Destination, and select Snowflake, following adding a destination. Supply the account identifier, warehouse, database, schema and role. If your Snowflake account runs on AWS in the same region as the bucket, this pipeline is considerably cheaper to operate, for reasons covered below.

Step 4: Create the connection and schedule against arrivals

Click Connections, then New connection, select your streams and an incremental sync mode so new files are picked up without re-reading what has already been processed. Set the frequency against how often files actually land rather than how fresh anyone would like the data.

Check the resulting table after the first sync, particularly with CSV, since that format carries no type information and the connector has to infer what each column contains.

Why do correct credentials still fail?

Because S3 treats listing a bucket and reading an object as separate permissions, and the connector needs both. A policy granting access to everything under the bucket path looks complete and leaves the connector unable to discover which files exist, which is the step that happens first.

Airbyte's documentation flags this directly, because object-level permissions alone are not sufficient to authenticate. The resource list in your policy needs two entries: the bucket ARN on its own for listing, and the same ARN with a wildcard path for reading the objects inside it.

The error that results reads like a credentials problem, so the instinct is to regenerate the access key, which changes nothing. Check the policy has both entries before touching any secret, and check the glob pattern too, since a pattern matching no files can produce a similar unhelpful result.

Why not use a Snowflake external stage?

It is a fair question, because Snowflake reads S3 natively through external stages and can load with COPY INTO or query files in place as external tables. For a one-off load of a few well-formed Parquet files, that is less machinery and you should prefer it.

A pipeline earns its place on everything that happens afterwards. Incremental tracking of which files have already been read, schema inference across formats, retries, and the same scheduling and alerting you use for every other source. A stage gives you a load mechanism and leaves the orchestration to you.

The cost question cuts the other way, though, and deserves stating plainly. Reading from S3 incurs egress charges on your AWS bill, and a stage loading within the same region and cloud avoids moving data across a boundary at all. If your Snowflake account runs on AWS in the bucket's region, weigh that honestly before choosing.

Frequently asked questions

My access key is correct but the connection fails. What now?

Check the IAM policy names both the bucket ARN and the path beneath it. Object-level permissions alone are not sufficient to authenticate, and the resulting error looks like a credentials problem.

Which streams does the S3 source provide?

None predefined. The streams are whatever you define from your bucket's contents, each with a format and a glob pattern selecting its files.

Should I use an external stage instead?

For a one-off load of well-formed files, often yes, especially if Snowflake runs on AWS in the same region. A pipeline wins on incremental tracking, schema inference and shared scheduling.

Who pays the egress charges?

You do, on your AWS bill rather than your Airbyte one. Keeping the warehouse in the same cloud and region as the bucket reduces it considerably.

Can I do this without writing code?

Yes. The setup above is UI-driven apart from the IAM policy, which the documentation provides for you to paste. You may want Snowflake views casting CSV columns.

Get your Amazon S3 data into Snowflake

Write the IAM policy with both resource entries before anything else, because that is where the time goes and the error will not tell you. Weigh an external stage honestly if your warehouse sits in the same region, define a stream per format with a precise glob pattern, and ask whether files are rewritten in place.

Airbyte's connector catalog includes 700+ pre-built connectors, so file feeds can reach a warehouse alongside the systems they describe. For the same engine on another cloud, see Azure Blob Storage to BigQuery, and for a similar file source, SFTP to PostgreSQL.

Start syncing now →