SFTP Bulk to Amazon S3 with AWS Glue: How to Move Your Data
Move SFTP files into Amazon S3 with AWS Glue using Airbyte. Choosing between delivery methods, the raw file size limit, and glob patterns for file selection.
Moving files from an SFTP server into Amazon S3 with AWS Glue turns a folder somebody drops files into every night, usually a partner feed or a vendor extract, into Iceberg tables your whole AWS estate can query. Athena, Glue jobs, EMR and Redshift Spectrum all read the same tables, and nobody has to remember which bucket the files went to.
This guide covers the managed path with Airbyte. One choice on the source shapes everything else, because SFTP Bulk offers two delivery methods that produce fundamentally different results, and only one of them gives you queryable tables.
SFTP Bulk to Amazon S3 with AWS Glue at a glance:
Why move data from SFTP to S3 with AWS Glue?
Two situations account for most of these pipelines.
The first is making a partner feed queryable. Files arriving on an SFTP server are the oldest integration pattern still in daily use, and they are invisible to everybody until somebody loads them. Landing them as Iceberg tables registered in Glue means an analyst can query last night's file without asking anyone for it.
The second is retention and openness. Object storage is cheap, and Iceberg means the data is readable by every AWS analytical service and by engines outside AWS. Compared with loading the same files into a warehouse, you keep your options open and pay for compute only when someone queries.
What do you need before you start?
Two items on each side, and the first is the decision the rest depends on:
A decision about delivery method. Replicate records parses your files into records; copy raw files transfers them without parsing their contents. Only the first produces tables you can query, and the second carries a maximum size of 1.5 GB per file. The SFTP Bulk source documentation covers both.
SFTP credentials and a glob pattern. A host, and either a username and password or a key pair. Then a glob pattern selecting which files to sync, remembering to include the folder if your files sit in a subdirectory, and using a double asterisk to match recursively.
An S3 bucket and a bucket policy. Created through the IAM console. The S3 Data Lake destination documentation includes the policy to paste in with your bucket name substituted.
A Glue database. AWS Glue is one of the supported Iceberg catalogs, alongside REST, Nessie and Polaris, and it is the one you want if the point is querying from Athena and the rest of the AWS estate.
Finally, find out how the files arrive. Whether the folder accumulates or is overwritten, whether names carry dates, and whether a file is ever rewritten after landing all change how you configure the source, and the person who set up the feed usually knows.
How do you build an SFTP to S3 pipeline in Airbyte?
Step 1: Choose the delivery method
If you want Iceberg tables in Glue that Athena can query, choose replicate records so the files are parsed into records. Choose copy raw files only when the contents should be preserved untouched, for instance images, archives or documents you are storing rather than analysing, and check your file sizes against the 1.5 GB limit that applies to that method.
Step 2: Configure the SFTP Bulk source
Click Sources in the left navigation, then New Source, and select SFTP Bulk, following adding a source. Supply the host and credentials, pick the delivery method, choose the file type and set the glob pattern. If your feed drops a fresh snapshot each night rather than accumulating, there is an option to load only the most recent file from the folder, which is exactly right for that pattern.
Step 3: Configure the S3 Data Lake destination
Click Destinations, then New Destination, and select S3 Data Lake, following adding a destination. Supply your bucket, region and credentials, then select AWS Glue as the catalog and give it the database name. The connector writes Iceberg tables to the bucket and registers them in Glue, which is what makes them queryable from Athena without further work.
Step 4: Create the connection and verify from Athena
Click Connections, then New connection, select your stream and a sync mode. Incremental works by file creation or modification time, so new files arriving in the folder are picked up without re-reading the ones already processed. After the first sync, query the table from Athena to confirm the Glue registration worked end to end.
Check the registered table names too. When using AWS Glue, the destination changes non-alphanumeric characters in table names and namespaces to underscores for Athena compatibility, so the names you see may not be spelled as you wrote them.
Why does the delivery method change everything?
Because the two methods are doing different jobs. Replicate records reads inside your files and emits rows, which the destination then writes as Iceberg tables with real columns. Copy raw files moves the bytes without parsing their contents, which is a file transfer with a lake as the target.
If somebody asks for partner data to be queryable in Athena, replicate records is the answer and copy raw files will disappoint them, because a bucket of untouched CSVs is not a table. If somebody asks for a compliance archive of exactly what the partner sent, the reverse is true and parsing would defeat the purpose.
The size limit follows from that distinction. Copying raw files is subject to a maximum of 1.5 GB per individual file, and that limit applies to the raw transfer process rather than to parsed replication. On a feed that occasionally produces a very large archive, this is the thing that will stop the sync.
How should you select and consolidate files?
The connector consolidates multiple files into a single stream, which is what you want for a lake. A folder holding a year of daily extracts with the same columns becomes one table rather than 365, and that is the difference between a dataset someone can query and a directory listing.
The glob pattern is how you control that. Include the folder in the pattern when files sit in a subdirectory, and use a double asterisk to match recursively through subdirectories. Being specific matters, because a pattern that accidentally matches two differently shaped feeds will consolidate them into one confused table.
One caveat about incremental syncing on this source. It works from file creation or modification time, so a partner who rewrites yesterday's file in place will have it picked up again, while one who edits a file without changing its timestamp will not. Ask which of those happens before assuming the feed is fully captured.
Frequently asked questions
Which delivery method should I use?
Replicate records if you want queryable Iceberg tables in Glue. Copy raw files only if the contents should be preserved unparsed, and note the 1.5 GB per file limit that applies to that method.
Which file formats are supported?
Avro, CSV, JSONL, Parquet and document file formats, each with a dynamic schema. Glob patterns work across all of them.
How do I sync files from subdirectories?
Include the folder in your glob pattern, and use a double asterisk to match recursively through subdirectories.
My feed drops a fresh snapshot each night. What should I configure?
Use the option to load only the most recent file from the folder path, which exists for exactly that pattern.
Can I do this without writing code?
The Airbyte setup is entirely UI-driven. The AWS side needs a bucket policy, which is JSON the documentation provides, and a Glue database.
Get your SFTP files into S3
Choose the delivery method first, because it decides whether you end up with tables or files. Then write a specific glob pattern, ask how the feed behaves before relying on incremental file times, set up the bucket and Glue database in advance, and query from Athena after the first sync to confirm the whole chain worked.
Airbyte's connector catalog includes 600+ pre-built connectors, so partner files can sit beside your database and application sources in one lake. For a database landing in the same place, see PostgreSQL to Amazon S3 with AWS Glue, and for the same source feeding a streaming platform, SFTP to Kafka.
Integrate with 700+ apps using Airbyte
Move data from 700+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.