SFTP to PostgreSQL: How to Move Your Data
Move SFTP files into PostgreSQL with Airbyte. Choosing the delivery method, writing a precise glob pattern, and what file-time incremental sync actually tracks.
Moving files from an SFTP server into PostgreSQL turns a folder somebody drops files into every night into rows an application or a reporting tool can query. Partner feeds and vendor extracts arrive this way constantly, and they are invisible to everybody until something loads them.
This guide covers the managed path with Airbyte. One choice on the source shapes everything else, because SFTP Bulk offers two delivery methods and only one of them gives you rows in a table rather than bytes in a bucket.
SFTP to PostgreSQL at a glance:
Why move data from SFTP to PostgreSQL?
Two situations account for most of these pipelines.
The first is making a partner feed usable by software you already run. An application or an internal tool that reads Postgres can query last night's file the moment it lands, without anybody writing a parser or scheduling a script that somebody then owns forever.
The second is joining a file feed to relational data you already hold, such as matching a vendor's product list against your own catalogue. If your feeds are large or you want to keep them indefinitely, a lakehouse suits that better than a single database.
What do you need before you start?
The first item decides the shape of everything downstream, so settle it before configuring:
A decision about delivery method. Replicate records parses your files into records; copy raw files transfers them without parsing their contents, subject to a maximum of 1.5 GB per file. Only the first produces queryable tables. The SFTP Bulk source documentation covers both.
SFTP credentials and a glob pattern. A host, and either a username and password or a key pair. Then a pattern selecting which files to sync, remembering to include the folder if they sit in a subdirectory and using a double asterisk to match recursively.
Knowledge of how the feed behaves. Whether the folder accumulates or is overwritten, whether names carry dates, and whether a file is ever rewritten after landing. Incremental syncing works from file creation or modification time, so those habits decide what gets picked up.
A PostgreSQL user with write access. On the target schema. If you already run a Postgres destination on an older connector version, read the migration guide before upgrading.
If your organisation restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list on the PostgreSQL side before you begin.
How do you build an SFTP to PostgreSQL pipeline in Airbyte?
Step 1: Choose the delivery method
For a relational destination this is straightforward: choose replicate records, because a database wants rows and copy raw files would leave you storing bytes you then have to parse yourself. The raw option exists for archival cases where object storage is the better target anyway.
Step 2: Configure the SFTP Bulk source
Click Sources in the left navigation, then New Source, and select SFTP Bulk, following adding a source. Supply the host and credentials, pick replicate records and your file type, and set the glob pattern. If the feed drops a fresh snapshot each night rather than accumulating, there is an option to load only the most recent file.
Step 3: Configure the PostgreSQL destination
Click Destinations, then New Destination, and select Postgres, following adding a destination. From version 3.0.0 the Postgres destination uses Direct Load architecture, writing records straight to final tables with no intermediate raw tables, and adds its own metadata columns alongside your data.
Step 4: Create the connection and expose views
Click Connections, then New connection, select your stream and an incremental sync mode so new files are picked up without re-reading the ones already processed. Then build views over the raw tables so anything consuming this reads a stable interface rather than Airbyte's metadata columns.
Set the frequency against how often files actually arrive. A nightly feed does not need an hourly sync, and a pipeline running when nothing has landed is just noise in your monitoring.
How should you select and consolidate files?
The connector consolidates multiple files into a single stream, which is what you want. A folder holding a year of daily extracts with the same columns becomes one table rather than 365, and that is the difference between a dataset somebody can query and a directory listing.
The glob pattern is how you control that. Include the folder in the pattern when files sit in a subdirectory, and use a double asterisk to match recursively. Being specific matters, because a pattern that accidentally matches two differently shaped feeds will consolidate them into one confused table.
That risk is sharper on a relational destination than elsewhere. A warehouse would absorb the mismatched columns and leave you to sort it out in SQL; Postgres wants a coherent table, so a loose pattern produces a failure or a mess rather than something you can quietly clean up later.
What does incremental actually track?
File creation or modification time, rather than anything inside the files. That is the right mechanism for a file feed and it has two consequences worth knowing before you rely on it.
A partner who rewrites yesterday's file in place gives it a new timestamp, so its contents are read again and those rows arrive twice. Where your data has a natural key, a deduplicating sync mode handles it; where it does not, you need a view that picks the latest version or a conversation with the partner.
The opposite case is worse. A partner who edits a file without changing its timestamp produces a file the sync will never revisit, and nothing reports it. Ask which of these happens before assuming the feed is fully captured, because the answer changes what you build.
Frequently asked questions
Which delivery method should I use?
Replicate records, for a relational destination. Copy raw files transfers bytes without parsing them, which is for archival cases where object storage is the better target.
How do I sync files from subdirectories?
Include the folder in your glob pattern and use a double asterisk to match recursively. Be specific, since a loose pattern can consolidate two differently shaped feeds into one table.
Why did the same rows arrive twice?
A file was rewritten in place, giving it a new modification time. Use a deduplicating sync mode where your data has a natural key, or a view that picks the latest version.
Is PostgreSQL big enough for a file feed?
For a focused feed, comfortably. Airbyte recommends relational destinations for roughly 10GB or less, so years of accumulated extracts may be better served by a lake.
Can I do this without writing code?
Yes. The setup above is entirely UI-driven. You will want SQL views so consumers read a stable interface rather than the raw synced tables.
Get your SFTP data into PostgreSQL
Choose replicate records, write a specific glob pattern because a relational destination is unforgiving of mismatched files, ask the partner whether files are ever rewritten or edited in place, and expose views rather than raw tables.
Airbyte's connector catalog includes 600+ pre-built connectors, so partner files can reach a database and a lake at once. For the Iceberg version, see SFTP Bulk to Amazon S3 with AWS Glue, and for fan-out to several consumers, SFTP to Kafka.
Integrate with 700+ apps using Airbyte
Move data from 700+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.