SFTP to Kafka: How to Stream Your Data
Move SFTP files into Kafka with Airbyte. Choosing between delivery methods, why this is not a file watcher, and the contract to give your consumers.
Putting files from an SFTP server onto Kafka gives several systems one feed to subscribe to rather than each polling the same folder. A partner drops a file overnight, and a validation service, a warehouse loader and an alerting job all read the same topic instead of three cron jobs racing each other.
One expectation to set immediately: this is not how you react the instant a file lands. Airbyte syncs on a schedule rather than watching the directory, so the topic receives batches at your sync interval. This guide covers the managed path, and the choice that decides what your consumers actually receive.
SFTP to Kafka at a glance:
Why move data from SFTP to Kafka?
Two situations account for most of these pipelines.
The first is fan-out. Files arriving on an SFTP server are the oldest integration pattern still in daily use, and when several systems need what is in them, each one ends up with its own credentials, its own schedule and its own idea of which files it has already seen. One pipeline into a topic replaces all of that.
The second is replay. A consumer that breaks can be fixed and re-run from an offset rather than asking the partner to resend last week's files. If you have exactly one consumer and it is a warehouse, skip the bus and load there directly.
What do you need before you start?
The first item decides the shape of everything downstream, so settle it before configuring:
A decision about delivery method. Replicate records parses your files into records; copy raw files transfers them without parsing their contents, subject to a maximum of 1.5 GB per file. Only the first gives consumers structured messages. The SFTP Bulk source documentation covers both.
SFTP credentials and a glob pattern. A host, and either a username and password or a key pair. Then a pattern selecting which files to sync, remembering to include the folder if they sit in a subdirectory and using a double asterisk to match recursively.
Topics created in advance. The destination fails on an unknown topic unless the broker creates topics automatically, which is discouraged in production where partition counts and retention should be chosen deliberately.
Knowledge of how the feed behaves. Whether the folder accumulates or is overwritten, whether names carry dates, and whether a file is ever rewritten after landing. Incremental syncing works from file creation or modification time, so those habits decide what gets picked up.
If your Kafka cluster restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.
How do you build an SFTP to Kafka pipeline in Airbyte?
Step 1: Choose the delivery method
If your consumers want records they can act on, choose replicate records so files are parsed. Choose copy raw files only when the bytes should travel untouched, for instance documents or archives being moved rather than analysed, and check your file sizes against the 1.5 GB limit that applies to that method.
Step 2: Configure the SFTP Bulk source
Click Sources in the left navigation, then New Source, and select SFTP Bulk, following adding a source. Supply the host and credentials, pick the delivery method and file type, and set the glob pattern. If the feed drops a fresh snapshot each night rather than accumulating, there is an option to load only the most recent file.
Step 3: Configure the Kafka destination
Click Destinations, then New Destination, and select Kafka, following adding a destination. Supply the bootstrap servers, security protocol and credentials, and a topic pattern. One topic per feed is usually right, since a consumer interested in one partner's files has no use for another's.
Step 4: Create the connection and schedule it
Click Connections, then New connection, select your stream and an incremental sync mode so new files are picked up without re-reading the ones already processed. Set the frequency against how often files actually arrive rather than how quickly anyone would like to hear about them.
Then check the topic names, because a naming transformation replaces certain special characters and folder-derived names do not always survive unchanged.
Why does the delivery method change everything?
Because the two methods are doing different jobs. Replicate records reads inside your files and emits rows, so each message on the topic is a record a consumer can act on. Copy raw files moves the bytes without parsing them, which is a file transfer that happens to use Kafka as the transport.
For a bus feeding several consumers, the first is almost always what you want, because the entire point is that subscribers receive structured data without each writing their own parser. Kafka is also a poor fit for large opaque payloads, which is the other reason the 1.5 GB per file limit on the raw method matters here more than it would elsewhere.
The exception is an archival or forwarding use case where the file must arrive exactly as the partner sent it. If that is your requirement, object storage is usually a better destination than a topic, and the S3 pairing in this series covers it.
What contract should you give your consumers?
The destination writes JSON and only JSON, so consumers cannot validate against a schema registry. Each message value carries the record identifier, the extraction timestamp, the stream name and your data as a nested structure, so consumers unwrap the envelope to reach the fields they want.
Messages are keyed by a UUID Airbyte generates rather than anything from your data, so records that belong together can land in different partitions and be consumed out of order. For a consumer aggregating rows that is harmless. For anything tracking the latest state of an entity, repartition on a business key first.
Be explicit about timing too. Airbyte syncs on a schedule rather than watching the directory, so a file arriving now appears at the next run. Tell consuming teams that plainly, because the word Kafka leads people to assume an event-driven behaviour this pipeline does not offer.
Frequently asked questions
Can I use this to react the moment a file arrives?
Not really. Syncs run on a schedule rather than watching the folder, so the topic receives batches at your sync interval. This pipeline is for durable, replayable delivery to several consumers.
Which delivery method should I use?
Replicate records, in almost all cases, so consumers receive structured messages. Copy raw files is for moving bytes untouched and carries a 1.5 GB per file limit.
How do I sync files from subdirectories?
Include the folder in your glob pattern and use a double asterisk to match recursively. Be specific, since a loose pattern can consolidate two differently shaped feeds into one stream.
Are messages ordered?
Not per entity. Messages are keyed by an Airbyte UUID rather than a business key, so repartition on one if a consumer maintains state in sequence.
Can I do this without writing code?
The Airbyte setup is entirely UI-driven. Consumers need to unwrap the JSON envelope, and any repartitioning is code in whatever stream processing you already run.
Get your SFTP data into Kafka
Choose the delivery method first, because it decides whether consumers get records or bytes. Write a specific glob pattern, ask how the feed behaves before relying on incremental file times, create your topics deliberately, and write down the contract: batches rather than a stream, a wrapped payload, and no ordering without a repartition.
Airbyte's connector catalog includes 600+ pre-built connectors, so partner files can reach a bus and a lake at the same time. For the same source landing as Iceberg tables, see SFTP Bulk to Amazon S3 with AWS Glue, and for another source feeding the same destination, Jenkins to Kafka.
Integrate with 700+ apps using Airbyte
Move data from 700+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.