SFTP to Kafka: How to Stream Your Data

Streaming SFTP files into Kafka is a batch-to-stream conversion, and the first thing to settle is which of Airbyte's two SFTP connectors you are using, because they are not variations on a theme. They have different capabilities and one of them will republish your entire directory on every sync.

The standard SFTP source reads CSV and JSON, gives you one stream per file, and has no notion of which files it has already seen. The SFTP Bulk source consolidates many files into a single stream, reads Avro, CSV, JSONL, Parquet, Excel and document formats, and syncs incrementally based on file creation or modification time. For a Kafka pipeline, where every record you emit is a message someone downstream has to handle, that difference is the difference between a working system and a loop that reproduces your archive daily.

Use SFTP Bulk unless you have a reason not to

Incremental loading is the deciding feature. SFTP Bulk syncs files based on when they were added or last modified, so a directory that gains one file a day produces one file's worth of messages a day. It also accepts glob patterns to select what to read, including double asterisk syntax for recursing into subdirectories, so you can pick up a dated folder structure without reconfiguring anything each month.

Two limits are worth knowing before you commit. Files above roughly 1.5 GB need splitting before upload, and the raw file copy delivery method is not a way around that: it copies bits without parsing them, works only with compatible destinations, and is subject to the same size ceiling. For a Kafka destination you want the record replication method anyway, since Kafka wants messages rather than files. Authentication is password or private key, and the key must be uploaded as a PEM file rather than pasted into the field.

What arrives in the topic

The Kafka destination writes JSON, and only JSON. There is no Avro output, which means consumers cannot validate against a schema registry and will be parsing JSON at whatever rate your topic moves. If your organisation standardised on Avro with a registry, this connector does not fit that convention and it is better to find out now than during integration testing.

Each message carries the Airbyte-assigned UUID as its key, and a value containing four fields: that same identifier, the timestamp the record was pulled from the source, the stream name, and your actual data as a JSON blob. Two consequences follow. Consumers have to reach inside the payload to get at the real content, so plan for that in whatever reads the topic. And because the key is a random UUID rather than a business identifier, records distribute across partitions arbitrarily. Kafka only guarantees ordering within a partition, so two rows describing the same customer can be consumed out of order. If ordering per entity matters, you need a repartitioning step downstream.

Topic naming, and one production decision

The topic_pattern parameter controls where messages land, and it accepts namespace and stream variables so output can be routed dynamically. Because SFTP Bulk consolidates files into a single named stream, a hardcoded topic name is usually the right answer here, which sidesteps the trap dynamic patterns bring.

That trap is worth stating anyway. If you do use dynamic topic names, a new stream appearing in the source will fail the connection unless Kafka is configured to create topics automatically. Enabling automatic topic creation fixes it and is generally discouraged in production, where you want topic creation to be a deliberate act with partition counts and retention set on purpose. Creating topics manually as streams change is the recommended path. Note also that a naming transformation is applied to the target topic name, so certain special characters are replaced, and the topic you get may not be spelled exactly as you wrote it.

Frequently asked questions

Should I use the SFTP source or SFTP Bulk?

SFTP Bulk for almost every case. It supports incremental loading based on file creation or modification time, handles six file formats, and consolidates files into one stream. The standard SFTP source reads only CSV and JSON.

Can Airbyte write Avro messages to Kafka?

Not currently. The Kafka destination writes JSON, so a schema registry cannot be used to validate what Airbyte produces.

Why are my messages spread randomly across partitions?

The message key is the UUID Airbyte assigns each record, not a field from your data. If you need per-entity ordering, repartition on a business key downstream.

How do I handle files larger than 1.5 GB?

Split them into smaller files before they reach the SFTP server. The raw file copy delivery method does not lift the size limit.

Get your SFTP data into Kafka

Choose SFTP Bulk, use a glob pattern that matches your folder structure rather than a fixed path, hardcode the topic, and tell whoever consumes the topic that the payload is nested inside a JSON envelope and that the key is not a business identifier. None of that is difficult, and all of it is easier to decide now than after a consumer is in production.

For the same destination fed by a database with genuine change capture, see PostgreSQL to Kafka. For another file-based source and the format handling that comes with it, see Parquet to Elasticsearch.

Start syncing now →