Parquet File to Kafka: How to Move Your Data

Sending Parquet files to Kafka turns a batch artefact into a stream of messages, which teams do to replay historical data through a streaming pipeline, backfill a topic that consumers already read, or bridge a file-based partner feed into an event-driven architecture. Airbyte does this without code, and the mismatch between the two formats is the interesting part.

Parquet is columnar, compressed, and strongly typed. Kafka messages are individual JSON documents. Everything Parquet is good at gets undone in transit, which is fine as long as you expect it.

What happens to Parquet's types?

The reason people choose Parquet is that it carries a schema. Column types are explicit, decimals have declared precision and scale, and integer widths are distinct. A reader does not have to guess.

The Kafka destination writes JSON and only JSON, with Avro described in the documentation as a future addition. So the schema information that made Parquet worth using is not carried through to your consumers, who receive JSON values and must apply their own expectations.

This matters most for decimals. A Parquet decimal with declared precision becomes a JSON number, and any consumer parsing that into a floating point type introduces exactly the imprecision the Parquet decimal existed to avoid. If your files carry monetary values, tell your consumers to parse those fields as strings or exact decimals rather than as doubles.

If your Kafka ecosystem runs a schema registry, this connector will not feed it directly and you will need a translation step between the topic and those consumers.

Reading files without republishing everything

The destination supports full refresh append and incremental append, and neither overwrite nor deduped modes. You cannot overwrite a log, so every sync adds messages.

That makes file selection the thing that controls your topic. If the connector re-reads a file it has already processed, every row on that file is published again, and consumers see duplicates with no deduplication anywhere in the pipeline to catch them.

For files on an SFTP server, the SFTP Bulk source is the right choice rather than the plain SFTP connector, because it supports Parquet and loads incrementally based on when files were added or modified. The plain SFTP source handles only CSV and JSON. For files in object storage, use the corresponding storage connector with Parquet selected as the format.

Write your glob pattern precisely. A pattern that is too broad picks up archive folders and test files, and each of those becomes messages on your topic that consumers will process as though they were real. Because the topic is append-only, there is no undoing it short of a topic reset.

What consumers will receive

Each row becomes a message. The key is the UUID Airbyte assigned the record, and the value is an envelope with four fields: _airbyte_ab_id, _airbyte_emitted_at, _airbyte_stream, and _airbyte_data containing the row itself. Consumers unwrap the data field.

Note what the emitted timestamp means. It records when Airbyte read the row, not when the underlying event happened. A file containing last month's events, synced today, produces messages stamped with today. If ordering or event time matters to consumers, they must read a timestamp column from inside the data payload rather than trusting the envelope.

A large Parquet file also produces a very large burst of messages in a single sync, since one file can hold millions of rows. Raise batch size and linger together for throughput, enable idempotence so producer retries do not create duplicates, and check the maximum request size against your widest row.

Topic naming

The topic_pattern field takes a literal name or a template using {namespace} and {stream}. For a file source that bulk-ingests matching files into a single stream, a hardcoded topic name is usually simplest and avoids the failure where a new stream resolves to a topic that does not exist. Airbyte's documentation recommends leaving automatic topic creation off in production and creating topics deliberately.

Frequently asked questions

Can I publish Avro to a schema registry?

Not directly. The connector writes JSON only, with Avro noted as a future addition, so a translation step is needed between the topic and registry-aware consumers.

How do I avoid republishing the same rows?

Use a source that loads incrementally by file modification time, such as SFTP Bulk, and write a precise glob pattern. The destination is append-only and cannot deduplicate.

Why do my message timestamps all show today?

The emitted timestamp records when Airbyte read the row, not when the event occurred. Consumers needing event time should read it from a column inside the data payload.

Will my decimal values stay exact?

Only if consumers parse them carefully. Parquet's declared precision does not survive into JSON, so a consumer reading them as doubles will introduce imprecision.

Get your Parquet data into Kafka

Choose a source that tracks files incrementally, write a precise pattern, pre-create your topic, and tell consumers where event time actually lives. If your target is a queryable store rather than a log, see our guides to Parquet to BigQuery and Parquet to MongoDB.

Start syncing now →