ClickHouse to Kafka: How to Move Your Data

Streaming ClickHouse to Kafka publishes analytical results onto topics that operational services can consume, which is the usual reason to run an analytics engine backwards: something downstream needs the numbers ClickHouse already computed. Airbyte will build it with no code, though two constraints sit at either end and both are worth knowing before you design anything.

The Kafka destination is a Marketplace-tier connector available on Airbyte Core and PyAirbyte but not on the Standard, Plus, Pro, or Enterprise Flex plans. And the ClickHouse source has no change data capture, so what you can publish is limited to what a cursor can see.

What can the ClickHouse source publish?

The ClickHouse source is built on the JDBC code base and supports full refresh and cursor-based incremental sync. Its documentation lists replicating incremental deletes and logical replication from the write-ahead log as coming soon, so neither is available today.

For this particular pairing that limitation matters less than it usually would. ClickHouse tables are typically append-heavy event stores where rows are written once, and a timestamp cursor captures everything you would want to publish. A topic is itself append-only, so the two shapes agree.

The exception is ReplacingMergeTree. A background merge that replaces a row does not advance the cursor, so the replacement may never be extracted and consumers see a version ClickHouse has already superseded. If convergence matters, read through a view applying FINAL. Note also that connector version 0.4.0 declared temporal column typing a breaking change, so check the migration guide if you are upgrading an existing connection.

One structural point: a single source cannot cover multiple ClickHouse databases. Grant your read-only user access to each schema and configure a separate source per schema, each connecting to the same instance.

What lands on the topic?

Each stream maps to a Kafka topic. The message key is the UUID Airbyte assigned the record, and the value is an envelope with four fields: _airbyte_ab_id, _airbyte_emitted_at, _airbyte_stream, and _airbyte_data holding your row as a JSON blob. Consumers must unwrap that data field, so write the envelope shape into the consumer contract before anyone builds against it.

One thing specific to ClickHouse deserves attention here. Array and Nested columns are heavily used for tags and event properties, and they arrive serialised inside that JSON payload rather than as a native structure. Consumers expecting a typed list will need to parse them, which is worth agreeing before rather than after.

The Kafka destination writes JSON only, with Avro described as a future addition, so consumers expecting Avro against a schema registry need a translation step in between. It supports full refresh append and incremental append but neither overwrite nor deduped modes, which is an honest reflection of what a log is.

Topic naming and producer settings

The topic_pattern field takes a literal name or a template using {namespace} and {stream}. One topic per table is the usual choice, which argues for the dynamic form and brings a dependency: a new table produces a new topic, and the sync fails if that topic does not exist and automatic creation is off at the broker. Airbyte's documentation recommends leaving it off in production and pre-creating topics deliberately.

A naming transformation is applied to the target topic, replacing some special characters, so check the resulting name rather than assuming your pattern survives verbatim.

The connector exposes most of the standard Kafka producer configuration. Enable idempotence so retries do not create duplicates, and set acks to all if losing a message is worse than failing a sync. Raise batch size and linger together for throughput, which matters here because ClickHouse tables are often large and a sync produces a burst of messages rather than a steady trickle. Check the maximum request size against your widest row, particularly if those Array columns are long.

Setting up the source

You need ClickHouse Server 21.3.10.1 or later and, ideally, a dedicated read-only user with SELECT on the relevant database. SSL is supported, and if the instance has no public address the connector can reach it through an SSH tunnel to a bastion host using an RSA key or a password, where the SSH login username is the operating system user on the bastion rather than a ClickHouse account.

Choose your cursor with the sync interval in mind. Airbyte syncs are batch jobs, so a topic fed this way receives bursts on a schedule rather than a continuous stream, and consumers should be built for that rhythm rather than for millisecond latency.

Frequently asked questions

Will deletions in ClickHouse reach the topic?

No. The source lists incremental deletes as coming soon, so deletions are never extracted and consumers will not learn about them.

Can I run this on a paid Airbyte Cloud plan?

The Kafka destination shows as available on Core and PyAirbyte only, not on Standard, Plus, Pro, or Enterprise Flex. Plan on a self-managed deployment.

How do ClickHouse arrays arrive for consumers?

Serialised inside the JSON payload rather than as a native structure. Consumers expecting typed lists need to parse them.

Is this a real-time stream?

No. Airbyte syncs are batch jobs, so the topic receives a burst of messages per sync. If you need sub-second latency, a purpose-built streaming tool is the better fit.

Get your ClickHouse data into Kafka

Pick a cursor suited to an append-heavy table, pre-create your topics, and design consumers around batched JSON envelopes. If your target is a queryable store rather than a log, see our guide to ClickHouse to BigQuery. For streaming from a different source, see n8n to Kafka.

Start syncing now →