n8n to Kafka: How to Stream Your Data

Syncing n8n to Kafka publishes your workflow execution history onto a topic, where other systems can consume it: an alerting service watching for failures, a stream processor computing rolling reliability metrics, or a fan-out to several downstream consumers at once.

One characteristic of this pairing shapes everything else, and it is worth understanding before you build. The n8n source supports full refresh only, and Kafka topics accumulate messages rather than being replaced. Put those together and every sync republishes your entire execution history to the topic. There is no overwrite in Kafka the way there is in a warehouse table.

That does not make the pipeline useless, but it does mean your consumers have to deduplicate, and your sync frequency has real consequences. If you were expecting a clean stream of new executions only, this is not that.

Should you build this pipeline?

Worth answering honestly before configuring anything.

It fits when Kafka is already your integration backbone and you want execution data available to several consumers without each of them polling the n8n API. It also fits when a stream processor downstream is going to deduplicate and aggregate anyway, in which case republished history costs you throughput but not correctness.

It fits badly when you want near-real-time alerting on failures, because a scheduled full refresh is a poor substitute for an event stream. n8n can call a webhook or post to Kafka directly from a workflow, and for alerting that is the better design. It also fits badly when you simply want to analyse execution history, where n8n to ClickHouse gives you a clean queryable snapshot with none of this complexity.

What does the n8n source provide?

One stream: executions. The n8n source reads from the n8n API and exposes the record of workflow runs. Workflow definitions, credentials, users, and tags are not available. It is a Marketplace connector carrying a low sync success rate in Airbyte's own metadata, so build monitoring in rather than assuming silence means success.

Create an API key under Settings, then API in n8n, and configure the source with that plus your instance host.

How should you name your topics?

Each stream is written to a corresponding topic, and the Kafka destination controls this through the topic_pattern parameter. With a single stream from n8n the choice is simpler than usual, but it still has consequences.

Because n8n produces exactly one stream, a hardcoded topic name is almost certainly right. Dynamic patterns exist for sources with many streams that change over time, and they bring a real cost: without auto topic creation enabled, a schema change on the source can fail your connection outright.

Note also that a naming convention transformation is applied to the target topic name, so certain special characters get replaced. Check what topic actually received your messages after the first sync rather than assuming.

What does each message look like?

This matters for your consumers, and it is where you solve the duplication problem.

The connector currently writes JSON only, with Avro noted as a future addition. Each record carries a UUID assigned by Airbyte in the message key. The value contains _airbyte_ab_id, the UUID assigned to that event, _airbyte_emitted_at, a timestamp for when the event was pulled from the source, and the record data itself.

Here is the important detail for deduplication: the Airbyte UUID is assigned per event processed, so the same n8n execution republished on the next sync gets a different _airbyte_ab_id. Deduplicating on that field will not work. Deduplicate on the n8n execution ID inside the record data instead, and treat _airbyte_emitted_at as the marker of which sync delivered it.

What do you need on the Kafka side?

Your brokers must be reachable from Airbyte, and the Airbyte user needs permission to write into the target topic. Create that topic in advance. Enabling auto.create.topics.enable works but is not recommended for production environments, and with a single hardcoded topic you gain nothing from it.

One authentication limitation is worth checking early. Community reports indicate the Kafka connectors cannot authenticate against brokers secured with SASL_SSL and SCRAM, because that setup requires keystore configuration the connector does not expose. If your cluster uses that combination, verify against current documentation before committing to this design, since it may block you entirely.

Set your sync frequency deliberately. Every run republishes the full execution history, so retention settings on the topic matter as much as the schedule. A daily sync against a busy n8n instance still means a lot of repeated messages, and topic retention is what stops that growing without bound.

Frequently asked questions

Why do I keep seeing the same executions in my topic?

Because the n8n source only supports full refresh, so every sync reads all executions and publishes them again. Kafka topics accumulate rather than being replaced, so consumers need to deduplicate on the n8n execution ID.

Can I deduplicate on _airbyte_ab_id?

No. That UUID is assigned per event processed, so a republished execution receives a new one each sync. Use the execution ID from the record data as your deduplication key.

Does the connector support Avro?

Not currently. The destination writes JSON only, with other formats noted as future work. If your Kafka ecosystem standardises on Avro with a schema registry, that is a meaningful constraint to weigh.

Should I use a hardcoded or dynamic topic name?

Hardcoded, for this source. n8n produces one stream, so dynamic naming adds the need for automatic topic creation without giving you anything in return.

Is there a better way to get n8n failures into Kafka in real time?

Usually yes. Have the workflow itself publish on failure, either directly or via a webhook. A scheduled full refresh is not a substitute for event-driven alerting, and this pipeline suits bulk history distribution rather than immediacy.

Get your n8n data into Kafka

Use a hardcoded topic, create it in advance, plan for consumer-side deduplication, and set topic retention with republished history in mind. Want a queryable snapshot rather than a stream? See our guides to n8n to ClickHouse and n8n to Databricks.

Start syncing now →