Confluence to Kafka: How to Move Your Data

Moving Confluence into Kafka publishes documentation changes where several systems can react to them. A search index, a compliance archive and a service that notices when a runbook changes all want the same events, and pointing each at the Confluence API means three integrations doing identical work.

This guide covers the managed path with Airbyte. Two things shape the build: the connector is Cloud only and one of its streams depends on your plan, and Confluence pages are large enough that message size becomes a real consideration.

Confluence to Kafka at a glance:

Why move data from Confluence to Kafka?

Two situations account for most of these pipelines.

The first is fan-out to systems that consume documentation differently. A search index wants the text, a compliance process wants the audit trail, and an internal tool wants to know which pages changed. Publishing once and letting each consume what it needs beats three teams maintaining three Confluence integrations.

The second is feeding a service rather than a report. The usual test applies, and with a single consumer a bus is overhead. If that consumer is a retrieval layer for a language model, Confluence to Weaviate takes the documentation straight there without the intermediate hop.

What do you need before you start?

Four things, and the first two are eligibility rather than configuration:

A Confluence Cloud instance. Data Center and Server deployments are not supported, so a self-hosted instance rules this out entirely. The Confluence source documentation covers the requirements and the streams.

A Standard or Premium plan, if you want the audit stream. Audit is gated behind those tiers, which matters because it is usually the stream a compliance consumer came for. Check before promising anybody an audit trail.

An API token and the account email. Generated against an Atlassian account, carrying that account's visibility. A service account is better than an individual's, since every consumer inherits what it can see and none of them can tell.

Topics created in advance, with message size considered. The destination writes to topics that already exist, and page content is larger than most things you will put on a bus, which makes the broker's maximum message size worth checking rather than assuming.

If your organisation restricts access by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.

How do you build a Confluence to Kafka pipeline in Airbyte?

Step 1: Confirm your deployment and your plan

Two checks, both quick, both capable of ending the project. Confluence Cloud is required, so a Data Center instance means finding another route entirely. And the audit stream needs a Standard or Premium plan, which is worth establishing before a compliance team is told they will be receiving audit events. Neither is something you can work around with configuration.

Step 2: Configure the Confluence source

Click Sources in the left navigation, then New Source, and select Confluence, following adding a source. Supply the domain, email and API token. Five streams are available covering spaces, pages, blog posts, groups and audit, and the connector returns JSON Schema types directly, so there is no type conversion to arrange.

Step 3: Configure the Kafka destination

Click Destinations, then New Destination, and select Kafka, following adding a destination. Supply the bootstrap servers, security protocol and topic configuration. Messages are JSON with no Avro option, and each wraps the record alongside its identifier, extraction timestamp and stream name, so consumers read through an envelope.

Step 4: Create the connection and separate the streams by topic

Click Connections, then New connection, select your streams and a sync mode. Give pages their own topic, since they are far larger than spaces or groups and their consumers are different people. Mixing a small configuration stream into the same topic as page content forces every consumer to handle both.

Daily suits documentation, which changes when somebody edits a page rather than continuously, and a gentler schedule keeps the volume predictable.

What can this connector actually reach?

Confluence Cloud, and five streams within it. Spaces, pages, blog posts, groups and audit describe the structure of your documentation and its content. Data Center and Server deployments are outside the connector entirely, which is the first thing to establish because it is a hard stop rather than a limitation to work around.

The audit stream is the one with a commercial condition attached. It needs a Standard or Premium plan, and it is frequently the reason somebody wanted this pipeline, since who changed which page and when is exactly what a compliance process asks for. Discovering the gate after the project is scoped is an uncomfortable conversation.

One genuinely pleasant detail: the connector returns JSON Schema types, so nothing needs converting or inferring on the way through. For a bus that matters more than usual, because consumers receive types that were declared rather than guessed, and a consuming team does not have to work out whether a field is a number because it looked like one in the samples they checked.

What happens when a page is larger than a message?

It is refused, and this is the pipeline where that becomes a genuine risk. Kafka brokers impose a maximum message size, commonly around one megabyte by default, and most things people publish sit comfortably underneath it. A Confluence page carrying years of accumulated documentation, tables and embedded content does not necessarily.

The failure is specific to the offending page rather than the pipeline, which makes it easy to miss. Everything syncs, one enormous runbook never appears on the topic, and nobody notices until somebody searches for it. Because the envelope adds to the record size as well, a page close to the limit can tip over it without being obviously enormous.

You have two sensible options and they suit different consumers. Raise the maximum message size on the topic, remembering that producers and consumers both need configuring to match, which is fine when you control the cluster. Or publish page metadata to the topic and let consumers fetch full content from Confluence when they need it, which keeps messages small at the cost of an extra call. Decide before the first sync, since changing the contract afterwards means rewriting every consumer.

Frequently asked questions

Does this work with Confluence Data Center?

No. The connector supports Confluence Cloud only, so Data Center and Server instances need a different route entirely.

Why can I not select the audit stream?

It requires a Standard or Premium Confluence plan. That is a subscription question rather than a configuration one, and worth checking before scoping compliance work around it.

One page never arrives on the topic.

It probably exceeds the broker's maximum message size, which is commonly about a megabyte by default. Raise the limit or publish metadata only and fetch content separately.

Do I need to convert types for consumers?

No. The connector returns JSON Schema types directly, so consumers receive declared types rather than having to infer them from samples.

Can I do this without writing code?

The pipeline, yes. The consumers are yours, and if you choose the metadata-only approach they will need to fetch page content from Confluence themselves.

Get your Confluence data into Kafka

Check that you are on Confluence Cloud and that your plan includes the audit stream, because both are hard stops rather than inconveniences. Count your consumers, since one does not justify a bus. Then decide how page content reaches the topic, whether by raising the maximum message size or publishing metadata and letting consumers fetch the body, and settle that before anybody builds against the contract.

Airbyte's connector catalog includes 600+ pre-built connectors, so documentation can reach every system that depends on it. For the same source into a warehouse, see Confluence to Snowflake, and for team conversation onto the same bus, Slack to Kafka.

Start syncing now →