Confluence to Databricks: How to Move Your Data
Move Confluence into Databricks with Airbyte. What the five streams actually contain, the plan requirement for audit records, and how to model the joins.
Syncing Confluence to Databricks is worth setting up for a narrower reason than most people expect, so it is worth being precise about what the connector returns before you build anything on top of it.
The Airbyte Confluence source reads the Confluence Cloud REST API and exposes spaces, pages, blog posts, groups and audit records. That is a good dataset for questions about documentation health and access: which spaces are alive and which have not been touched in two years, who has permission to what, what changed and when. It is not a content pipeline for building a search corpus or a retrieval index, and if that is your goal you will be happier knowing now.
The audit stream is the reason to build this
Confluence keeps audit records for a limited retention period, and the interface for querying them is not built for analysis. Landing them in Databricks changes both facts at once: the history stops expiring, and it becomes something you can join against your identity provider, your HR data, and your other audit sources in a single query.
There is a prerequisite. The audit stream requires a Confluence Standard or Premium plan. On the free tier it is simply not available, and a connection configured to sync it will not return the data you are looking for. Check the plan before you scope the project, because the audit stream is usually the part that justifies it.
A connector with no type conversion
One unusual and genuinely pleasant property of this source: the Confluence Cloud REST API uses the same JSON Schema types Airbyte uses internally, so no type conversion happens on the way through. String is string, date-time is date-time, and nested objects and arrays arrive as they were.
In practice this removes a whole class of debugging. When something looks wrong in Databricks you can be confident the connector did not silently coerce a value, which narrows the investigation to the API response or your own downstream model. On sources where the type mapping is lossy, that certainty is worth quite a lot.
Setting it up
The source needs three things: an Atlassian API token, your Confluence domain name, and the email address of the account the token belongs to. Create the token against a service account rather than an individual's login if you can, because a connector that breaks when someone leaves the company is a connector that will break when someone leaves the company.
On the Databricks side, each stream becomes a table in Unity Catalog, and the connector needs permission to create Volumes for Avro staging. Confluence data is small by warehouse standards, so performance is not the concern here. The concern is that page and blog post records reference their space and their author by identifier, so the joins that make this dataset useful only work if you sync spaces and groups alongside the content streams rather than selecting pages on their own.
Frequently asked questions
Can I use this connector to build a search index over our Confluence content?
It is not what the connector is built for. The streams describe spaces, pages, blog posts, groups and audit records, which suits documentation governance and access analysis rather than content retrieval.
Why is my audit stream returning nothing?
The audit stream requires a Confluence Standard or Premium plan. On lower tiers the data is not exposed through the API at all.
Does the connector work with Confluence Data Center?
It is built against the Confluence Cloud REST API and configured with a Cloud domain, Atlassian account email and API token. Treat Cloud as the supported deployment.
Which streams should I select?
Sync spaces and groups alongside pages and blog posts. Content records reference spaces and authors by identifier, so selecting content streams alone leaves you with keys you cannot resolve.
Get your Confluence data into Databricks
This is a small, well-behaved pipeline with one plan-level prerequisite and one modelling requirement. Confirm you are on Standard or Premium if the audit stream is the point, sync the reference streams alongside the content ones, and use a service account for the token.
For another SaaS source whose Airbyte streams describe configuration rather than activity, Genesys to Databricks covers a closely related situation. For the same lakehouse fed by a high-volume relational source, see PostgreSQL to Databricks.
Integrate with 600+ apps using Airbyte
Move data from 600+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.