Confluence to Databricks: How to Move Your Data
Move Confluence into Databricks with Airbyte. Why this is documentation governance rather than content, the plan-gated audit stream, and Cloud-only support.
Moving Confluence into Databricks is a documentation governance pipeline rather than a content one, and being clear about that saves a wasted project. The connector gives you the structure of your wiki, which spaces exist, which pages live in them, who wrote what and when, rather than a searchable corpus of the prose.
That is genuinely useful once a Confluence instance reaches the size where nobody can describe it. This guide covers the managed path with Airbyte, and the two things that shape what you actually get: a narrow stream list, and one stream gated behind your Atlassian plan.
Confluence to Databricks at a glance:
Why move data from Confluence to Databricks?
Two situations account for most of these pipelines.
The first is documentation sprawl. Any Confluence instance a few years old contains thousands of pages, most of which nobody has opened since the quarter they were written. Knowing which spaces are abandoned, which pages have not changed in two years and who owns them is how you clean that up without deleting something important.
The second is joining documentation activity to everything else, such as whether teams that document more ship differently. If instead you want to search the prose or build a retrieval corpus, a vector store is the right destination and the chunking question is a different article.
What do you need before you start?
Two of these can rule out part of the project, so check them first:
A Confluence Cloud instance. The connector works against Cloud rather than Data Center or Server installations, so a self-hosted Confluence is not in scope for this pipeline. The Confluence source documentation covers the requirements.
An Atlassian API token and the account email. Used together as basic auth, alongside your domain name. The token inherits the permissions of the account that created it, so a service account with deliberately scoped access is better than an individual's login.
A Standard or Premium plan, if you want the audit stream. Audit records are gated behind those plans, so on a free or lesser tier that stream is simply unavailable. Check before promising anyone a record of who changed what.
Unity Catalog access. Each stream becomes a table in Unity Catalog and the connector needs permission to create Volumes for Avro staging. Agree the catalog and schema before the first sync.
If your organisation restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.
How do you build a Confluence to Databricks pipeline in Airbyte?
Step 1: Check the streams answer your question
Write down the question somebody asked you and check it against the five streams. If it is about which spaces exist, who is writing, or what has gone stale, this connector answers it. If it contains the words search or what does the page say, it does not, and no configuration changes that.
Step 2: Configure the Confluence source
Click Sources in the left navigation, then New Source, and select Confluence, following adding a source. Supply the API token, the email address of the account that created it, and your domain name. Airbyte tests the connection immediately.
Step 3: Configure the Databricks destination
Click Destinations, then New Destination, and select Databricks, following adding a destination. Supply your workspace details, the catalog and schema, and authentication. Volume is never a concern here, since even a large Confluence instance amounts to a modest number of rows.
Step 4: Take the reference streams alongside pages
Click Connections, then New connection, and select streams with a sync mode. Pages and blog posts carry the activity, and spaces and groups are what turn identifiers into names. Without them your tables reference spaces by key and authors by account identifier, which is technically complete and practically unusable.
Daily or weekly is plenty. A wiki changes at the speed of people writing documentation, and there is nothing to gain from syncing hourly.
What can five streams actually tell you?
More than the number suggests, because the pieces join up. Spaces give you the top-level structure. Pages and blog posts carry titles, authors, timestamps and the space each belongs to. Groups tell you who is organised how, and audit records, where available, describe what changed.
Join those and you can find spaces nobody has touched in a year, pages whose author has left the company, documentation concentrated in one team, and how creation rates have moved across quarters. That is the governance picture, and it is the thing nobody can assemble by browsing.
What it cannot tell you is whether a page is any good, or what it says. Stale and useful are not the same thing, and a page untouched for two years may be perfectly correct. Treat the output as a list of candidates for human review rather than a deletion queue.
Why is there no type conversion to worry about?
Because Confluence's API returns data in types that already correspond to JSON Schema, which is what Airbyte's protocol uses. So no conversion step is applied, and what you see in Databricks is what Confluence said rather than a best guess at what it meant.
That is a genuine convenience and worth appreciating given how much of this series is about sources where the opposite is true. There is no sampling, no inference and no sparse-field problem, because the API declares its types rather than leaving them to be discovered.
The modelling work that remains is joining rather than casting. Build a table resolving page records to space names and author names, and put your staleness definition in one place, because otherwise two teams will disagree about what counts as abandoned and both will be citing the same dataset.
Frequently asked questions
Does this give me the content of my pages?
Not as a searchable corpus. The streams describe structure and activity. For search over the prose, a vector store is the right destination.
Why can I not sync the audit stream?
Audit records require a Standard or Premium Atlassian plan. On lesser tiers that stream is unavailable regardless of configuration.
Does this work with Confluence Data Center?
No. The connector works against Confluence Cloud rather than Data Center or Server installations.
Do I need to cast the columns downstream?
Generally not. Confluence returns types corresponding to JSON Schema, so no conversion is applied. The modelling work is joining rather than casting.
Can I do this without writing code?
Yes. The setup above is entirely UI-driven. You will want a Spark or SQL model joining pages to spaces and authors, and defining what counts as stale.
Get your Confluence data into Databricks
Confirm you are on Confluence Cloud and check your plan before promising audit data, take the reference streams so identifiers resolve to names, define staleness once in a model, and treat the result as a list for human review rather than a deletion queue.
Airbyte's connector catalog includes 700+ pre-built connectors, so documentation activity can sit beside delivery and collaboration sources. For the same source in a warehouse, see Confluence to Snowflake, and for another wiki whose streams describe structure rather than prose, Notion to MySQL.
Integrate with 700+ apps using Airbyte
Move data from 700+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.