Confluence to Weaviate: How to Move Your Data
Move Confluence into Weaviate with Airbyte. Chunking wiki pages for retrieval, the text versus metadata split, and filtering out stale documentation.
Moving Confluence into Weaviate is how a wiki becomes something an assistant can answer from. Your documentation holds the accumulated knowledge of the company and is famously hard to search, and retrieval augmented generation needs that content as vectors rather than as pages.
This is not replication, and treating it as though it were is the fastest way to build a retrieval system nobody trusts. Your pages are chunked, converted into vectors by an embedding model, and indexed for similarity search. What lands is not your wiki.
Confluence to Weaviate at a glance:
Why move data from Confluence to Weaviate?
Two situations account for most of these pipelines, and both are about retrieval rather than reporting.
The first is an internal assistant answering questions from your own documentation. Confluence is where policies, runbooks and architectural decisions live, and an assistant that cannot cite them is guessing. Vectors are what let it find the right paragraph rather than the right page.
The second is semantic search in a product or portal, where people describe what they need rather than knowing which page it is on. If instead you want to know which spaces are abandoned or who writes most, that is governance and a warehouse is the right destination.
What do you need before you start?
The Confluence side is simple and the Weaviate side has a running cost:
A Confluence Cloud instance and an API token. The connector works against Cloud rather than Data Center or Server installations. Use the token with the account email as basic auth, alongside your domain name. The Confluence source documentation covers the requirements.
A service account with deliberately chosen access. The token inherits the permissions of the account that created it, and on a retrieval system that becomes a security property: anything the account can read may end up quoted back to whoever asks the assistant a question.
A Weaviate instance on 1.21.2 or later. Self-hosted or through Weaviate Cloud Services, with its URL and credentials. The Weaviate destination documentation lists the configuration fields.
An embedding decision, and a budget for it. Embed through OpenAI or Cohere with an API key, load precomputed vectors from a field using the From Field option, or choose No Embedding if your Weaviate class already has a vectorizer. Fake embeddings exist for testing and generate random vectors, which is useful before you spend anything.
Finally, a decision about which spaces are in scope, because a company wiki contains a great deal that nobody should be surfacing through an assistant.
How do you build a Confluence to Weaviate pipeline in Airbyte?
Step 1: Decide what a retrievable unit is
A Confluence page is rarely the right answer on its own, because pages vary from a two-line note to a fifty-section handbook. Decide whether you want whole pages chunked into passages or something narrower, and note which fields carry the prose people search and which carry attributes they filter by.
Step 2: Configure the Confluence source
Click Sources in the left navigation, then New Source, and select Confluence, following adding a source. Supply the API token, the email address of the account that created it, and your domain name. Select the pages and blog posts streams, since those carry the content you are indexing.
Step 3: Configure the Weaviate destination
Click Destinations, then New Destination, and select Weaviate, following adding a destination. Supply the cluster URL and credentials, pick your embedding method, then configure processing: which fields are text, which are metadata, and the chunk size. Chunk length is measured in tokens using the tiktoken library, up to a maximum of 8,191.
Step 4: Test with fake embeddings first
Click Connections, then New connection, select your streams and a sync mode. Run the first sync with fake embeddings to confirm the chunking and field split look right in Weaviate, then switch to your real provider. One extra sync can save a large embedding bill spent on a misconfigured selection.
Note that the destination generates a UUID for each Weaviate object rather than letting you nominate an identifier, storing the Airbyte record identifier in a dedicated property so you can trace an object back to its source page.
Why is a wiki page a difficult retrievable unit?
Because Confluence pages have no consistent size or shape. A support runbook might be three paragraphs, while an engineering handbook runs to dozens of sections covering unrelated topics under one title. Chunking both the same way serves neither well.
Chunk too large and a match returns a wall of text where only one paragraph was relevant, which an assistant then has to summarise and often gets wrong. Chunk too small and a passage loses the context that made it meaningful, so a retrieved fragment about restarting a service does not say which service.
The practical approach is to start moderate, retrieve some real questions your colleagues have actually asked, and read what comes back. Chunk size is one of the few settings where the only honest tuning method is looking at results, and it is worth doing before anyone builds an assistant on top.
What should be text and what should be metadata?
Text fields are concatenated and chunked, and those chunks are what gets embedded and searched. Metadata is stored as properties and used for filtering only, so it can narrow a result set and can never influence which chunks match.
On a wiki the page title deserves particular thought. It is short, it is often the most descriptive summary of what a page contains, and treating it as metadata means a search for its exact subject may not match the page it names. Including it in the text is usually right.
Space, author and last-updated date belong in metadata, where they let you restrict retrieval to one space or exclude anything not touched in three years. That second filter is worth having, because a retrieval system happily quotes an outdated runbook with complete confidence.
Frequently asked questions
Does this work with Confluence Data Center?
No. The connector works against Confluence Cloud rather than Data Center or Server installations.
Why does filtering on a field work but searching it does not?
It is configured as a metadata field. Metadata is stored for filtering and is not embedded, so move it into the text fields if it should affect relevance.
What chunk size should I use?
There is no universal answer. Start moderate, retrieve against real questions and read the results, since pages vary enormously in length and the only honest tuning method is inspection.
How do I stop the assistant quoting outdated pages?
Keep the last-updated date as a metadata field and filter on it at retrieval time. Nothing in the pipeline distinguishes a current runbook from an abandoned one.
Can I do this without writing code?
Yes. Chunking, embedding and indexing are all configured in the destination interface, and the field split is a selection rather than a transformation.
Get your Confluence data into Weaviate
Scope the service account carefully, because on a retrieval system its permissions decide what an assistant can quote. Put the page title in the text rather than metadata, keep the last-updated date for filtering out stale pages, validate with fake embeddings before spending anything, and tune chunk size by reading real results.
Airbyte's connector catalog includes 600+ pre-built connectors, so the same wiki can feed a vector store and a warehouse at once. For the governance view of the same source, see Confluence to Databricks, and for another document source feeding the same destination, MongoDB to Weaviate.
Integrate with 700+ apps using Airbyte
Move data from 700+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.