PostgreSQL to Weaviate: How to Move Your Data
Move PostgreSQL into Weaviate with Airbyte. Chunking and embedding explained, why metadata fields are filter-only, and what a full refresh costs you.
Moving PostgreSQL into Weaviate is not replication, and treating it as though it were is the fastest way to build a retrieval system nobody trusts. Every other pipeline in this series copies rows from one place to another. This one transforms them: your records are chunked, converted into vectors by an embedding model, and indexed for similarity search.
What lands in Weaviate is therefore not your table. It is a set of text chunks with vectors attached and some of your columns alongside as properties. This guide covers the managed path with Airbyte, and it spends its time on the two decisions that determine whether retrieval works: which fields become text, and which become metadata.
PostgreSQL to Weaviate at a glance:
Why move data from PostgreSQL to Weaviate?
Two situations account for most of these pipelines.
The first is retrieval augmented generation. You want an assistant that answers questions using your own content, and that content sits in Postgres: support articles, product descriptions, policy documents, ticket histories. Semantic search over it needs vectors, and building the chunking and embedding yourself is a pipeline you then own forever.
The second is semantic search in a product. Users searching by meaning rather than keyword, or a recommendation feature matching similar items. Both want the same thing, which is your relational content living somewhere that understands similarity. Neither wants a copy of your schema.
What do you need before you start?
One of these is a running cost rather than a one-off, which is unusual for this series:
A Weaviate instance on 1.21.2 or later. Self-hosted or through Weaviate Cloud Services, along with its URL and credentials, either an API token or a username and password. The Weaviate destination documentation lists the configuration fields.
An embedding decision, and a budget for it. You can embed through OpenAI or Cohere with an API key, load precomputed vectors from a field in your source using the From Field option, or choose No Embedding if your Weaviate class already has a vectorizer configured. There are also fake embeddings for testing, which generate random vectors and are useful for validating the pipeline shape before spending anything.
A decision about which columns are content. Text fields are concatenated and embedded. Metadata fields are stored alongside as properties. Getting this split right is the whole job, for reasons in the next section.
Logical replication on PostgreSQL, if you want change capture. A replication slot and a publication, plus monitoring on that slot, because an unconsumed slot causes write-ahead log to accumulate on the source database until the disk fills.
If your organisation restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list on both systems before you begin.
How do you build a PostgreSQL to Weaviate pipeline in Airbyte?
Step 1: Decide what a retrievable unit is
Before touching Airbyte, decide what a good search result looks like. A support article, a product, a paragraph. Then build a view in PostgreSQL producing one row per that thing, with the searchable prose in a small number of columns and the filterable attributes in others. Doing this in the database is easier than fighting the connector's field selection later.
Step 2: Configure the PostgreSQL source
Click Sources in the left navigation, then New Source, and select Postgres, following adding a source. Supply host, port, database, schema and credentials, and choose your replication method. Change data capture applies to tables rather than views, so a view-based approach uses cursor-based incremental instead and needs a maintained timestamp column.
Step 3: Configure the Weaviate destination
Click Destinations, then New Destination, and select Weaviate, following adding a destination. Supply the cluster URL and credentials, pick your embedding method and key, then configure processing: which fields are text, which are metadata, and the chunk size. Chunk length is measured in tokens using the tiktoken library, with a maximum of 8,191, which is the limit of the ada-002 model.
Step 4: Test with fake embeddings, then run properly
Click Connections, then New connection, select your stream and an incremental sync mode. Run the first sync with the fake embeddings option to confirm the chunking and field split produce objects that look right in Weaviate, then switch to your real embedding provider. That costs one extra sync and can save a large embedding bill spent on a misconfigured field selection.
Note that the destination generates a UUID for each Weaviate object rather than letting you nominate an ID field, and stores the Airbyte record identifier in a dedicated property so you can trace an object back to its source row. If you are upgrading from an older connector version, the migration guide covers what changed.
Why is the text and metadata split the whole job?
Because the two are used for completely different things and the distinction is not reversible without re-embedding. All the fields you designate as text are concatenated into a single string and then split into chunks, and those chunks are what gets embedded and searched. Metadata fields are stored as properties on the object and used for filtering.
The consequence people miss is that metadata is not retrievable by similarity. If a product's category matters to how someone searches, putting it in metadata means it can narrow a result set but can never influence which chunks match. If you want the category to affect relevance, it has to be in the text.
A useful rule of thumb: anything a human would type into a search box belongs in text, and anything they would tick in a filter belongs in metadata. Note also that permitted metadata types are limited, so complex values are serialised into their JSON representation rather than kept as structured properties. You can reach nested fields with dot notation, and wildcards let you select across all entries in an array.
What does a full refresh actually cost?
More than on any other destination in this series, because embedding is billed per token by whoever provides the model. A full refresh does not just move the data again, it sends every chunk back through the embedding API and charges you for it.
That changes the sync mode calculation completely. On a warehouse, a nightly full refresh of a modest table is a mild inefficiency. Here it is a recurring invoice for re-computing vectors that have not changed. Use incremental wherever the source supports it, so only new and modified rows are embedded.
Two other levers help. Send fewer fields as text, since concatenating columns nobody searches inflates every chunk and therefore every bill. And think about chunk size: smaller chunks mean more of them per record and finer-grained retrieval, larger chunks mean fewer embedding calls and more context per match. There is no universally right answer, which is another reason to validate with fake embeddings before committing.
Frequently asked questions
Can I use my own precomputed embeddings?
Yes. The From Field embedding option loads vectors already stored in your source, and the field must contain a JSON array of numbers. There is also a No Embedding option if your Weaviate class has its own vectorizer configured.
Why does filtering on a field work but searching it does not?
Because it is configured as a metadata field. Metadata is stored for filtering and is not embedded, so it cannot influence similarity matching. Move it into the text fields if it should affect relevance.
How large can a chunk be?
Chunk length is measured in tokens using the tiktoken library, up to 8,191, which is the maximum supported by the text-embedding-ada-002 model.
How do I trace a Weaviate object back to its source row?
The destination generates a UUID for each object and stores the Airbyte record identifier in a dedicated property, which you can use to identify objects by source record.
Can I do this without writing code?
Yes. Chunking, embedding and indexing are all configured in the destination interface. You will want to write the PostgreSQL view that shapes your retrievable unit, which is where the design work sits.
Get your PostgreSQL data into Weaviate
Decide what a retrievable unit is and build it as a view, split text from metadata knowing that only text is searchable, validate the whole shape with fake embeddings before spending anything, then use incremental sync so you are not paying to re-embed rows that have not changed.
Airbyte's connector catalog includes 700+ pre-built connectors, so the same source can feed a warehouse alongside your vector store. For PostgreSQL into a lakehouse, see PostgreSQL to Databricks, and for a document destination serving an application, PostgreSQL to MongoDB.
Integrate with 700+ apps using Airbyte
Move data from 700+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.