MySQL to Weaviate: How to Move Your Data

Moving MySQL into Weaviate turns rows your application already stores into something searchable by meaning. Product descriptions, support notes and knowledge articles sit in tables where the only way to find them is an exact match or a LIKE clause somebody regrets writing.

This guide covers the managed path with Airbyte. Two things shape the build: a default on managed MySQL will break change capture the first time the pipeline pauses, and a table row is not automatically a document worth embedding.

MySQL to Weaviate at a glance:

Why move data from MySQL to Weaviate?

Two situations account for most of these pipelines.

The first is semantic search over content you already hold. A catalogue, a help centre or a case archive lives in MySQL because that is where the application writes it, and finding the most relevant entries for a vaguely worded question is something no relational query will do well.

The second is building a retrieval layer for a language model over your own records. The distinction worth drawing is between meaning and words: if people search with terms that appear in the text and expect exact matching with filters, that is keyword search and MySQL to Elasticsearch does it better and more cheaply than embedding everything.

What do you need before you start?

Four things, and the first is a setting rather than a credential:

Binlog retention set to something sensible. On RDS it defaults to zero, and Airbyte recommends 168 hours. A user with SELECT, REPLICATION CLIENT and REPLICATION SLAVE is the other half. The MySQL source documentation covers the grants and the setting.

A view that defines a document. Rarely is one row one document. Usually the text worth embedding is spread across columns, or across a table and its children, and a view is where you assemble that into the thing a search should return.

A Weaviate instance on 1.21.2 or later. Earlier versions are not supported. You need its URL and credentials, whether hosted or run yourself.

An embedding provider, and a sense of the bill. Embedding is charged by token, and a large table of text is a lot of tokens. Knowing roughly how much text you are about to send is worth ten minutes before committing to it.

If your database restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.

How do you build a MySQL to Weaviate pipeline in Airbyte?

Step 1: Fix binlog retention before anything else

On RDS the retention is zero by default, which means binlogs are discarded as soon as they are no longer needed internally and your pipeline has nothing to resume from after any interruption. Set it to 168 hours, giving a week of margin, and do it now rather than after the first failure. This costs one command and prevents the specific situation where a paused pipeline cannot restart without a full resnapshot.

Step 2: Configure the MySQL source

Click Sources in the left navigation, then New Source, and select MySQL, following adding a source. Supply the host, port, database and credentials, then select your view rather than the underlying tables. If any table still uses MyISAM, be aware the initial snapshot can lock it, so schedule that first sync considerately.

Step 3: Configure the Weaviate destination

Click Destinations, then New Destination, and select Weaviate, following adding a destination. Supply the instance URL and authentication, then nominate which columns are text and which are metadata, and set your chunk size. Chunks are counted in tokens with 8,191 the ceiling, so the figure does not translate into characters.

Step 4: Create the connection and test before you pay

Click Connections, then New connection, select your stream and a sync mode. Run the first pass with the fake embedding option, which fills vectors with random numbers, and confirm your objects, chunks and metadata are shaped correctly. Then switch to a real provider and sync again, having verified the structure without paying for it.

Weaviate assigns its own identifier to each object and keeps the Airbyte record identifier in a separate property, so join back through that rather than assuming your primary key became the object identifier.

Why does change capture stop working after a pause?

Because on RDS the binlog retention parameter defaults to zero, and change capture depends entirely on those logs still existing when the pipeline next runs. With retention at zero they are removed as soon as the database has finished with them internally, which is usually long before your pipeline comes back.

The result catches people out because everything works perfectly until the first interruption. A maintenance window, a failed sync, a connection paused while somebody investigates something unrelated, and the pipeline cannot resume from where it stopped. The remedy is a full resnapshot, which on a large table is both slow and heavy on the database you were trying not to disturb.

Setting retention to 168 hours gives a week of margin, which comfortably covers a weekend outage and most investigations. It is the single most valuable thing to do before the first sync on any managed MySQL, and it is worth checking rather than assuming somebody set it during an earlier project, since the default quietly reasserts itself on new instances.

What is a document when your source is a table?

Whatever you decide it is, and the default answer of one row per object is often wrong. A product row holds a name, a short description and a long description in three columns, and a search should probably treat those as one piece of text rather than embedding a column in isolation. Assembling that is a job for a view in MySQL, where the logic is visible and reviewable.

Relational structure makes this sharper than it is for a document source. The text worth searching frequently spans a parent and its children, such as a case and its notes, and Weaviate will not join anything. Either denormalise in the view so each object carries its full context, or accept that a search returns fragments and your application reassembles them afterwards.

Then classify columns from the queries rather than from the schema. Anything somebody might phrase a question about belongs in text, where it is chunked and embedded. Anything used to narrow results, such as status, category, region or a date, belongs in metadata, which is filterable and never searched by meaning. Putting a searchable column in metadata is the quiet failure here, because results still appear and are simply drawn from a smaller pool than anybody intended.

Frequently asked questions

Why can my pipeline not resume after being paused?

Binlog retention is probably zero, which is the RDS default. Set it to 168 hours as recommended, and expect a full resnapshot to recover this time.

Should each row become one object?

Only if a row is genuinely the thing somebody searches for. More often the text spans several columns or several tables, which makes a view the right place to assemble a document.

Why does searching one column return nothing?

It was probably classified as metadata, which is filterable and never embedded. Metadata columns are invisible to semantic search regardless of what they contain.

How do I avoid paying to test this?

Use the fake embedding option for the first pass. The vectors are random and meaningless, and the shape of your objects, chunks and metadata is real and checkable.

Can I do this without writing code?

The pipeline, yes, though the binlog setting is database administration. The view assembling your documents is SQL and it is the part that decides whether the search is any good.

Get your MySQL data into Weaviate

Set binlog retention to 168 hours before the first sync, because the RDS default of zero turns any pause into a full resnapshot. Build a view that assembles a document rather than assuming a row is one, since relational text usually spans columns and tables that Weaviate will not join. Classify columns from the queries you expect, and test the whole thing with fake embeddings before paying to vectorise anything.

Airbyte's connector catalog includes 600+ pre-built connectors, so application data can power semantic search without a bespoke indexer. For a comparable relational source into the same destination, see PostgreSQL to Weaviate, and for a document source into the same destination, MongoDB to Weaviate.

Start syncing now →