Parquet File to Weaviate: How to Move Your Data

Moving Parquet files into Weaviate turns a folder of columnar data into a searchable vector index. Parquet stores text efficiently and carries its schema inside the file, but it has no notion of similarity: finding the three most relevant support tickets means reading every row of every file. Weaviate answers that question in a way a file format cannot.

This guide covers the managed path with Airbyte. Two decisions shape the build, and both are much easier to make now than to unpick later: which of your fields count as text and which count as metadata, and how you split files that do not agree on a schema.

Parquet File to Weaviate at a glance:

Why move data from Parquet File to Weaviate?

Two situations account for most of these pipelines.

The first is a corpus that already sits in object storage. Exported documentation, transcripts, scraped articles and support histories tend to land as Parquet because that is what the pipeline upstream produced. The text is right there, the analytics tools around it can filter and count it, and none of them can retrieve by meaning. Building a retrieval layer over files you already hold is considerably cheaper than extracting everything again from the original systems.

The second is adding similarity search beside analytics you already run. A product catalogue or ticket archive is queried by exact match today, and somebody asks for things like this one. Weaviate holds that alongside the filters you rely on. The pairing stops making sense when your Parquet files carry no meaningful free text. Numeric fact tables, event logs and dimensional exports gain nothing from being embedded, and you pay per token to discover it; those belong in a column store, and Parquet File to ClickHouse covers that route.

What do you need before you start?

Four things, and one of them is a decision rather than a credential:

A storage connector, not a Parquet connector. There is no standalone Parquet source. You choose the connector for wherever the files sit, such as S3, Azure Blob Storage, Google Cloud Storage or SFTP Bulk, and set the file format to Parquet. The S3 source documentation covers the file-based configuration that these connectors share.

A Weaviate instance on 1.21.2 or later. Earlier versions are not supported. You need the instance URL and its credentials, whether that is Weaviate Cloud or a deployment you run yourself.

A decision about which fields are text and which are metadata. Text fields are chunked and embedded, and they are what a similarity query actually searches. Metadata fields are stored and can be filtered on, but they are never embedded. Changing your mind later means reindexing, so it is worth ten minutes now.

An embedding provider, or vectors you already have. OpenAI or Cohere with an API key, precomputed vectors read from a field, or no embedding at all if you intend to vectorise inside Weaviate. This is the part of the pipeline with a bill attached, charged by token, so know roughly how much text you are about to send.

If your storage account or Weaviate instance restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.

How do you build a Parquet File to Weaviate pipeline in Airbyte?

Step 1: Split your files by glob before configuring anything

Parquet carries its schema inside each file, which is a gift until two files disagree. A column added last quarter, or a type that shifted from integer to string, cannot be reconciled into a single stream. Work out which groups of files genuinely share a schema and give each group its own glob pattern and its own stream. Remember that a glob must include the subfolder path, and that a double asterisk matches recursively. Doing this first is far less painful than discovering it halfway through a sync.

Step 2: Configure the Parquet source

Click Sources in the left navigation, then New Source, and select the connector for your storage, following adding a source. Supply that store's credentials, then add one stream per glob group with the format set to Parquet. Because the schema is declared inside the files, you are confirming types here rather than watching the connector guess at them from a sample.

Step 3: Configure the Weaviate destination

Click Destinations, then New Destination, and select Weaviate, following adding a destination. Supply the instance URL and authentication. This is where you nominate the text fields and the metadata fields, and set your chunk size. Chunks are counted in tokens using tiktoken, with 8,191 the ceiling, so the number does not translate neatly into words or characters.

Step 4: Create the connection and choose an embedding

Click Connections, then New connection, select your streams and a sync mode. The embedding choice sits here and it is the decision that costs money. Run the first pass with the fake embedding option, which fills vectors with random numbers, and confirm your objects, chunks and metadata land the way you expected. Then switch to the real provider and sync again.

Weaviate generates its own UUID for every object and stores the Airbyte record identifier in a dedicated property. Join back through that property rather than assuming your primary key became the object identifier, because it did not.

What happens when your Parquet files disagree about schema?

Nothing reconciles them. The schema lives inside each file rather than in a catalogue above the files, so two objects in the same folder can legitimately declare different columns, or different types for a column of the same name, and there is no authority to arbitrate between them.

This is the flip side of what makes Parquet pleasant to load. Types are declared rather than sampled, so a column that says it holds a timestamp holds a timestamp, and you skip the inference step that CSV and JSON sources need. That guarantee is per file, not per folder, and nothing about the format extends it.

The practical answer is the glob pattern. Rather than pointing one stream at everything and hoping, split along whatever boundary your schema changes actually follow: a date prefix, an export version, a source system. Each glob becomes a stream whose shape holds still, which is what the destination needs and what anyone querying it later will assume.

Why does the text and metadata split decide what you can search?

Because Weaviate treats the two differently at ingest, not at query time. Text fields go through chunking and embedding and become vectors. Metadata fields are stored beside the object and can be filtered on, but they are never chunked, never embedded and never reached by a similarity query.

So a field you classified as metadata is invisible to the search most people build this pipeline for. If ticket subject lines went in as metadata, find tickets like this one will never match against them, and the failure is quiet. You still get results. They are simply drawn from a smaller pool than you assumed, and nothing in the output says so.

Decide this from the queries you intend to run rather than from how the columns look. Anything somebody might phrase a question about belongs in text; anything used to narrow results, such as region, product line, date or status, belongs in metadata. Chunk size feeds into the same decision, because a long document split across several chunks retrieves by fragment rather than as a whole. That is usually what you want, but it changes what a match means.

Frequently asked questions

Is there a Parquet source connector?

No. You configure the connector for wherever the files live and set the format to Parquet, so your storage system decides which connector you end up using rather than the file format.

How large can a chunk be?

Up to 8,191 tokens. Length is counted in tokens with tiktoken rather than in characters, so the figure does not map cleanly onto a word count and is worth checking against a sample of your longest documents.

Can I use embeddings I already have?

Yes. The From Field option reads vectors already present in your data, which suits cases where you generated them elsewhere and need consistency with an existing index.

How do I test this without paying for embeddings?

Use the fake embedding option. It generates random vectors, so search results are meaningless, but the structure of your objects, chunks and metadata is real and you can check all of it before spending anything.

Can I do this without writing code?

Yes. The setup above is entirely UI-driven. What remains is judgement rather than code: which fields are text, which are metadata, and where your globs divide.

Get your Parquet File data into Weaviate

Split by glob before anything else, because drift between files is the failure that costs most to unpick. Classify text and metadata from the queries you mean to run, not from how the columns happen to look. Test with fake embeddings, confirm the objects are shaped correctly, and only then pay to vectorise the real thing.

Airbyte's connector catalog includes 600+ pre-built connectors, so files sitting in object storage can reach a vector store without a bespoke loader. For the same files in a keyword search engine, see Parquet File to Elasticsearch, and for a text-heavy source into the same destination, Confluence to Weaviate.

Start syncing now →