Parquet to Elasticsearch: How to Index Your Files
Load Parquet files into Elasticsearch with Airbyte. Columnar to document mapping, index size expansion, upsert behaviour, and why to define mappings first.
Parquet and Elasticsearch disagree about what data is for. Parquet stores columns, compressed and typed, so an engine can scan one field across millions of rows without touching the rest. Elasticsearch stores documents, inverted-indexed, so it can find the handful of records matching a phrase.
Moving between them is not really a format conversion. It is a change of purpose, and most of the decisions in this pipeline follow from that.
What happens when columns become documents?
Each source stream becomes an index. Each row becomes a document. Each column becomes a field in that document. A Parquet file of ten million rows and forty columns becomes ten million documents with forty fields each.
The size implication is immediate and it is the most common surprise here. Parquet's compression depends on column locality, storing similar values next to each other. Documents destroy that locality by design, and Elasticsearch then adds an inverted index on top. Expect the indexed footprint to be a multiple of the Parquet file size rather than a fraction of it, and plan capacity accordingly. The cheapest mitigation is to select only the columns you will actually query.
How do Parquet types map to Elasticsearch fields?
The Elasticsearch destination maps text to string, date to date-time, object to object, array to array, boolean to boolean, and both integer and number onto numeric field types.
Parquet's type system is richer than what survives that trip. Parquet distinguishes 32-bit from 64-bit integers and carries decimal logical types with explicit precision and scale. Those distinctions flatten. If your files hold financial decimals, verify what actually lands in the index before you trust an aggregation over it.
The larger risk is dynamic mapping. If you let Elasticsearch infer the mapping, it guesses from the first documents it sees. A field that is null in the first batch and populated later, or numeric early and string later, produces a mapping that either rejects subsequent documents or quietly coerces them. Define the index mapping explicitly before the first sync. It costs ten minutes and prevents the failure where a sync succeeds for a week and then starts rejecting records.
How do you avoid duplicate documents?
The destination supports full refresh overwrite, full refresh append, and incremental append. Neither deduped mode is available.
The compensation is a configuration option called Upsert Records. With it enabled, the connector indexes using the record's identifier, so re-indexing the same document replaces it instead of adding a second copy. That is how you get idempotent reloads without a deduped sync mode, and for a file-based source where the same file may be read more than once, it is usually what you want. Decide on this during setup rather than after, because switching it later changes what a re-sync does to an index that already holds data.
Writes are performed in batches. The documentation notes that large records may affect performance and that variable batch sizing is a wanted enhancement, so unusually wide rows are worth testing before committing to a full load.
Setting up the source and destination
On the source side, point the file source at your Parquet, choose Parquet as the format, and select the storage provider. For a public HTTPS file that is all that is needed. For object storage you will supply credentials as well.
On the destination side you need Elasticsearch 7.x or above and an endpoint URL. Authentication is optional, but if you use it, the account needs permission to create an index if one does not exist and to create documents. Basic auth and API key ID with secret are both supported, and a CA certificate can be supplied where your cluster requires one.
If your Elasticsearch instance sits on a private network, the connector can reach it through an SSH tunnel to a bastion host, using either an RSA private key or password authentication. Note that the login username is the operating system user on the bastion, not an Elasticsearch account, which is a common misconfiguration.
Frequently asked questions
Why is my index so much larger than the Parquet file?
Parquet compresses by column, and Elasticsearch stores documents plus an inverted index. The expansion is expected rather than a fault. Reduce it by selecting only the columns you will query.
How do I stop duplicate documents on a re-sync?
Enable Upsert Records so the connector replaces documents by identifier instead of appending. There is no deduped sync mode on this destination, so that setting is the mechanism.
Should I let Elasticsearch infer the mapping?
No. Define it before the first sync. Inferred mappings are guessed from early documents and break later when a field's observed type changes.
Can I connect to a private Elasticsearch cluster?
Yes, through an SSH tunnel to a bastion host with either key or password authentication, configured on the destination alongside a CA certificate if needed.
Get your Parquet data into Elasticsearch
Define the mapping first, turn on upserts, and size the cluster for documents rather than for compressed columns. If the goal is analytics rather than search, a columnar destination keeps Parquet's shape intact, so see our guide to Parquet to BigQuery. Indexing from a different source instead? See n8n to Elasticsearch.
Integrate with 600+ apps using Airbyte
Move data from 600+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.