Elasticsearch to Snowflake: How to Move Your Data
Move Elasticsearch to Snowflake with Airbyte. Why there is no incremental sync, how union types land, index selection, and the decimal type decision.
Moving data from Elasticsearch to Snowflake converts search documents into warehouse rows, so log, event, or catalogue data living in an index can be queried with SQL and joined against the rest of your data. Airbyte handles it with no code. One property of the source governs the whole design and is best known before you schedule anything: the Elasticsearch source has no incremental sync.
Every sync reads the entire index. That is perfectly workable for reference data and steadily expensive for a growing log store, so it is the first thing to test your use case against.
Why is there no incremental sync?
The Elasticsearch source documents full refresh sync as supported and incremental sync as not supported. There is no cursor field to configure because the connector does not offer one.
Sync duration and cost therefore scale with the size of the index rather than with how much changed. An index that doubles in size doubles the work of every sync, permanently. That makes scheduling a budgeting exercise: hourly syncs against a large log index cost far more than the freshness is usually worth, and daily or weekly is frequently the honest answer.
It also matters for the cluster serving your production search traffic. Elasticsearch calls may be rate limited depending on your deployment, and a repeated full scan is exactly the read pattern most likely to run into that. Test against a replica or an off-peak window before committing to a frequent schedule.
Use full refresh overwrite as your sync mode. Since every run carries the whole index, append would stack a complete duplicate copy in Snowflake on each sync. If your index names roll by date, selecting only recent indices is the closest practical substitute for genuine incremental movement.
How do Elasticsearch types arrive in Snowflake?
Elasticsearch has no dedicated array type. Any field can hold zero or more values provided they share a type, which means every field is potentially an array, and the connector's type map reflects that honestly.
A long maps to integer or array, a double to number or array, and keyword and text to a union of string, array, number, and integer. Two mappings matter especially: date fields map to string or array rather than a temporal type, and nested fields map to object or string.
In Snowflake this generally means union-typed values landing as VARIANT rather than as clean typed columns. Build a typed model on top of the landed tables, casting and flattening once in a single place, rather than pointing dashboards directly at them and asking every analyst to handle the ambiguity in each query. Note too that Snowflake caps VARCHAR at 16 MB and VARIANT at 128 MB, with oversized values nulled and the change recorded in the row's metadata.
Setting up the connection
The source needs very little: an Elasticsearch endpoint URL, and credentials, which are optional depending on how your cluster is secured. The connector then discovers every index in the domain automatically.
That automatic discovery deserves a second look before you enable streams, because system and internal indices appear alongside your own. Select deliberately rather than enabling everything, since each selected index is a full scan on every run. This is a Marketplace-tier connector with a sync success rate Airbyte reports as low and a usage rate reported as high, so it is well exercised even though it is not first-party maintained.
Loading into Snowflake
The Snowflake destination is Airbyte-supported with a high reported sync success rate and has used direct load since version 4.0.0, writing straight to final tables.
Set the Decimal Data Type option before your first sync. It defaults to FLOAT, which is approximate binary floating point, while NUMBER(38,9) is exact and is what Airbyte's documentation recommends. Changing it later converts the column in place, nulls values with more than 29 integer digits, and cannot recover precision that was never stored, so only a full refresh restores the original numbers.
Create dedicated Snowflake entities using Airbyte's setup script, and give the pipeline its own X-Small warehouse with a sixty-second auto-suspend. Snowflake bills per second and resumes the warehouse on every load, so on a pipeline that repeatedly reloads a full index, idle time and warehouse size are the two costs worth minimising.
Frequently asked questions
Can I sync only new documents?
No. The source supports full refresh only and offers no cursor field. If your index names roll by date, selecting only recent indices is the nearest practical equivalent.
Why are my columns union-typed rather than plain values?
Elasticsearch has no dedicated array type, so any field may hold multiple values and the connector maps types accordingly. Cast and flatten in a downstream model rather than in every query.
Which sync mode should I use?
Full refresh overwrite. Every run carries the entire index, so append would add a complete duplicate copy on each sync.
How do I keep Snowflake costs down here?
Reduce sync frequency, select only the indices and fields you query, and use a dedicated X-Small warehouse with a one-minute auto-suspend.
Get your Elasticsearch data into Snowflake
Schedule around full-scan economics, select indices deliberately, set the decimal type first, and build a typed model over what lands. For the same source into other destinations, see our guides to Elasticsearch to BigQuery and Elasticsearch to ClickHouse.
Integrate with 600+ apps using Airbyte
Move data from 600+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.