n8n to Elasticsearch: How to Index Your Data

Syncing n8n to Elasticsearch makes your workflow execution history searchable. Error messages, node output, and run metadata become full-text queryable, which is a better fit for debugging than SQL is: you rarely know the exact string to filter on when you are trying to work out why something broke last Tuesday.

The n8n source can only do full refresh syncs, which normally means duplicated data piling up in the destination. Elasticsearch has a specific setting that solves this, and turning it on is the single most important decision in this setup.

Why upsert matters so much here

The n8n source supports full refresh only. Every sync re-reads every execution record from the API. With most destinations that leaves you choosing between overwriting the table each run or accumulating duplicates forever.

The Elasticsearch destination offers a third option. When you configure it, you choose whether to use the upsert method when indexing new documents. With upsert enabled, a record that already exists updates the existing document rather than creating a second copy. Re-reading the same execution on every sync stops being a problem, because each one simply overwrites itself in place.

This is worth appreciating in contrast with a streaming destination. Publishing the same data to Kafka republishes every message, and consumers must deduplicate downstream because a topic cannot overwrite its own contents. An Elasticsearch index can. That makes it a considerably better fit for a full-refresh source than a log-structured destination is.

Leave upsert off and you get duplicate documents for every execution on every sync, which will quietly inflate your index and skew any count you run against it.

What you can and cannot sync

The source exposes exactly one stream: executions. Workflow definitions, credentials, users, and tags are not available through this connector. If you need those, the Connector Builder is the route to a custom n8n connector.

The stream becomes an index, and each record becomes a document in it. That mapping is straightforward, but the destination supports namespaces, so you can control where the index sits if you are running several pipelines into the same cluster.

Two support facts are worth knowing before you plan around this pipeline. Both connectors are Marketplace rather than certified, and both carry a low sync success rate in Airbyte's own metadata. The destination is also available on Airbyte Core and PyAirbyte but not on the Standard, Plus, Pro, or Enterprise Flex plans, so check your plan covers it before designing anything. Elasticsearch 7.x or later is required, and the connector sits at version 0.2.0. Treat monitoring as part of the build rather than something to add later.

Setting up the pipeline

In n8n, open Settings, then API, and create an API key. Configure the source with that key and your instance host. Self-hosted and cloud instances both work.

The destination has six configuration fields. Most are optional, but which ones you need depends on how your cluster is secured.

Three of those rows cause most of the friction in practice.

• Index creation permissions. If you use authentication, the user needs permission to create an index when one does not already exist, as well as to create documents. A user that can write documents but not create indices will fail on the first sync into a new index.
• The CA certificate. For a secured cluster you may need to supply the certificate, which lives on the Elasticsearch server at /usr/share/elasticsearch/config/certs/http_ca.crt. On a containerised cluster you can copy it out with a docker cp command against the container name.
• Network access. An SSH tunnel through a bastion host is supported if the cluster is not directly reachable. Otherwise add the Airbyte Cloud IP addresses to whatever restricts inbound traffic.

Then create the connection, select the executions stream, and set a frequency. Because every sync re-reads the full history, daily or a few times a day is usually the right cadence rather than hourly.

What about performance?

The connector performs bulk writes, which is the right approach for indexing throughput. Airbyte's documentation notes that large records may impact performance and that the connector does not currently support variable batch sizes.

That matters for this particular source because n8n execution records can carry substantial nested payloads, especially where workflows move large objects between nodes. If you see indexing slow down or the cluster show strain, the size of individual execution documents is the first thing to look at, and reducing what your workflows retain in execution data is more effective than tuning the pipeline.

Frequently asked questions

Should I enable upsert?

Yes, for this pipeline. The n8n source re-reads every execution on every sync, so without upsert you accumulate a duplicate document per execution per run. With it, records update in place.

Why did my first sync fail on permissions?

The authenticated user probably cannot create indices. Writing documents and creating an index are separate permissions, and the connector needs both when the target index does not yet exist.

Can I sync anything besides executions?

No. The n8n source exposes a single executions stream. Workflow definitions, credentials, and users are not available through it.

Where do I get the CA certificate?

From the Elasticsearch server itself, at /usr/share/elasticsearch/config/certs/http_ca.crt. On a dockerised cluster, copy it out of the container with docker cp before configuring the destination.

Is Elasticsearch better than a warehouse for this data?

It depends on the question you are asking. Elasticsearch suits searching error text and investigating specific failures. For counts, failure rates, and trends over time, a columnar store answers faster and more cheaply.

Get your n8n data into Elasticsearch

Enable upsert, check index creation permissions, and set a cadence that respects the full refresh behaviour. Want aggregate reporting rather than search? See our guides to n8n to ClickHouse and n8n to Kafka.

Start syncing now →