Sentry to Elasticsearch: How to Index Your Data

Indexing Sentry into Elasticsearch is worth doing when the question you keep asking is not "how many" but "where else have I seen this". Sentry groups errors into issues using its own fingerprinting, which is excellent until you need to search across the raw text of every stack trace for a library name, a hostname, a customer identifier buried in a tag.

Before any of that, one constraint decides whether this pipeline is available to you at all. The Elasticsearch destination runs on Airbyte Core and PyAirbyte, and it is not offered on the Standard, Plus, Pro or Enterprise Flex plans. If you are on a paid Cloud tier the connector will not appear in your catalogue, and everything below assumes a self-managed deployment.

What you are actually indexing

Sentry events are deeply nested documents. An event carries an exception with a list of stack frames, each with a filename, function, line number and often a snippet of surrounding source. It carries tags, contexts describing the runtime and the device, breadcrumbs leading up to the failure, and user information if you attach it.

Airbyte preserves that structure, which is exactly right for Elasticsearch and exactly why the index mapping deserves a few minutes of thought before the first sync. Dynamic mapping will guess, and its guesses on a document this shaped tend to produce a very wide mapping full of fields nobody queries. Decide which fields need full-text analysis, which should be keyword fields for exact matching and aggregation, and which are noise, and define the mapping before you index a few million documents under a schema you then have to reindex.

Retention still decides your sync mode

Sentry SaaS keeps error events for 30 or 90 days depending on plan. Full Refresh Overwrite replaces your index contents with whatever Sentry currently returns, which means every sync silently discards everything Sentry has already dropped.

Use an append mode. It matters more here than in a warehouse, because a search index is most valuable precisely when it reaches further back than the tool it was built from. An engineer searching for a rare exception signature wants two years of it, not four weeks.

Source configuration and scopes

You need a Sentry auth token, or for Sentry SaaS an internal integration, granting read access on organisations, projects, and issues and events. In scope terms that is org:read, event:read and project:read. A token missing one of the three does not fail the connection test cleanly. It fails that stream with an HTTP 403 while the rest of the sync succeeds, leaving you a partially populated index and a green connection.

The source takes an organisation slug and a project slug, and the project setting determines which project the events, issues and project_detail streams read. There is no wildcard, so a fleet of Sentry projects means a fleet of sources. Decide early whether they write into one shared index with a project field, which makes cross-service search trivial, or separate indices per project, which makes retention policies and access control easier. Changing your mind later means a reindex. Set the host name to your region-specific domain if you use one, without a scheme or trailing path, and lower the connection's concurrent workers if backfilling several projects at once starts returning HTTP 429.

Frequently asked questions

Why can I not find the Elasticsearch destination in Airbyte Cloud?

It is available on Airbyte Core and PyAirbyte, and not on the Standard, Plus, Pro or Enterprise Flex plans. This pipeline needs a self-managed deployment.

Should I use one index per Sentry project or one shared index?

One shared index with a project field makes cross-service search simpler. Separate indices make retention and access control simpler. Decide before the first sync, because switching means reindexing.

Do I need to define an index mapping first?

Strongly recommended. Sentry events are deeply nested, and dynamic mapping will produce a sprawling schema you will want to change once real queries start arriving.

Can Airbyte index Sentry events older than my retention window?

No. The connector reads what Sentry still holds, so the index starts accumulating from your first sync onwards.

Get your Sentry data into Elasticsearch

Check your Airbyte plan first, because that decides everything else. Then spend the time on the index mapping rather than the connector, set an append sync mode so the index outlives Sentry's retention, and choose your index layout before the first million documents rather than after.

If SQL aggregation over error history matters more than free-text search, Sentry to BigQuery covers the same source and a different destination. For the mapping and plan constraints seen from a relational source, see PostgreSQL to Elasticsearch.

Start syncing now →