Sentry to BigQuery: How to Archive Your Data
Move Sentry into BigQuery with Airbyte. Why 30 or 90 day retention makes your warehouse the archive, the three required scopes, and the projects stream change.
Moving Sentry into BigQuery is a retention problem before it is an analytics one. Sentry SaaS retains error events for 30 or 90 days depending on your plan, so any question spanning a year, or correlating this quarter's incidents with a release from last spring, cannot be answered from Sentry alone.
This guide covers the managed path with Airbyte. Your warehouse becomes the long-term record, which makes setting this up early more valuable than it first appears, and there is one breaking change in the connector's history worth knowing about before you upgrade anything.
Sentry to BigQuery at a glance:
Why move data from Sentry to BigQuery?
Two situations account for most of these pipelines.
The first is retention. With error events kept for 30 or 90 days, any trend analysis across quarters is impossible in Sentry itself. A warehouse accumulating issues and events answers it, and the value compounds the longer the pipeline has been running.
The second is joining. Errors against deployments, support tickets against releases, incident volume against engineering effort. Sentry holds one side of each and the value appears only next to something it has never seen.
What do you need before you start?
Short list, and one item makes diagnosis unusually easy when you get it wrong:
An auth token with three read scopes. Created in Sentry under account settings and auth tokens. To sync every stream, grant organisation read, project read and event read. The Sentry source documentation covers the setup.
Your organisation slug. The connector is scoped to one organisation, which since version 1.0.0 also determines which projects the projects stream returns.
Your plan's retention period. Thirty or ninety days, and it sets the outer bound on how long you can go between syncs before losing events permanently. Worth looking up rather than assuming.
A BigQuery service account and dataset. With the BigQuery User and BigQuery Data Editor roles. Set the dataset location to match your deployment and incident datasets, since joining them is the point.
If your organisation restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.
How do you build a Sentry to BigQuery pipeline in Airbyte?
Step 1: Create the token with all three scopes
Grant organisation read, project read and event read together rather than incrementally. This connector is unusually pleasant to debug, because a token missing a scope causes the corresponding stream to return an HTTP 403 rather than quietly producing nothing, but it is still faster to grant them all at once.
Step 2: Configure the Sentry source
Click Sources in the left navigation, then New Source, and select Sentry, following adding a source. Supply the auth token and your organisation slug. If you run self-hosted Sentry, set the API base URL to your own instance rather than the default.
Step 3: Configure the BigQuery destination
Click Destinations, then New Destination, and select BigQuery, following adding a destination. Supply the project, dataset and service account key. Batched standard inserts is the default and is adequate, though a noisy application producing large event volumes may justify Cloud Storage staging.
Step 4: Sync comfortably inside your retention window
Click Connections, then New connection, select your streams and a sync mode. Frequency here is not about dashboard freshness, it is about staying ahead of a retention window that keeps moving. Daily against thirty days leaves ample margin; a connection left broken over a long holiday does not.
Rate limits are returned in response headers and vary by endpoint. If syncs receive 429 responses, reduce the number of concurrent workers rather than assuming something is broken.
Why does retention change how you treat this pipeline?
Because it makes this an archive rather than a replication. Airbyte can only read events Sentry still holds, so your BigQuery history starts on the day of your first sync and nothing recovers what has already aged out.
That argues for setting this up before anybody asks for it. The value of an error archive is entirely in its age, and the difference between configuring it this quarter and next is a quarter of history you will never have when somebody wants to know whether a class of failure has been getting worse.
It also makes a paused connection unusually expensive. On most pipelines a pause produces staleness you catch up on. Here, if the pause outlasts your retention window, those events are gone from the source as well, so treat a stalled sync as an incident rather than a backlog.
What changed in the projects stream?
Version 1.0.0 migrated the projects stream to an organisation-scoped endpoint, because Sentry deprecated the legacy one. The stream now returns only projects belonging to the organisation you configured, and several fields come back null: avatar, colour, the internal and public flags, organisation and status.
If you need those fields, there is a separate all projects detail stream that retrieves them. Be aware of what it costs: it makes one project-detail request for every project returned by the organisation list, so on an instance with a few hundred projects that is a few hundred requests against endpoint-specific rate limits.
If you are upgrading across that version, read the migration guide first. It flags a risk of data loss, which combined with a source that only retains 30 or 90 days is the kind of combination worth taking seriously rather than clicking through.
Frequently asked questions
Can I backfill older Sentry events?
Only as far back as Sentry retains them, which is 30 or 90 days depending on your plan. Your warehouse history starts at your first sync, so set the pipeline up early.
A stream is returning a 403. What is wrong?
Your auth token is missing a scope. Grant organisation read, project read and event read to sync every stream.
Why are fields in my projects table null?
Since version 1.0.0 the projects stream uses an organisation-scoped endpoint, and avatar, colour, the internal and public flags, organisation and status return null. Sync the all projects detail stream to retrieve them.
My syncs are getting 429 responses. What should I change?
Reduce the number of concurrent workers. Sentry's limits vary by endpoint and are returned in the response headers.
Can I do this without writing code?
Yes. The setup above is entirely UI-driven. You will want BigQuery views joining issues to your deployment records, which is where the useful analysis lives.
Get your Sentry data into BigQuery
Grant all three scopes at once, find out your plan's retention period because it bounds how long you can go between syncs, alert on failures rather than letting a broken connection sit, and set this up before somebody asks for a year of history you do not have.
Airbyte's connector catalog includes 700+ pre-built connectors, so error data can sit beside delivery and deployment sources. For search over the same records, see Sentry to Elasticsearch, and for build history from the other side of delivery, Jenkins to PostgreSQL.
Integrate with 700+ apps using Airbyte
Move data from 700+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.