GitLab to BigQuery: How to Move Your Data

Moving GitLab into BigQuery is how engineering metrics stop being a screenshot pasted into a slide. Cycle time, review latency, deployment frequency and reliability trends all need history and joins, and neither is something the GitLab interface offers across a long window.

This guide covers the managed path with Airbyte. The thing to get right before anything else is scope, because the connector's default configuration is considerably broader than most people intend, and on a large GitLab instance that difference is enormous.

GitLab to BigQuery at a glance:

Why move data from GitLab to BigQuery?

Two situations account for most of these pipelines.

The first is engineering effectiveness measured over time rather than sampled. How long reviews take now compared with six months ago, whether a process change helped, which repositories carry the most unplanned work. Those are aggregate questions across a long window, and a warehouse answers them while a merge request list does not.

The second is joining engineering data to everything else: incidents against deployments, support tickets against releases, headcount against throughput. GitLab holds one side of each of those and the value appears only once it sits next to something GitLab does not know about.

What do you need before you start?

The connector is easy to configure and easy to point at far more of your GitLab estate than you meant to:

A personal access token with read_api. Generated under Settings and then Access Tokens, or use OAuth. Be aware what that scope covers: read access to the API including all groups and projects the account can reach, plus the container and package registries. The GitLab source documentation covers the fields.

A service account to hold it. Because the token inherits its owner's access, one created against an individual sees everything they can see and stops working when they leave. Scope a service account to the groups you want, and the breadth of read_api stops mattering.

A decided list of groups or projects. You supply group paths or project paths. If both fields are blank, the connector retrieves every group accessible to the token and syncs their projects, which on a large instance is an enormous first sync into tables nobody asked for.

A BigQuery service account and dataset. With the BigQuery User and BigQuery Data Editor roles. Set the dataset location to match the datasets holding your incident and deployment data, since it cannot be changed after creation and joining is the point.

If you run self-hosted GitLab, note the API URL setting, which defaults to gitlab.com, and remember your administrator may have configured different rate limits from GitLab.com's.

How do you build a GitLab to BigQuery pipeline in Airbyte?

Step 1: Decide your scope explicitly

Write down the specific group paths or project paths you intend to sync. This matters more here than on most connectors, because the empty-field default is not a narrow one, and on a warehouse destination an over-broad sync also means storage and query cost for data nobody will look at.

Step 2: Configure the GitLab source

Click Sources in the left navigation, then New Source, and select GitLab, following adding a source. Authenticate with OAuth or your personal access token, set the API URL if you are self-hosted, set a start date, and fill in your groups or projects. You supply one or the other rather than both.

Step 3: Configure the BigQuery destination

Click Destinations, then New Destination, and select BigQuery, following adding a destination. Supply the project, dataset and service account key. Batched standard inserts is the default loading method and is adequate for most instances, since engineering activity is modest by warehouse standards.

Step 4: Split activity from reference streams

Click Connections, then New connection, and select streams with a sync mode for each. Merge requests, issues, commits, pipelines and jobs carry the activity most metrics need. Projects, groups, users and members are reference data resolving the identifiers in those records, and belong on a slower schedule or a second connection.

If syncs are slow or you see persistent rate limit errors, there is a concurrent threads setting controlling how many parallel requests the connector makes. GitLab.com applies per-endpoint limits to the groups, members and projects APIs among others, and the connector retries requests receiving HTTP 429 automatically.

Why is my first sync so much larger than expected?

Almost always the empty scope fields. Both the groups and projects settings are optional, and leaving both blank does not mean nothing, it means everything: the connector lists every group the token can reach and syncs their projects.

That interacts badly with the read_api scope, which grants read access across all groups and projects the account can see. A token belonging to a well-connected engineer at a large company can reach a great deal, and none of those teams know a pipeline is reading them. It is not a security hole, since the token only sees what its owner already could, but it is a scope decision to make on purpose.

Name your groups or projects explicitly and use a service account scoped to them. On a warehouse destination this also controls what you pay to store and query, so the discipline pays twice.

How should you model GitLab data for metrics?

Start with the plan question, because it decides what exists. Epics and epic issues are gated to GitLab Ultimate and GitLab.com Gold accounts, so on other plans they are simply not exposed. If epic-level roadmap reporting is why you are building this, confirm the plan before scoping the project.

Where you do have them, the epics stream uses the internal identifier as its primary key rather than the global one, which makes matching records against the GitLab interface much easier. Use that field when joining epics to anything else rather than assuming the global ID.

Then define your derived measures once. Cycle time, review latency and throughput are calculations rather than fields, and they depend on decisions about which events count as started and done. Put them in a view so every dashboard reads the same definition, because two teams computing cycle time differently causes more argument than any pipeline fault.

Frequently asked questions

What happens if I leave the groups and projects fields blank?

The connector retrieves every group accessible to your token and syncs their projects. On a large instance that is a very large first sync, so name your groups or projects explicitly.

Why can I not see the epics stream?

Epics and epic issues are available only on GitLab Ultimate and GitLab.com Gold accounts. On other plans the data is not exposed through the API.

Does this work with self-hosted GitLab?

Yes. Set the API URL to your instance instead of the gitlab.com default, bearing in mind your administrator may have configured different rate limits.

What scope does the access token need?

read_api, which grants read access to the API including all groups and projects the account can reach, plus the container and package registries. Scope the account itself rather than the token.

Can I do this without writing code?

Yes. The setup above is entirely UI-driven. You will want BigQuery views defining your cycle time and throughput measures, which is where those definitions belong anyway.

Get your GitLab data into BigQuery

Name your groups or projects, use a service account scoped to them, split reference streams onto a slower schedule, check your plan before promising epic data, and define your metrics once in a view. The connector rarely hits GitLab's rate limits under normal usage, so scope is the thing to get right.

Airbyte's connector catalog includes 600+ pre-built connectors, so engineering data can sit beside incident and support sources. For the same source feeding a streaming platform, see GitLab to Kafka, and for build history from the other side of delivery, Jenkins to PostgreSQL.

Start syncing now →