Gitlab to ClickHouse: How to Move Your Data

Moving GitLab into ClickHouse suits one question better than any other: how has our continuous integration behaved across thousands of runs. Pipeline and job records accumulate quickly on an active organisation, and aggregating a year of them is exactly what a column store is for.

This guide covers the managed path with Airbyte. Two things shape the build: only some GitLab streams are large enough to justify this destination, and pipeline records describe things that were still happening when you read them.

Gitlab to ClickHouse at a glance:

Why move data from Gitlab to ClickHouse?

Two situations account for most of these pipelines.

The first is continuous integration analysis at scale. Failure rates by job, queue times by runner and how build duration has drifted over a year are heavy aggregations over a lot of rows, and interactive speed changes how often anybody investigates rather than assumes.

The second is powering a dashboard engineering teams actually watch. If your interest is delivery metrics joined to commercial data under governed access, Gitlab to Snowflake suits that shape better than a column store will.

What do you need before you start?

Four things, and the first decides whether this destination is the right one:

A clear idea of which streams you want. Pipelines, jobs and commits are the ones that grow; groups, projects and labels are small configuration. The GitLab source documentation lists what is available.

A token with the read_api scope, and an explicit scope list. The groups and projects fields are optional, and leaving both blank means every group your token can reach, which is rarely what anybody wanted.

Your API URL if you self-host. The connector works with gitlab.com and self-hosted instances alike, and the default assumes the former.

A ClickHouse database and a sorting key per table. Almost every question about continuous integration bounds a period, so the sorting key usually begins with a date.

If your instance or database restricts traffic by IP, add the Airbyte Cloud IP addresses to the relevant allow lists before you begin.

How do you build a Gitlab to ClickHouse pipeline in Airbyte?

Step 1: Separate the large streams from the small ones

Sort your chosen streams into two groups before configuring anything: the ones that accumulate with every build and the ones that describe how your organisation is arranged. The first group is why you are using this destination and the second is along for the ride. Knowing which is which shapes your sorting keys, your schedule and whether anybody should be surprised that the labels table is tiny.

Step 2: Configure the GitLab source

Click Sources in the left navigation, then New Source, and select GitLab, following adding a source. Authenticate, set the API URL if you self-host, supply a start date and name your groups or projects. Where a group holds more than a hundred projects, list projects explicitly rather than relying on group expansion.

Step 3: Configure the ClickHouse destination

Click Destinations, then New Destination, and select ClickHouse, following adding a destination. Supply the host, port, database and credentials. Records land in typed columns over the native protocol, and the types settled at table creation are the ones you keep, so look at them after the first sync.

Step 4: Create the connection and count your projects

Click Connections, then New connection, select your streams and a sync mode. Use incremental where offered. Then compare the number of projects that arrived against what GitLab reports, because retrieval through a group is capped and says nothing about it.

Then set the sorting keys, because a populated table sorted badly is a slow table with extra steps.

Which GitLab data justifies a column store?

Pipelines, jobs and commits, and really only those. An active organisation produces build records constantly, each carrying a status, a duration and a set of references, and after a year there are enough of them that aggregating in a row store becomes noticeable. That is the workload this destination exists for.

Everything else is small. Groups, projects, labels, milestones and members describe how your organisation is arranged rather than what it did, and there are hundreds rather than millions. Those tables are perfectly fine here and nothing about them needed a column store, so nobody should be evaluating this project on how quickly the labels table queries.

Keep the scope honest while you are deciding. Both scope fields are optional and an empty configuration takes every group the token can see, and separately, a reported limitation caps projects retrieved through a group at one hundred without raising an error. On a large organisation those two work against each other, so name your projects and check the count that arrived.

What about pipelines that had not finished?

They arrive twice, which is the quiet arithmetic problem in this pairing. A sync running at eleven catches a pipeline that is still going, records it as running with no duration, and a later sync catches the same pipeline as succeeded or failed. Both rows are true and only one is the answer.

In a column store that matters because deduplication happens during background merges rather than on write, so for a period both versions are present in the table. Counting failures directly will include pipelines that were merely in progress, and calculating average duration will average in records that had no duration to report.

So report through views applying FINAL, and consider filtering to terminal statuses for any measure about outcomes, since a pipeline still running is not a result. Sort these tables by the date the pipeline started, because that is what every query bounds, and keep status and reference columns typed for low cardinality since they take a small set of repeated values across millions of rows.

Frequently asked questions

Why is my failure rate too high?

Probably counting pipelines that were still running when a sync caught them, alongside their finished versions. Report through a view applying FINAL and filter to terminal statuses.

Why are some projects missing?

A reported limitation caps projects retrieved through a group at one hundred without erroring. List projects explicitly for large groups and check your connector version.

Is ClickHouse overkill for GitLab data?

For the catalogue streams, yes. For pipelines, jobs and commits on an active organisation it is exactly the right shape, which is why the split matters.

What should I sort these tables by?

A date first, since nearly every question bounds a period. Adding project early in the key helps if most queries look at one project at a time.

Can I do this without writing code?

The pipeline, yes. The views applying FINAL and the sorting keys are SQL, and they are what makes this fast and correct rather than merely populated.

Get your Gitlab data into ClickHouse

Split your streams into the ones that grow with every build and the ones that describe your organisation, because only the first group needs this destination. Name your groups or projects and check the project count that arrived, since group retrieval is capped without warning. Then sort by date, keep status columns typed for low cardinality, and report through views applying FINAL so pipelines that were merely running do not count as outcomes.

Airbyte's connector catalog includes 600+ pre-built connectors, so engineering activity can be analysed at whatever speed the questions demand. For the same source into a warehouse, see Gitlab to BigQuery, and for a relational source into the same destination, MySQL to ClickHouse.

Start syncing now →