GitLab to Kafka: How to Stream Your Data

Getting GitLab data onto Kafka usually starts with webhooks. They deliver events as they happen, which sounds ideal until you need history: a consumer that starts today can never see what happened before it existed, and a consumer that was down for an hour has simply lost that hour. The other common approach is a script against the GitLab API, which works until the person who wrote it changes team.

This guide covers the managed path: replicating GitLab into Kafka topics with Airbyte, so several teams can subscribe to the same feed with replay available. The main thing to get right is scope, because the default configuration is considerably broader than most people intend.

GitLab to Kafka at a glance:

Why move data from GitLab to Kafka?

Two situations account for most of these pipelines.

The first is engineering metrics with several consumers. A DORA dashboard, a deployment tracker, a code review analytics tool and an internal status page all want pipeline and merge request data. Giving each its own GitLab integration means four tokens with broad scopes and four things to fix when something changes. One pipeline into topics they subscribe to is fewer moving parts.

The second is replay, which is the concrete advantage over webhooks. Kafka retains what it receives, so a consumer with a bug can be fixed and re-run from an offset rather than losing the window it was broken for. Webhooks deliver once and assume you were listening.

What do you need before you start?

The connector is easy to configure and easy to point at far more of your GitLab estate than you meant to:

A personal access token with read_api. Generated under Settings and then Access Tokens, or use OAuth. Be aware what that scope covers: read access to the API including all groups and projects the account can reach, plus the container registry and the package registry. It is broad by design.

A service account to hold it. Because the token inherits its owner's access, a token created against an individual sees everything they can see and stops working when they leave. Scope a service account to the groups you actually want, and the breadth of read_api stops mattering.

A decided list of groups or projects. You can supply group paths or project paths. If both fields are blank, the connector retrieves every group accessible to the token and syncs their projects. On a large GitLab instance that is an enormous first sync into topics nobody sized for it. The GitLab source documentation covers both fields.

Topics created in advance. The destination fails on an unknown topic unless the broker creates topics automatically, which is discouraged in production where partition counts and retention should be chosen deliberately.

If you run self-hosted GitLab, note the API URL setting, which defaults to gitlab.com. And if your Kafka cluster restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.

How do you build a GitLab to Kafka pipeline in Airbyte?

Step 1: Create the token and decide your scope

Generate the token against a service account with read_api, then write down the specific group paths or project paths you intend to sync. Doing this before you open Airbyte matters more here than on most connectors, because the empty-field default is not a narrow one. Create your Kafka topics at the same time.

Step 2: Configure the GitLab source

Click Sources in the left navigation, then New Source, and select GitLab, following adding a source. Authenticate with OAuth or your personal access token, set the API URL if you are self-hosted, set a start date, and fill in your groups or projects. You supply one or the other rather than both.

Step 3: Configure the Kafka destination

Click Destinations, then New Destination, and select Kafka, following adding a destination. Supply the bootstrap servers, security protocol and credentials, and a topic pattern. The pattern accepts namespace and stream variables, so one topic per stream is straightforward and is usually what consumers want, since a deployment tracker has no interest in project labels.

Step 4: Select streams and schedule

Click Connections, then New connection, pick your source and destination, and select streams and a sync mode for each. Merge requests, issues, commits, pipelines and jobs carry the activity most consumers want. Projects, groups, users and members are reference data that resolves the identifiers in those records, and belong on a slower schedule or a second connection.

If syncs are slow or you see persistent rate limit errors, there is a concurrent threads setting that controls how many parallel requests the connector makes. GitLab.com applies per-endpoint limits to the groups, members and projects APIs among others, and the connector retries requests that receive HTTP 429 automatically. Self-hosted instances may have entirely different limits configured by your administrator.

Why is my first sync so much larger than expected?

Almost always the empty scope fields. Both the groups and projects settings are optional, and leaving both blank does not mean nothing, it means everything: the connector lists every group the token can reach and syncs their projects.

That interacts badly with the read_api scope, which grants read access across all groups and projects the account can see. A token belonging to a well-connected engineer at a large company can reach a great deal, and none of those teams know a pipeline is reading them. It is not a security hole, since the token only sees what its owner already could, but it is a scope decision that should be made on purpose.

Name your groups or projects explicitly, and use a service account whose access is scoped to them. Then the broad token scope stops mattering, because the account cannot reach anything you did not intend.

Which streams might not appear at all?

Epics and epic issues are gated by your GitLab plan, available only on GitLab Ultimate and GitLab.com Gold accounts. On lower tiers they are simply not exposed, so a connection configured to sync them will not return what you are looking for. If epics are the reason you are building this pipeline, confirm the plan before scoping the project.

One quirk worth knowing if you do have them. The epics stream uses the internal identifier as its primary key rather than the global one, which makes matching records against what people see in the GitLab interface much easier. If a consumer is joining epics to something else, use that field rather than assuming the global ID.

As for what lands in the topic: the Kafka destination writes JSON only, so a schema registry cannot validate it. Each message is keyed by a UUID Airbyte generates rather than the record identifier, so updates to the same merge request can be consumed out of order. For dashboards and metrics that is fine. For anything tracking pipeline state transitions in sequence, repartition on the record ID first.

Frequently asked questions

What happens if I leave the groups and projects fields blank?

The connector retrieves every group accessible to your token and syncs their projects. On a large instance that is a very large first sync, so name your groups or projects explicitly.

Why can I not see the epics stream?

Epics and epic issues are available only on GitLab Ultimate and GitLab.com Gold accounts. On other plans the data is not exposed through the API.

Does this work with self-hosted GitLab?

Yes. Set the API URL to your instance instead of the gitlab.com default. Bear in mind your administrator may have configured different rate limits from GitLab.com's.

What scope does the access token need?

read_api, which grants read access to the API including all groups and projects the account can reach, plus the container and package registries. Scope the account itself rather than the token.

Can I do this without writing code?

Yes. The setup above is entirely UI-driven, from creating the source through to scheduling the connection. If you would rather manage pipelines as infrastructure, the same source, destination and connection can be provisioned from configuration files instead.

Get your GitLab data into Kafka

Name your groups or projects, use a service account scoped to them, split reference streams onto a slower schedule, and check your plan before promising anyone epic data. The connector rarely hits GitLab's rate limits under normal usage, so scope is the thing to get right rather than throughput.

Airbyte's connector catalog includes 600+ pre-built connectors, so the same GitLab source can feed a warehouse alongside the topic. For the same destination fed by another workflow tool, see n8n to Kafka, and for the equivalent pipeline out of the other major code host, GitHub to MySQL.

Start syncing now →