Jenkins to Kafka: How to Move Your Data

Putting Jenkins build data onto Kafka gives several teams one feed to subscribe to rather than each writing their own integration against the controller. A deployment tracker, an engineering metrics service, a status page and a chat notifier can all read the same topic.

One expectation to set immediately: this is not how you trigger things when a build finishes. Airbyte syncs on a schedule rather than tailing Jenkins, so notifications belong in a Jenkins plugin or webhook. This pipeline is for durable, replayable history that several consumers share.

Jenkins to Kafka at a glance:

Why move data from Jenkins to Kafka?

Two situations account for most of these pipelines.

The first is fan-out. Several systems want build history, and giving each one credentials to the Jenkins controller means several integrations to maintain and several things reading a machine that exists to run builds. One pipeline into a topic they subscribe to is fewer moving parts.

The second is replay. Kafka retains what it receives, so a consumer with a bug can be fixed and re-run from an offset rather than losing the window it was broken for. If you have exactly one consumer and it is a database, skip the bus and write there directly.

What do you need before you start?

Short list, and the last item is the one that determines your schedule:

A Jenkins user and API token. Generated by logging into Jenkins and visiting the configure page for the account. Use a service account scoped to the jobs you need, since the token runs as that account and inherits what it can see. The Jenkins source documentation covers the fields.

A reachable Jenkins server. Jenkins is almost always self-hosted and frequently sits inside a private network, so a failed connection test is usually networking rather than credentials. Establish that path first.

Topics created in advance. The destination fails on an unknown topic unless the broker creates topics automatically, which is discouraged in production. With only two streams this is a one-time task rather than an ongoing process.

Your build retention settings. Find out how many builds your jobs keep before discarding them. That number sets how often this pipeline must run, and it is configured per job rather than globally.

If your Kafka cluster restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.

How do you build a Jenkins to Kafka pipeline in Airbyte?

Step 1: Work out both retention windows

Look at the discard old builds configuration on your busiest jobs, then look at the retention you intend to set on the Kafka topics. Those two numbers sit in series and between them decide what a consumer can ever see, which is the subject of the section below and the reason this step comes first.

Step 2: Configure the Jenkins source

Click Sources in the left navigation, then New Source, and select Jenkins, following adding a source. Supply the server URL, username and API token. Airbyte tests the connection immediately, and if it fails check network reachability before credentials.

Step 3: Configure the Kafka destination

Click Destinations, then New Destination, and select Kafka, following adding a destination. Supply the bootstrap servers, security protocol and credentials, and a topic pattern. One topic per stream is the natural arrangement, keeping the job catalogue separate from build activity so consumers take only what they need.

Step 4: Create the connection and sync often

Click Connections, then New connection, and select both streams with a sync mode each. Builds supports incremental, so use it. Set the frequency against your Jenkins discard policy rather than how fresh anyone wants the topic, which on a busy controller may mean several times a day.

That is comfortable to do, because the connector should not run into Jenkins API limitations under normal usage. Your constraint is what the controller still holds rather than how fast you may ask for it.

Why do two retention windows matter?

Because this pipeline has one at each end and they compound. Jenkins discards old builds so the controller does not fill its disk, and Kafka expires messages after its own retention period. A build has to survive the first window long enough to be read, and then survive the second long enough for a consumer to see it.

The Jenkins side is the sharper edge, because retention is configured per job and is usually tightest on exactly the jobs you most want to measure. A pipeline running on every commit generates the most builds, fills the most disk, and therefore gets the most aggressive discard policy. Your headline job is the one most likely to lose data between syncs.

So set the sync frequency against the shortest Jenkins retention among the jobs you care about, and set Kafka retention against how far behind a consumer might realistically fall. If anyone needs history beyond both, a consumer writing to a database or warehouse is what provides it, and the topic is the transport rather than the archive.

What contract should you give your consumers?

The destination writes JSON and only JSON, so consumers cannot validate against a schema registry. Each message value carries the record identifier, the extraction timestamp, the stream name and your build data as a nested structure, so consumers unwrap the envelope to reach the fields they want.

Messages are keyed by a UUID Airbyte generates rather than the job name or build number, so builds for the same job can land in different partitions and be consumed out of order. For a metrics consumer aggregating results that is harmless. For anything tracking the latest state of a job, repartition on the job identifier first.

Be explicit about timing too. Airbyte syncs on a schedule rather than streaming, so the topic receives batches at your sync interval and a build finishing now appears at the next run. Tell consuming teams that plainly, because the word Kafka leads people to assume something this pipeline does not offer.

Frequently asked questions

Can I use this to notify people when a build fails?

Not well. Syncs run on a schedule rather than continuously, so notifications belong in a Jenkins plugin or webhook. This pipeline is for durable history several consumers share.

Which streams does the Jenkins connector provide?

Builds, which supports incremental sync, and jobs. Console logs, test reports and artefacts are not included.

Why is my build history incomplete?

Either Jenkins discarded builds before a sync read them, or Kafka expired messages before a consumer did. Check both windows, and sync more often than your tightest discard policy.

Are builds for the same job ordered?

Not by default. Messages are keyed by an Airbyte UUID rather than the job identifier, so repartition on the job if a consumer tracks state in sequence.

Can I do this without writing code?

The Airbyte setup is entirely UI-driven. Consumers need to unwrap the JSON envelope, and any repartitioning is code in whatever stream processing you already run.

Get your Jenkins data into Kafka

Work out both retention windows before anything else, sync more often than Jenkins discards, create your topics deliberately, and write down the contract for consuming teams: batches rather than a stream, a wrapped payload, and no per-job ordering without a repartition.

Airbyte's connector catalog includes 600+ pre-built connectors, so build data can reach a database and a bus at the same time. For the durable archive version, see Jenkins to PostgreSQL, and for repository activity feeding the same destination, GitLab to Kafka.

Start syncing now →