Jenkins to BigQuery: How to Move Your Data

Moving Jenkins into BigQuery gives engineering leaders the build history Jenkins itself keeps throwing away. Jenkins shows recent runs for a job and prunes older ones to stay healthy, so questions about whether the pipeline has been getting slower over a year are usually unanswerable at exactly the moment somebody asks.

This guide covers the managed path with Airbyte. Two things shape the build: the connector's surface is narrower than people assume, and because retention is configured per job, your busiest pipeline is usually the one forgetting fastest.

Jenkins to BigQuery at a glance:

Why move data from Jenkins to BigQuery?

Two situations account for most of these pipelines.

The first is measuring delivery over time. Build duration, failure rate and how often the main branch is broken are all trend questions, and trends need history that Jenkins deliberately does not keep. A warehouse turns a disposable operational record into something you can chart across quarters.

The second is joining builds to everything around them, such as incidents, deployments or issue tracking, which is where the genuinely interesting correlations live. The pairing is wrong if you want to react to builds as they happen, since this is scheduled batch loading into an analytical store; Jenkins to Kafka is the route when something needs to respond promptly.

What do you need before you start?

Four things, and the second one sets your urgency:

Jenkins credentials and the instance URL. A username and API token for an account that can see the jobs you care about. The Jenkins source documentation covers the configuration, which is short.

The retention setting on your busiest job. Build retention is configured per job, and the pipeline running fifty times a day usually has the tightest limit precisely because it generates the most. That job is also the one whose history you most want, so find its setting first.

A BigQuery dataset in the right location. Location is fixed at creation and BigQuery will not join across locations, so put this where your incident and issue data already lives, since joining to them is the point.

Realistic expectations about scope. Two streams, builds and jobs, and no console output, test results or artefacts. That is enough for delivery metrics and not enough for failure analysis, which is worth saying before somebody scopes a project around it.

If your Jenkins instance restricts traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.

How do you build a Jenkins to BigQuery pipeline in Airbyte?

Step 1: Check retention on the jobs you care about most

Open the configuration for your two or three busiest pipelines and look at how many builds they keep. Somebody usually tightened this years ago when the controller was running out of disk, and the number is often smaller than anyone remembers. Whatever it is, that is how much history exists to capture, and a job keeping fifty builds while running fifty times a day is discarding yesterday every morning.

Step 2: Configure the Jenkins source

Click Sources in the left navigation, then New Source, and select Jenkins, following adding a source. Supply the instance URL and credentials, then select both streams. Jobs gives you the structure your builds refer to, and without it a builds table is a list of numbers against identifiers nobody recognises.

Step 3: Configure the BigQuery destination

Click Destinations, then New Destination, and select BigQuery, following adding a destination. Supply the project identifier, dataset and service account credentials. Build records are small and few by warehouse standards, so batched standard inserts are ample and the storage cost of keeping years of them is negligible.

Step 4: Create the connection and sync inside the retention window

Click Connections, then New connection, select your streams and a sync mode. Builds supports incremental, and your schedule needs to be comfortably shorter than the retention window on the busiest job. Rate limits are rarely a concern here, so there is no reason to be conservative about frequency.

Alert on sync failure, because a pipeline down for longer than a job's retention loses those builds permanently rather than catching up later.

What does this connector not give you?

Console logs, test reports and build artefacts. There are two streams here, builds and jobs, and they describe that a build ran, how long it took and how it ended. What they do not describe is why it ended that way, which is the thing everybody hopes a build pipeline dataset will tell them.

That shapes what you can honestly promise. Delivery metrics are well served: duration trends, failure rates, which jobs are flaky, how often the pipeline blocks a release. Failure analysis is not, because identifying which test broke or what the compiler complained about needs output this connector does not carry, and somebody expecting that will be disappointed after the work is done.

Retention is the other constraint and it has an awkward shape. Because it is set per job, the limits are usually tightest exactly where builds are most frequent, which is where your most useful signal lives. A quiet job keeping two hundred builds holds two years; a busy one keeping two hundred holds four days. Check both rather than assuming a single instance-wide behaviour.

How should BigQuery hold history Jenkins has already deleted?

As an accumulating record you protect rather than a mirror you can rebuild. Once a build has been pruned from Jenkins, your table is the only place it exists, which quietly changes the status of that dataset from convenience to archive. A full refresh overwrite would replace years of accumulated builds with whatever the retention windows currently allow.

The partitioning needs explaining to whoever queries it. Tables arrive partitioned daily on the extraction timestamp rather than on when the build ran, so a query filtering on build date looks sensible and prunes nothing, reading every partition you have collected. Teach people to filter on both, or expose a view that handles it, since BigQuery bills on bytes scanned and the habit is easy to form early and painful to retrofit.

Deduplicate as well, since a build seen across several syncs can appear more than once depending on how the connection is configured. A view selecting one row per build identifier gives analysts something dependable and keeps the raw table untouched underneath. Storage here is cheap enough that keeping everything indefinitely is entirely reasonable, which is not true of many pipelines and is the quiet advantage of this one.

Frequently asked questions

Can I get console output or test results?

No. The connector carries builds and jobs only, with no console logs, test reports or artefacts, so it supports delivery metrics rather than failure analysis.

Why is my busiest job missing the most history?

Retention is per job and usually tightest where builds are most frequent. A job keeping two hundred builds and running fifty times a day holds only a few days.

Why are my queries reading so much data?

Because tables partition on the extraction timestamp, not the build date. Filtering only on build date prunes nothing, so filter on both or use a view that does.

Do I need both streams?

Effectively yes. Jobs provides the context that makes builds legible, and without it you have durations attached to identifiers with no names.

Can I do this without writing code?

The pipeline, yes, and it is a short setup. A deduplicating view and sensible partition filtering are modest SQL worth writing once.

Get your Jenkins data into BigQuery

Check retention on your busiest jobs first, since that is where the window is tightest and the signal most valuable. Be clear that this covers builds and jobs and not logs, tests or artefacts, so nobody scopes failure analysis around it. Sync comfortably inside the retention window and alert on failure. Then treat the BigQuery tables as the archive they have become, filter on the partitioning column as well as the build date, and deduplicate in a view.

Airbyte's connector catalog includes 600+ pre-built connectors, so delivery history can outlive the system that produced it. For the same source into an operational database, see Jenkins to PostgreSQL, and for a comparable developer platform into the same destination, Github to BigQuery.

Start syncing now →