Jenkins to PostgreSQL: How to Move Your Data

Moving Jenkins into PostgreSQL gives you a build history that outlives Jenkins itself. Most Jenkins jobs are configured to discard old builds so the controller does not fill its disk, which is sensible operationally and means the record you would want for any trend analysis is being deleted on a rolling basis.

This guide covers the managed path with Airbyte. The connector is deliberately narrow, exposing builds and jobs and nothing else, and knowing that up front stops you scoping a project around console logs or test results that this pipeline does not provide.

Jenkins to PostgreSQL at a glance:

Why move data from Jenkins to PostgreSQL?

Two situations account for most of these pipelines, and Postgres suits both better than a warehouse would.

The first is delivery metrics over a long window. Build success rate, duration trends, which jobs fail most, whether last quarter's pipeline work actually helped. Those are aggregate questions across months of builds, and Jenkins is not built to answer them, particularly once old builds have been discarded.

The second is feeding an internal tool. An engineering dashboard or a status page that already reads Postgres can query this directly. Build records are small, so a database you already run is cheaper and simpler than provisioning warehouse capacity for a few hundred thousand rows.

What do you need before you start?

Short list, and the last item is the one that determines whether the pipeline is worth building at all:

A Jenkins user and API token. Generate the token by logging into Jenkins and visiting the configure page for your account. Use a service account scoped to the jobs you need rather than an individual's login, because the token runs as that account and inherits what it can see. The Jenkins source documentation covers the fields.

A reachable Jenkins server. Jenkins is almost always self-hosted and frequently sits inside a private network, so a failed connection test here is usually networking rather than credentials. Establish that path before configuring anything else.

Realistic expectations about the streams. Builds and jobs. That gives you the build record and the job catalogue, which is enough for delivery metrics and not enough for failure analysis, because console output, test reports and artefacts are not part of it.

Your build retention settings. Find out how many builds your jobs keep before discarding. That number tells you how often this pipeline needs to run, and it is the single most useful thing to know before configuring anything.

On the destination side you need a Postgres user with write permission on the target schema. If you already run a Postgres destination on an older connector version, read the migration guide before upgrading.

How do you build a Jenkins to PostgreSQL pipeline in Airbyte?

Step 1: Check how long your builds survive

Look at the discard old builds configuration on your busiest jobs. A job running fifty times a day and keeping the last hundred builds retains two days of history. Sync weekly against that and you are permanently losing most of it. This is the number that sets your schedule, and it varies enormously between jobs on the same controller.

Step 2: Configure the Jenkins source

Click Sources in the left navigation, then New Source, and select Jenkins, following adding a source. Supply the server URL, the username and the API token. Airbyte tests the connection immediately, and if it fails, check network reachability before credentials.

Step 3: Configure the PostgreSQL destination

Click Destinations, then New Destination, and select Postgres, following adding a destination. From version 3.0.0 the Postgres destination uses Direct Load architecture, writing records straight to final tables with no intermediate raw tables, and adds its own metadata columns alongside your data.

Step 4: Sync builds incrementally and jobs alongside

Click Connections, then New connection, and select both streams with a sync mode each. Builds supports incremental, so use it and let the history accumulate. Jobs is small and describes your controller's configuration rather than its activity, so a full refresh of it on the same schedule costs nothing.

Set the frequency against the retention figure from step 1, not against how fresh anyone wants the dashboard. On a busy controller that may mean several times a day, which is fine because the connector should not run into Jenkins API limitations under normal usage.

Why does build retention decide your schedule?

Because Airbyte can only read what Jenkins still holds. Once a job discards a build, that record is gone from the controller and no sync will ever recover it. Your Postgres history therefore starts on the day of your first sync and only contains what survived long enough to be read.

The trap is that retention is configured per job rather than globally, and it is usually tightest on exactly the jobs you most want to measure. A pipeline that runs on every commit generates the most builds, fills the most disk, and therefore gets the most aggressive discard policy. Your headline job is the one most likely to lose data between syncs.

So set your schedule against the shortest retention among the jobs you care about, and check it again if someone tightens a discard policy to reclaim disk. A freshness check on the builds table is worth adding too, because a stalled connection here does not just mean stale data, it means data that is being permanently deleted at the source while nobody is looking.

What can this pipeline not tell you?

It cannot tell you why a build failed. Console output, test reports and build artefacts are not in the two streams, so failure analysis stays in Jenkins or in whatever log aggregation you already run. What you get is that a build failed, which job it belonged to, when it ran and how long it took.

That is genuinely enough for delivery metrics, which are mostly counts, rates and durations grouped by job and by week. It is not enough for the question about which test is flakiest, and scoping a project around that expectation is how these pipelines disappoint people.

One modelling note. Because you have both streams, join builds to jobs so your dashboards show job names rather than identifiers, and derive duration and success rate in views so every consumer reads the same definition. Two teams computing build success rate slightly differently causes more disagreement than any pipeline fault.

Frequently asked questions

Which streams does the Jenkins connector provide?

Builds, which supports incremental sync, and jobs. Console logs, test reports and artefacts are not included.

Why is my build history missing older records?

Jenkins discarded them before a sync could read them. Airbyte can only capture builds the controller still holds, so sync more often than your jobs discard old builds.

Will this hit Jenkins API rate limits?

It should not under normal usage, which means you can schedule frequently enough to stay ahead of build retention without worrying about throttling.

My connection test fails. What should I check first?

Network reachability. Jenkins is usually self-hosted and often sits inside a private network, so the path from Airbyte to the controller is the more common problem than the token.

Can I do this without writing code?

Yes. The setup above is entirely UI-driven. You will want SQL views joining builds to jobs and defining your duration and success rate measures, which is where the metric definitions belong.

Get your Jenkins data into PostgreSQL

Check your discard policies first and set the schedule against the tightest one, sync both streams so your dashboards can show job names, and put the metric definitions in views. Then add a freshness check, because a stalled sync here means builds being deleted at the source while nobody notices.

Airbyte's connector catalog includes 700+ pre-built connectors, so build data can sit alongside your code hosting and incident sources. For repository activity from the other side of delivery, see GitHub to MySQL, and for another operational source where retention drives the design, n8n to PostgreSQL.

Start syncing now →