Jenkins to MongoDB: How to Move Your Data
Move Jenkins build history into MongoDB with Airbyte. When a document store is the right destination, build retention limits, and indexing for your application.
Moving Jenkins into MongoDB puts build history where an application can read it. An internal developer portal, a service catalogue or a deployment tracker that needs to show recent builds is simpler to build against a database it already connects to than against the Jenkins API.
One honest word on the pairing. MongoDB is a document store built for application reads, not for aggregation, so if your goal is engineering metrics rather than serving a tool, a relational database or a warehouse will serve you better. This guide assumes you want the former.
Jenkins to MongoDB at a glance:
Why move data from Jenkins to MongoDB?
One situation fits this pairing well, and one does not, and the difference is worth establishing before you build.
It fits serving an application. A developer portal showing the last ten builds for a service performs a key-based lookup, which is what MongoDB is built for, and reading from your own database rather than the Jenkins controller means one fewer thing that can be slow or unavailable when somebody loads the page.
It does not fit engineering metrics. Cycle time, failure rates and trends across teams are aggregate questions, and a document store is a poor place to ask them. If that is your goal, the Jenkins to PostgreSQL guide covers a better-suited destination and the reasoning behind it.
What do you need before you start?
Short list, and the last item is the one that determines your schedule:
A Jenkins user and API token. Generated by logging into Jenkins and visiting the configure page for the account. Use a service account scoped to the jobs you need, since the token runs as that account and inherits what it can see. The Jenkins source documentation covers the fields.
A reachable Jenkins server. Jenkins is almost always self-hosted and frequently sits inside a private network, so a failed connection test is usually networking rather than credentials. Establish that path first.
A MongoDB cluster and a write user. On the target database, ideally a collection namespace dedicated to synced data so the pipeline cannot collide with collections your application writes to.
Your build retention settings. Find out how many builds your jobs keep before discarding them. That number sets how often this pipeline must run, and it is configured per job rather than globally.
If your MongoDB cluster restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.
How do you build a Jenkins to MongoDB pipeline in Airbyte?
Step 1: Check the retention policy on your busiest jobs
Look at the discard old builds configuration on the jobs your application will display. That number is the outer bound on how long you can go between syncs, and it is usually tightest on exactly the jobs you care about most, for reasons covered below.
Step 2: Configure the Jenkins source
Click Sources in the left navigation, then New Source, and select Jenkins, following adding a source. Supply the server URL, username and API token. Airbyte tests the connection immediately, and if it fails check network reachability before credentials.
Step 3: Configure the MongoDB destination
Click Destinations, then New Destination, and select MongoDB, following adding a destination. Supply the connection details and credentials. Airbyte adds its own metadata fields to each document, so decide whether your application tolerates them or reads through a projection that excludes them.
Step 4: Create the connection and index for your reads
Click Connections, then New connection, and select both streams with a sync mode each. Builds supports incremental, so use it. Then create indexes matching how your application queries, typically by job identifier and build timestamp, because MongoDB will not do that for you and a portal scanning a collection is a slow portal.
Set the frequency against your Jenkins discard policy rather than how fresh anyone wants the portal, which on a busy controller may mean several times a day.
Why does build retention set your schedule?
Because Jenkins discards old builds so the controller does not fill its disk, and once a build is discarded it is gone from the source. A sync that runs less often than the discard policy will miss builds permanently, and nothing about the pipeline will tell you.
The awkward part is which jobs this affects. Retention is configured per job and is usually tightest on the pipeline running on every commit, because that is the one generating the most builds and consuming the most disk. So your busiest and most interesting job is the one most likely to lose data between syncs.
Set the frequency against the shortest retention among the jobs you care about, and add a freshness check on the destination so a stalled sync surfaces within a day rather than when somebody notices the portal is showing week-old builds.
What does the application actually get?
Build records and job definitions, and nothing else. Console logs, test reports and artefacts are outside this connector's scope, so a portal that wants to show why a build failed cannot get that here and will need to link out to Jenkins for the detail.
That is worth designing around rather than working against. Store the build record with its status and timestamps, and link to the Jenkins URL for anyone who needs the log. The pipeline gives your application the fast summary and Jenkins remains the place for the detail, which is a reasonable division of labour.
On the jobs side, take that stream even though it changes rarely, because build records reference jobs by identifier and without it your portal shows build numbers against nothing a human recognises.
Frequently asked questions
Is MongoDB a good destination for Jenkins data?
For serving an application by key-based lookup, yes. For engineering metrics and aggregation, a relational database or warehouse is better suited.
Which streams does the Jenkins connector provide?
Builds, which supports incremental sync, and jobs. Console logs, test reports and artefacts are not included.
Why is my build history incomplete?
Jenkins discarded builds before a sync read them. Retention is set per job and is usually tightest on your busiest pipeline, so sync more often than your shortest discard policy.
Do I need to create indexes myself?
Yes. Create indexes matching your application's query patterns, typically job identifier and build timestamp, because the pipeline will not create them for you.
Can I do this without writing code?
The Airbyte setup is entirely UI-driven. Creating indexes and writing the queries your application uses is work on the MongoDB side.
Get your Jenkins data into MongoDB
Confirm this is an application pipeline rather than a metrics one, because a warehouse suits the second better. Then check retention on your busiest jobs, sync more often than they discard, take the jobs stream so identifiers resolve, and create the indexes your portal needs.
Airbyte's connector catalog includes 600+ pre-built connectors, so build data can reach an application and an analytical store at once. For the metrics version, see Jenkins to PostgreSQL, and for fan-out to several consumers, Jenkins to Kafka.
Integrate with 700+ apps using Airbyte
Move data from 700+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.