Jira to Databricks: How to Move Your Data

Moving Jira into Databricks is how engineering and delivery metrics stop being screenshots. Cycle time, throughput, where work waits and whether last quarter's process change actually helped are all aggregate questions across a long window, and the Jira interface answers none of them well.

This guide covers the managed path with Airbyte. The Jira connector's catalogue is enormous and mostly describes how your instance is configured rather than what your teams are doing, so stream selection is the design work here rather than an afterthought.

Jira to Databricks at a glance:

Why move data from Jira to Databricks?

Two situations account for most of these pipelines.

The first is delivery analytics over time. Jira reports show the current board, and questions about how long work has taken across two years, or whether cycle time improved after a team reorganised, need history and aggregation. A lakehouse gives you both, plus notebooks for the analysis that does not fit a dashboard.

The second is joining. Issues against deployments, incidents against releases, support tickets against engineering work. Jira holds one side of each and the value appears only once it sits next to something Jira has never heard of.

What do you need before you start?

The connector is undemanding to connect and easy to over-configure:

An Atlassian API token. Used with the email address of the account that created it, as basic auth. The token inherits that person's permissions, so it sees exactly the projects and issues they can see. The Jira source documentation covers every field.

A service account to own it. A token created against an individual breaks when they change team, and its visible data changes silently when their permissions change. On a metrics pipeline that second failure is worse, because the numbers shift without anyone knowing why.

A decided list of projects. The projects field is optional, and leaving it empty replicates every project the token can see. On a large Atlassian tenancy that is hundreds of projects belonging to teams who have no idea a pipeline is reading them.

Unity Catalog access. Each stream becomes a table in Unity Catalog and the connector needs permission to create Volumes for Avro staging. Agree the catalog and schema before the first sync, particularly if you plan to split streams across two connections.

One pleasant note: rate limits are rarely a problem here. The connector is not expected to hit Jira's API limitations under normal usage, so your constraint is what you select rather than how fast you can pull it.

How do you build a Jira to Databricks pipeline in Airbyte?

Step 1: Separate activity streams from reference streams

Before opening Airbyte, split what you want into two lists. Activity is issues, issue comments, issue worklogs, board issues and sprint issues. Reference is projects, users, issue fields and workflow statuses, which resolve the identifiers in the activity records. Everything else in the catalogue describes screens, schemes and avatars, and almost none of it belongs in a lakehouse.

Step 2: Configure the Jira source

Click Sources in the left navigation, then New Source, and select Jira, following adding a source. Supply the API token, your domain in the form yourcompany.atlassian.net, the account email, your project list and a start date.

Step 3: Configure the Databricks destination

Click Destinations, then New Destination, and select Databricks, following adding a destination. Supply your workspace details, the catalog and schema, and authentication. Jira issue payloads nest heavily, and Airbyte preserves that structure, which suits a lakehouse where working with nested data in Spark is ordinary rather than awkward.

Step 4: Build two connections on different schedules

Click Connections, then New connection, and create one connection for the activity streams on a frequent schedule and a second for reference streams daily or weekly, each with a sync mode per stream. Each then has its own failure boundary, and a slow reference sync stops being able to delay the issue data your metrics depend on.

Run against a couple of projects first and check the resulting tables before extending to the whole list. Jira instances vary enormously in how they are configured, and what a field contains in one project is not always what it contains in another.

Why does the start date not limit my sync?

This is the configuration detail most likely to surprise you, and it is documented rather than hidden. The start date applies only to a subset of streams: board issues, issue comments, issue properties, issue remote links, issue votes, issue watchers, issue worklogs, issues, pull requests and sprint issues. Every other stream replicates all data regardless.

So a start date of last January bounds your issue activity and hands you the complete contents of every configuration stream you selected alongside it. On an instance carrying a decade of workflow schemes, that is a lot of data landing in tables nobody asked for.

Stream selection therefore has to do the work you might have expected the start date to do, which is exactly why step 1 happens before you open Airbyte. The same logic applies to the projects field: optional does not mean narrow, it means everything.

How should you model Jira data for metrics?

Custom fields are the first obstacle. Jira exposes them under generated identifiers rather than the display names your teams use, so without a mapping layer your analysts are reading fields named after numbers. Syncing the issue fields stream gives you exactly the lookup needed, and joining it once in a modelled table is the fix.

Then the derived measures. Cycle time, lead time and throughput are calculations rather than fields, and they depend on decisions about which status transitions count as started and done. Define them once in a Spark or SQL model so every dashboard uses the same definition, because two teams computing cycle time differently causes more argument than any pipeline fault.

Keep the raw landed tables underneath. Jira configurations change, teams rename statuses, and being able to reprocess history from what you already hold is worth more on this source than on most, because the meaning of a field can shift without the data looking any different.

Frequently asked questions

Why is my Jira sync pulling data older than the start date?

The start date applies to about ten activity streams only. All other streams replicate their full contents regardless of the value you set.

Which Jira streams should I select?

Issues, issue comments, issue worklogs, board issues and sprint issues for activity. Add projects, users, issue fields and workflow statuses as reference data so identifiers can be resolved.

Can I limit the sync to certain projects?

Yes. The projects field takes a list, and leaving it empty replicates every project the token's account can see.

Why are my custom fields named after numbers?

Jira exposes custom fields under generated identifiers rather than display names. Sync the issue fields stream and join it in a modelled table to resolve them.

Can I do this without writing code?

The Airbyte setup is entirely UI-driven. You will want Spark or SQL models resolving custom fields and defining your delivery metrics, which is where those definitions belong anyway.

Get your Jira data into Databricks

Name your projects, select a small set of activity streams with the reference streams that explain them, split them across two connections, and use a service account for the token. The start date will not narrow your first sync on its own, so the catalogue selection has to.

Airbyte's connector catalog includes 600+ pre-built connectors, so delivery data can sit beside build and incident sources in one lakehouse. For the same source feeding a streaming platform, see Jira to Kafka, and for build history from the other side of delivery, Jenkins to PostgreSQL.

Start syncing now →