Jira to BigQuery: How to Move Your Data

Moving Jira into BigQuery gives you delivery history you can query alongside everything else the business measures. Jira answers questions about the board in front of you, not about how throughput has drifted over two years across a dozen teams, and it has no way to join an issue to the customer who raised it or the release that resolved it.

This guide covers the managed path with Airbyte. Two things shape the build: the dataset location is fixed at creation and cannot be changed afterwards, and a large part of Jira's catalogue replicates in full on every sync no matter what start date you set.

Jira to BigQuery at a glance:

Why move data from Jira to BigQuery?

Two situations account for most of these pipelines.

The first is engineering measurement that spans teams and years. Jira reports per board and per project, which suits the people working the board and fails the person asking whether anything has actually improved since last year. That question needs issues, worklogs, comments and sprint membership in one place, with enough history to see a trend rather than a snapshot.

The second is joining delivery data to commercial data that already lives in BigQuery. Support volumes, product usage, billing and marketing tend to land there first, and a bug becomes a different conversation once you can see which accounts hit it and what it cost. The pairing is a poor fit if you want issue changes to arrive somewhere within seconds, because this is a scheduled batch pipeline rather than a stream; Jira to Kafka is the honest route for that.

What do you need before you start?

Four things, and the third one cannot be undone once you have chosen it:

An API token, the account email and your domain. The token carries the visibility of the Atlassian account that generated it, so a service account beats an individual's token that stops working the day they leave. The Jira source documentation covers generating one.

A decision about which projects to include. The projects field is a filter and an empty one means everything the token can reach, which on an old instance sweeps in abandoned experiments and other departments' work. Naming projects keeps both the sync and the eventual query bill smaller.

A dataset in the right location, and a service account. Dataset location is set at creation and is immutable afterwards. BigQuery will not join across locations, so if your support and billing data sits in one region, this dataset needs to sit there too. The service account needs permission to create and write tables.

An inventory of the custom fields that matter. Story points, team, severity and anything else your organisation added arrive under generated identifiers rather than their display names, and no two instances number them alike. Knowing the handful you report on saves considerable guesswork.

If your network restricts outbound or inbound traffic by IP, add the Airbyte Cloud IP addresses to the relevant allow list before you begin.

How do you build a Jira to BigQuery pipeline in Airbyte?

Step 1: Fix the dataset location before you create anything

This is the one decision here with no undo. A dataset's location is set when it is created and cannot be altered, and BigQuery will not run a query that joins datasets in different locations. Find out where the data you intend to join already lives, create the Jira dataset there, and you will avoid the unpleasant discovery six weeks later that the only fix is to recreate the dataset and resync everything.

Step 2: Configure the Jira source

Click Sources in the left navigation, then New Source, and select Jira, following adding a source. Supply the domain, email and API token, then your project list and start date. Include the issue fields stream regardless of whether anyone asked for it, since it is the only thing that maps generated field identifiers back to readable names.

Step 3: Configure the BigQuery destination

Click Destinations, then New Destination, and select BigQuery, following adding a destination. Supply the project identifier, dataset and service account credentials. Batched standard inserts are the default and are fine for most Jira instances; Cloud Storage staging is there for genuinely large loads, which a Jira catalogue rarely is unless you have kept a decade of comments.

Step 4: Create the connection and choose streams, not just a date

Click Connections, then New connection, select your streams and a sync mode. Stream selection is the decision that matters, because the start date bounds only a small part of the catalogue and everything else arrives in full each time. Every stream you keep is a table that grows a new partition on every sync, whether or not anybody queries it.

Daily is right for most reporting. Jira's rate limits rarely trouble this connector, so schedule on freshness rather than out of caution, and remember that a more frequent schedule means more partitions rather than more data.

What does the start date actually bound?

Around ten streams, all of them issue-centric. Issues, board issues, sprint issues, comments, worklogs, votes, watchers, properties, remote links and pull requests carry timestamps the connector can filter on. That is the part of Jira that behaves like an event log, and it is usually the part people came for.

The rest of the catalogue describes configuration: workflows, screens, permission schemes, field configurations, issue types, resolutions and a long tail beyond those. None of it has a date worth filtering on, so all of it replicates completely on every run. That is reasonable behaviour and it still surprises people who set a start date of last January and expect a correspondingly small sync.

So treat stream selection as your volume control and the start date as a history setting. Most delivery reporting needs a handful of configuration streams rather than the full catalogue, and in BigQuery the difference is not just storage: each of those tables carries a fresh partition every sync, and an unfiltered query against one reads all of them.

Why does partitioning decide what this costs to query?

Because BigQuery bills on bytes scanned, and the tables arrive partitioned daily on the extraction timestamp rather than on anything from Jira. Clustering is applied on that timestamp together with your primary keys. Nothing here is partitioned by issue creation date or resolution date, which is what an analyst would reach for first.

That mismatch is where the money goes. A query filtering on when an issue was created looks sensible and prunes nothing, so it reads every partition the table has accumulated since the pipeline started. The habit worth teaching is to filter on the partitioning column as well, narrowing to the syncs you care about before applying any Jira-side date logic.

The configuration streams make this sharper. Because they replicate fully every run, a table of workflows or screens holds a complete copy per sync, and after a year of daily syncs an innocent count of issue types reads several hundred near-identical snapshots unless it is bounded to the latest partition. Build a view exposing only the most recent extraction for each configuration stream and point people at that instead of the raw table.

Frequently asked questions

Can I change the dataset location later?

No. Location is fixed when the dataset is created, and since BigQuery will not join across locations, the only remedy is creating a dataset in the right place and syncing again. Worth getting right on day one.

Why is my sync still large after setting a start date?

The date bounds only about ten issue-centric streams. Configuration streams have nothing to filter on and replicate in full every run, so trimming stream selection is what reduces the sync.

Why are my custom fields named as identifiers?

That is how Jira exposes them. Sync the issue fields stream for the mapping between identifiers and display names, and join to it in a view rather than scattering raw identifiers through your queries.

How do I keep query costs down?

Filter on the partitioning column, which is the extraction timestamp rather than any Jira date. For configuration streams that replicate fully, restrict to the latest extraction, since otherwise every query reads one copy per sync since the pipeline began.

Can I do this without writing code?

The pipeline, yes, since the setup above is entirely UI-driven. You will want a couple of views in BigQuery, one resolving custom field names and one exposing the latest extraction of each configuration stream, and both are modest SQL written once.

Get your Jira data into BigQuery

Create the dataset where the data you will join already lives, because that choice is permanent. Name your projects, control volume through stream selection rather than the start date, and teach people to filter on the extraction timestamp. Add a view over the latest extraction of each configuration stream and another resolving custom field names, and most of the cost and confusion in this pipeline disappears.

Airbyte's connector catalog includes 600+ pre-built connectors, so delivery data can sit beside the commercial data that explains it. For the same source in an operational database, see Jira to PostgreSQL, and for a comparable engineering source into the same destination, Github to BigQuery.

Start syncing now →