Jira to Amazon S3 with AWS Glue: How to Move Your Data
Move Jira into S3 with AWS Glue using Airbyte. Why full-replication streams write equality deletes every sync, and how Glue rewrites your table names.
Moving Jira into Amazon S3 with AWS Glue keeps delivery history in an open format several engines can read. Years of issues, worklogs and comments are worth retaining and rarely queried daily, which is an expensive combination to keep in a warehouse and a natural one for Iceberg tables on object storage.
This guide covers the managed path with Airbyte. Two things shape the build: Jira's generated field names meet Glue's naming rules in ways worth anticipating, and the streams that replicate in full every sync are considerably more expensive here than in a warehouse.
Jira to Amazon S3 with AWS Glue at a glance:
Why move data from Jira to Amazon S3 with AWS Glue?
Two situations account for most of these pipelines.
The first is retention at a sensible price. Delivery history is worth keeping indefinitely and is consulted occasionally, so storage costs matter more than query latency. Iceberg on S3 holds it cheaply while remaining readable by Athena, Spark and Redshift Spectrum without another copy.
The second is feeding transformation work with more than one engine. The counterargument is operational, since a lake means catalogs and table maintenance to look after. If what you want is interactive reporting with none of that, Jira to BigQuery asks considerably less of you and answers the same questions.
What do you need before you start?
Four things, and the second one costs you on every sync if you get it wrong:
An API token, the account email and your domain. The token carries its owner's visibility, so a service account outlasts an individual's. The Jira source documentation covers generating one.
A deliberately short list of streams. Jira's catalogue is large and most of it describes configuration rather than events. Every configuration stream you keep replicates in full on every sync, which in Iceberg is more expensive than it sounds.
An S3 bucket and a Glue catalog. The pipeline writes data files to one and registers tables in the other, so credentials need both. Choose namespace and table names that Glue will leave alone.
A plan for table maintenance. Compaction and snapshot expiry are jobs you schedule. Budget for them alongside the pipeline rather than after query times start drifting.
If your network restricts traffic by IP, add the Airbyte Cloud IP addresses to the relevant allow list before you begin.
How do you build a Jira to Amazon S3 with AWS Glue pipeline in Airbyte?
Step 1: Select streams as though each one has a running cost
Go through Jira's catalogue and keep only what your analysis genuinely uses. Issues, worklogs, comments and sprint membership usually earn their place. Screens, permission schemes, field configurations and the long tail of configuration streams usually do not, and each one you keep replicates completely on every sync forever. In a warehouse that is storage; here it is storage plus a maintenance burden explained below.
Step 2: Configure the Jira source
Click Sources in the left navigation, then New Source, and select Jira, following adding a source. Supply the domain, email and token, then name your projects rather than leaving the field empty, which would pull every project the token can see. Include the issue fields stream, since it is what translates generated custom field identifiers into names.
Step 3: Configure the S3 Data Lake destination
Click Destinations, then New Destination, and select the S3 Data Lake, following adding a destination. Choose AWS Glue as the catalog and supply the bucket, region and credentials. Keep namespace and table names to letters, digits and underscores, because Glue converts anything else and a renamed table is one nobody expects.
Step 4: Create the connection and schedule conservatively
Click Connections, then New connection, select your streams and a sync mode. Daily is right for delivery analysis, and more frequent syncs multiply the maintenance cost rather than the insight. Jira's rate limits are rarely the constraint here; your own table maintenance is.
Schedule compaction and snapshot expiry from the first week rather than the first complaint.
How do Jira's names survive the journey into Glue?
Mostly intact, with two things worth knowing. Glue replaces non-alphanumeric characters in table and namespace names with underscores, so anything you name with a hyphen or a dot arrives looking different from what you typed. Choose names that survive the conversion and the catalog matches your expectations rather than surprising whoever browses it.
The bigger naming problem comes from Jira itself. Custom fields arrive under generated identifiers rather than the labels everybody uses, so story points, team and severity land in columns nobody recognises, and two Jira instances number the same concept differently. The issue fields stream holds the mapping, which is why it belongs in your selection whether or not anyone requested it.
Resolve that once rather than in every query. Because a lake is read by several engines, a mapping left unresolved gets reimplemented in Athena by one team and in Spark by another, and they will disagree about which identifier means what. A single modelled table exposing the handful of custom fields your reporting uses under readable names saves that argument permanently.
Why do configuration streams cost so much in Iceberg?
Because Iceberg has no update in place and those streams rewrite themselves constantly. An upsert here is an equality delete followed by an insert, so a stream that replicates in full on every sync produces a wave of delete files and a complete set of new rows every single run, whether or not a single value changed.
That is the interaction worth understanding, because Jira's start date bounds only about ten issue-centric streams and everything else in that large catalogue replicates fully every time. In a warehouse the consequence is modest storage. Here it is delete files accumulating on tables describing screens and permission schemes that nobody has queried since the pipeline was built, quietly slowing every read against them.
The remedies are both simple and both need deciding. Keep few configuration streams, since the cheapest table to maintain is the one you never created. And give the ones you do keep a gentler schedule than your issue streams, because a permission scheme changes a few times a year and syncing it hourly generates maintenance work for no benefit whatsoever. Separate connections make that easy and are worth the extra configuration.
Frequently asked questions
Why is my sync still large after setting a start date?
The date bounds only around ten issue-centric streams. Everything else replicates in full every run, so stream selection is the control that actually works.
Why are reads getting slower on tables nobody uses?
Full replication writes equality deletes on every sync, and those accumulate until compaction runs. Drop the streams you do not need and schedule maintenance for the ones you keep.
Why did my table name change in Glue?
Glue converts non-alphanumeric characters in table and namespace names into underscores. Pick names that survive that so the catalog reads the way you intended.
Why are custom fields named as identifiers?
That is how Jira exposes them. Sync the issue fields stream for the mapping and resolve it in one modelled table, so several query engines do not each invent their own version.
Can I do this without writing code?
The pipeline, yes. Compaction jobs and the table resolving custom field names are not, and a lake without maintenance degrades quietly rather than failing.
Get your Jira data into Amazon S3 with AWS Glue
Select streams as though each carries a running cost, because in Iceberg the ones replicating fully every sync genuinely do. Name your projects rather than taking everything, keep names alphanumeric so Glue leaves them alone, and resolve custom field identifiers once in a modelled table rather than letting each query engine do it differently. Then schedule compaction, and give slow-moving configuration streams a slower schedule than your issues.
Airbyte's connector catalog includes 600+ pre-built connectors, so delivery history can live in open formats rather than expensive storage. For the same source into a lakehouse, see Jira to Databricks, and for a relational source into the same destination, PostgreSQL to Amazon S3 with AWS Glue.
Integrate with 700+ apps using Airbyte
Move data from 700+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.