Tempo to Kafka: How to Move Your Data

Moving Tempo into Kafka publishes logged hours where several systems can consume them at once. A billing process, a resourcing tool and a warehouse loader all want the same worklogs, and pointing each at Tempo's API means three integrations, three tokens and three people maintaining them.

This guide covers the managed path with Airbyte. Two things shape the build: the connector carries hours without carrying what the hours were spent on, and that gap is multiplied rather than solved by putting the data on a bus.

Tempo to Kafka at a glance:

Why move data from Tempo to Kafka?

Two situations account for most of these pipelines.

The first is that logged hours feed several processes at once. In a consultancy, the same worklogs drive invoicing, utilisation reporting and resourcing decisions, and those are usually three different systems owned by three different teams. Publishing once removes the situation where each maintains its own integration and they quietly disagree about the numbers.

The second is feeding a service rather than a report, such as something that flags unbilled time. The usual test still applies: with one system at the other end, a bus adds a moving part and earns nothing, and a direct pipeline into a warehouse is the simpler answer. Kafka starts paying at three or four consumers.

What do you need before you start?

Four things, and the third is the one that determines how useful the topic is:

An API token, generated for a specific person. Found under Tempo, then Settings, then Data Access, then API integration. The connector reads what its owner is permitted to read, and on a bus every consumer inherits that boundary without being able to see it. The Tempo source documentation covers generating one.

Topics, created before the first sync. The destination writes to topics that already exist. Worklogs and the three configuration streams change at completely different rates, so give them separate topics rather than one.

A decision about enrichment. Worklogs reference issues by key and nothing here explains what those issues are. Publishing them raw means every consumer solves that separately, which is the central design question of this pipeline rather than a detail.

An honest count of your consumers. Worth doing before the work rather than after, because a bus with a single reader is a warehouse pipeline carrying extra operational weight.

If your organisation restricts access by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.

How do you build a Tempo to Kafka pipeline in Airbyte?

Step 1: Decide whether you are publishing raw or enriched worklogs

Raw means the topic carries exactly what Tempo sends, and every consumer joins to Jira itself. Enriched means you add an intermediate step that combines worklogs with issue data and publishes the result to a second topic, which consumers read instead. The second is more work once and less work repeatedly, and choosing before anybody builds a consumer is considerably easier than changing the contract afterwards.

Step 2: Configure the Tempo source

Click Sources in the left navigation, then New Source, and select Tempo, following adding a source. Supply the API token, which is the whole configuration. Four streams are available and worklogs is the one your consumers came for, though accounts and customers matter if anybody is attributing time to clients.

Step 3: Configure the Kafka destination

Click Destinations, then New Destination, and select Kafka, following adding a destination. Supply the bootstrap servers, security protocol and topic configuration. Messages are JSON with no Avro option, and each wraps the record alongside its identifier, the extraction timestamp and the stream name, so consumers read through an envelope rather than finding Tempo's own shape at the top level.

Step 4: Create the connection and allow for retrospective logging

Click Connections, then New connection, select your streams and a sync mode. Daily suits time tracking, and consumers need to understand that people log hours days after the fact. A worklog for last Tuesday can arrive on Friday, so anything treating a period as closed at midnight will be wrong on Monday.

Record what the token's owner can see, somewhere consumers can read it, because nothing on the topic reveals that boundary.

What is actually on the topic?

Hours, and very little else. The connector offers four streams: accounts, customers, worklogs and workload schemes. A worklog tells you who logged how long against which issue key, and the issue key is where the description stops. What that issue was, which project it belonged to, whether it was billable work or a bug fix, none of it is here.

That is reasonable, since Tempo is an add-on to Jira and Jira holds the issues. It does mean a topic of raw worklogs is incomplete by design rather than by accident, and a consuming team told to read from it will discover that within an hour of starting.

The permission boundary compounds it. The token belongs to one person and returns their view of Tempo, so a token issued by a team lead publishes that team's worklogs and nothing else. On a bus this is invisible: consumers receive what arrives and have no way to know what did not. Use a service account with documented access, and write down what it covers next to the topic name.

Why does every consumer end up doing the same join?

Because incomplete data on a bus multiplies the work of completing it. One consumer joining worklogs to Jira issues is a reasonable piece of engineering. Four consumers each doing it separately is the same logic written four times, in four languages, by four teams who will disagree about edge cases such as which project an issue moved between.

This is the opposite of what a bus is supposed to achieve. The point of publishing once is that consumers read rather than integrate, and a topic requiring every reader to fetch context elsewhere has moved the integration rather than removed it. It also puts four times the load on Jira, from services that were meant to be reading from Kafka.

So publish the join. Run a Jira pipeline alongside this one, combine worklogs with issue details in a stream processor or a scheduled job, and publish the enriched result to its own topic that consumers actually read. Keep the raw topic for anything that genuinely wants it. Note too that compaction will not tidy either topic, since messages are keyed by a generated identifier rather than by worklog, so set retention deliberately rather than expecting old versions to collapse.

Frequently asked questions

Does this include the Jira issues themselves?

No. Four streams cover accounts, customers, worklogs and workload schemes, and worklogs reference issues by key only. Run the Jira connector alongside this one.

Should consumers read the raw topic?

Usually not. Publish an enriched topic combining worklogs with issue context, so the join happens once rather than in every consumer and Jira is not queried four times over.

Will compaction remove older versions of a worklog?

No, because messages are keyed by a generated identifier rather than the worklog. Compaction has nothing to match, so size retention on the assumption that versions accumulate.

Why are some people's hours missing?

The token is issued for an individual and returns what they can see. For organisation-wide publishing, generate it from an account with matching visibility and document what that covers.

Can I do this without writing code?

The pipeline, yes, and the Tempo configuration is a single token. The enrichment step is real work, and it is the thing that makes the topic worth consuming.

Get your Tempo data into Kafka

Count your consumers first, because one does not justify a bus. Then decide about enrichment before anybody builds against the raw topic, since hours against issue keys force every consumer to join to Jira separately, which is the integration work a bus was meant to eliminate. Use a service account, document what it can see, create topics with deliberate retention, and tell consumers that people log time retrospectively.

Airbyte's connector catalog includes 600+ pre-built connectors, so time tracking can reach every system that depends on it. For the same source into a column store, see Tempo to ClickHouse, and for the issue data that gives worklogs meaning on the same bus, Jira to Kafka.

Start syncing now →