Looker to Kafka: How to Move Your Data

Moving Looker into Kafka publishes two quite different things: an inventory of your reporting estate, and the results of saved Looks. Looker answers both questions through its API, one instance at a time, but it cannot broadcast, so every service that wants a current picture polls it separately with its own key.

This guide covers the managed path with Airbyte. Two things shape the build: running a Look executes a real query against the database underneath Looker, which a schedule repeats indefinitely, and a topic holding those results is a sequence of complete snapshots rather than a stream of changes.

Looker to Kafka at a glance:

Why move data from Looker to Kafka?

Two situations account for most of these pipelines, and they use different halves of the connector.

The first is broadcasting the catalogue. Once an instance holds hundreds of dashboards, several systems want to know what exists: a governance tool, an internal search index, a service that flags dashboards nobody has opened in a year. Publishing the inventory once and letting each of them consume it beats three teams writing three pollers against the same API.

The second is distributing agreed numbers. A Look encodes filters and definitions somebody signed off, and publishing its results means several services consume the same figure rather than each reimplementing the logic and drifting apart. The test is still consumer count. With one system at the other end, a bus adds a moving part and earns nothing, and a direct pipeline into a warehouse is the shorter road. Kafka starts paying at three or four consumers.

What do you need before you start?

Four things, and the second decides what your consumers actually receive:

An API3 key and your Looker domain. The key is a client identifier and secret generated for a user in the Looker admin panel, and the domain is your instance address. Self-hosted instances are supported. The Looker source documentation points at Looker's own instructions for generating one.

A deliberate choice of which user owns that key. An API3 key carries its owner's permissions, covering both the content they can see and, when a Look runs, the rows they are allowed. On a bus this matters more than usual, because every consumer inherits that decision and none of them can see it.

Topics, created before the first sync. The destination writes to topics that already exist. Keep catalogue streams and Look results in separate topics, since they change at completely different rates and interest completely different consumers.

A shortlist of Looks and a sense of what each costs to run. Running a Look is not like reading a catalogue. It issues a query against the database Looker sits over, and on a schedule that query runs again every time. Know which Looks are cheap and which are not before you commit them to a recurring job.

If your organisation restricts access by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.

How do you build a Looker to Kafka pipeline in Airbyte?

Step 1: Decide which Looks are worth running on a schedule

A Look that takes forty seconds interactively is an occasional annoyance; the same Look on a fifteen-minute schedule is a recurring load on your production database forever. Look at the queries behind your candidates, keep the ones encoding logic genuinely worth distributing, and drop the rest. This is the decision that determines whether the pipeline is a quiet background job or something your database administrator asks you about.

Step 2: Configure the Looker source

Click Sources in the left navigation, then New Source, and select Looker, following adding a source. Supply the domain, client identifier and client secret. If you intend to publish Look results, name the Look here rather than expecting every saved Look in the instance to arrive on its own.

Step 3: Configure the Kafka destination

Click Destinations, then New Destination, and select Kafka, following adding a destination. Supply the bootstrap servers, security protocol and topic configuration. Messages are JSON, with no Avro option and therefore no schema registry, which is worth agreeing with consuming teams if the rest of your platform assumes registered schemas.

Step 4: Create the connection and split the two schedules

Click Connections, then New connection, select your streams and a sync mode. Use separate connections for the catalogue and for Look results. A BI inventory changes when somebody builds a dashboard, so daily is generous; a Look feeding a service may want a different rhythm entirely, and tying them together means running expensive queries on the catalogue's schedule.

Looker's own rate limits are unlikely to trouble you. The constraint that matters sits one layer down, on the database answering the queries, and it does not appear in any Airbyte log.

Why does running a Look cost more than reading the catalogue?

Because one describes and the other executes. Reading dashboards, folders and models asks Looker about itself, which is cheap and harmless however often you do it. Running a Look makes Looker issue a query against the database beneath it, so the cost lands on your warehouse or production database rather than on the BI tool.

A schedule turns that from an occasional cost into a permanent one. Interactively, an expensive Look runs when somebody chooses to look at it, which is self-limiting. Pointed at a bus on a short interval, it runs whether or not any consumer needed a fresh answer, and it keeps running long after the service that originally wanted it was decommissioned.

There is a second cost that is not about compute. The rows a Look returns depend on the data permissions of the user whose key executed it, so if your instance applies row-level restrictions by user attribute, the topic contains that user's view of the data and every consumer silently inherits it. Use a service account with documented access, and write down what it can see next to the topic name, because nothing downstream can work it out.

What does a topic of Look results actually contain?

A sequence of complete snapshots, not a stream of changes. Each sync runs the Look and publishes its whole result set, so a Look returning two hundred rows puts two hundred messages on the topic every run, whether or not a single value moved since the last one. Consumers that assume each message represents something new will count the same figures repeatedly.

Log compaction is the obvious instinct and it does not help here, because messages are keyed by an Airbyte-generated UUID rather than by anything meaningful. Compaction keeps the latest message per key, and with every message carrying a unique key there is nothing for it to collapse. The topic grows by a full result set per sync until retention removes the oldest.

So give consumers a way to tell batches apart. The message envelope carries the extraction timestamp alongside the record identifier and stream name, and that timestamp is what marks one snapshot off from the next: a consumer wanting current state reads the most recent extraction and discards the rest. Set retention to hold a sensible number of snapshots rather than a duration picked for a genuine event stream, and if a consumer needs per-entity ordering, rekey into a second topic on a field from the result rather than hoping the original topic provides it.

Frequently asked questions

Can I publish the actual data behind a dashboard?

Through a Look, yes. The connector can run a saved Look and return its result rows, which is unusual among BI connectors, and that query executes against your underlying database each time the sync runs.

Why does my topic keep receiving the same rows?

Because each sync republishes the Look's entire result set. Use the extraction timestamp in the message envelope to identify the latest snapshot, and treat earlier ones as history rather than as new activity.

Will log compaction keep only the newest version of each row?

No. Messages are keyed by a generated UUID, so every message has a unique key and compaction has nothing to collapse. Size your retention on the assumption that each sync adds a full copy.

Does this work with a self-hosted Looker?

Yes. Self-hosted instances are supported, so supply your own domain rather than a hosted address.

Can I do this without writing code?

The pipeline, yes, and it is entirely UI-driven. The consumers are yours to write, and anything needing current state has to handle snapshot boundaries itself, which is a modest piece of logic but not configuration.

Get your Looker data into Kafka

Keep the catalogue and Look results on separate connections and separate topics, because one is cheap and slow-moving and the other is a recurring query against your database. Use a service account and record what it can see, since every consumer inherits its data permissions invisibly. And tell consuming teams that a Look topic holds snapshots rather than changes, with the extraction timestamp marking where one ends and the next begins.

Airbyte's connector catalog includes 600+ pre-built connectors, so a reporting estate can reach every system that needs to know about it. For a comparable configuration-heavy source onto the same bus, see Genesys to Kafka, and for another BI catalogue into a warehouse, Metabase to BigQuery.

Start syncing now →