BigQuery to Kafka: How to Stream Your Data

Most teams who want BigQuery data in Kafka are running the warehouse backwards. Data went in for analysis, and now something operational needs it: a fraud service, a personalisation cache, a consumer group that was never given a database of its own. It is a legitimate pattern, and Airbyte will build the pipeline with no code.

It also meets a hard edge quickly. Three characteristics of the destination govern every decision you make here, and it is much cheaper to see them before you build than after.

Why is the destination append-only?

The Kafka destination supports full refresh append and incremental append. Overwrite, overwrite with dedup, and append with dedup are all unavailable.

This is less a limitation of the connector than an honest reflection of what a topic is. You cannot overwrite a log. Every sync adds messages, and reconciling multiple versions of the same row is work that happens downstream, in your consumer or in a compaction step.

Two other constraints are worth stating plainly. The connector writes JSON and only JSON, with Avro described in the documentation as a future addition, so consumers expecting Avro against a schema registry will need a translation step. And it is listed as available on Core and PyAirbyte but not on the Standard, Plus, Pro, or Enterprise Flex plans, which means this is a self-managed pipeline in practice.

What does a record look like on the topic?

Each Airbyte stream is written to a Kafka topic. The message key is the UUID Airbyte assigned the record. The value is an envelope with four fields: _airbyte_ab_id, _airbyte_emitted_at, _airbyte_stream, and _airbyte_data, which holds your actual row as a JSON blob.

Consumers have to unwrap that data field. Teams routinely assume the row arrives at the top level and discover otherwise during integration testing, so put the envelope shape in the consumer contract before anyone writes against it.

How should you name your topics?

The topic_pattern field takes either a literal name or a template. Set it to a fixed string and every stream from every namespace lands in one topic. Use the {namespace} and {stream} variables and messages fan out instead, so a pattern of airbyte_syncs/{namespace}/{stream} sends namespace n1, stream s1 to the topic airbyte_syncs/n1/s1.

Dynamic patterns bring an operational dependency. If a schema change introduces a new stream and the matching topic does not exist, the sync fails unless auto.create.topics.enable is on at the broker. Airbyte's documentation is direct that enabling it is not recommended in production, and suggests creating topics manually instead. Dynamic naming is therefore convenient in development and a standing task in production. A hardcoded topic avoids the problem entirely.

Note also that a naming transformation is applied to the target topic through the StandardNameTransformer, which replaces some special characters. Check the resulting topic name rather than assuming your pattern survives verbatim.

What does extraction from BigQuery cost?

The BigQuery source supports full refresh and cursor-based incremental sync. It does not support change data capture, because BigQuery has no change log to read.

Incremental works by issuing a query filtered on the cursor, fetching only rows above the last value seen. Two consequences follow. Every sync is a billed BigQuery query, so sync frequency is a spending decision as much as a latency one. And if your table is partitioned, choosing a cursor aligned with the partition column lets BigQuery prune partitions rather than scanning the table, which the documentation illustrates by contrasting a filter on the partition column against one on an unrelated timestamp.

There is one quiet trap. Records whose cursor field is null are skipped rather than failing the sync. A nullable updated timestamp will silently drop rows, and nothing in the sync log will tell you it happened.

Setting up the connection

On the source side, create a Google Cloud service account with the BigQuery User and BigQuery Data Editor roles, generate a JSON key, and supply it with your project ID. Setting a default dataset ID is optional, but the documentation notes it dramatically speeds up schema discovery, which is worth doing on a large project.

On the destination side, confirm your brokers are reachable from Airbyte, that the writing principal has produce permission, and that your topics exist. Then configure bootstrap servers, topic pattern, security protocol, and SASL settings. The connector exposes most of the standard Kafka producer configuration, including acks, idempotence, compression, batch size, and linger. Some SSL configurations are noted as not yet available, so check yours against the connector before committing to a design.

When should you use something else?

If you need exactly-once semantics, Avro against a schema registry, or sub-minute latency, this is not the pipeline. Airbyte syncs are batch jobs, and each one produces a burst of messages rather than a steady stream. Datastream, a Pub/Sub bridge, or a purpose-built streaming tool will serve those requirements better.

If what you actually want is BigQuery data in a queryable system rather than a log, a database or warehouse destination is both a better fit and a better-supported one.

Frequently asked questions

Can I get exactly-once delivery into Kafka?

No. The connector supports append semantics only, and Airbyte's delivery guarantee is at least once. You can enable idempotence at the producer level, but end-to-end deduplication belongs in your consumer or in a downstream compaction step.

Why does my sync fail after a source schema change?

A new or renamed stream resolves to a topic that does not exist yet. Either pre-create the topic or enable automatic topic creation at the broker, understanding that Airbyte's documentation advises against the latter in production.

Does the Kafka destination work on paid Airbyte Cloud plans?

It is listed as available on Core and PyAirbyte only. On Standard, Plus, Pro, and Enterprise Flex it is not available, so plan on a self-managed deployment.

How do I keep BigQuery query costs down?

Use incremental sync with a cursor aligned to your table's partition or clustering column, and reduce sync frequency. Every sync issues a billed query, so frequency and cost move together.

Get your BigQuery data into Kafka

Match the connector's shape to your use case, pre-create your topics, and align your cursor with the table's partitioning. Want BigQuery data somewhere queryable instead? See our guide to BigQuery to PostgreSQL. Streaming from a different source into the same topics? See n8n to Kafka.

Start syncing now →