Move Kafka data anywhere it matters.

Send Kafka data to any warehouse, lake or AI app.

Sync Kafka free Read the docs

Source

Kafka

Snowflake
BigQuery
Postgres
Databricks
Claude

Destinations: 50+
Standard support: 0.4.2

Overview

What is the Airbyte Kafka connector?

The Airbyte Kafka connector extracts data from Kafka and loads it into 50+ destinations such as Snowflake, BigQuery, Databricks or Postgres, with full refresh and incremental sync. It publishes tables and events you can select per connection.

01
Sync to your warehouse
Land Kafka data in 50+ destinations on a schedule you control.
02
Incremental syncs
Pull only the records that changed since the last run instead of reloading everything.
03
One authorization
Authenticate once; every scheduled sync keeps running.
04
Cloud or self-hosted
Run the Kafka connector on OSS.

Setup

Live in ten steps

First, you need to have a Kafka source connector that you want to connect to Airbyte. You can download the connector from the Apache Kafka website or any other reliable source.
Once you have the Kafka source connector, you need to configure it with the necessary settings such as the Kafka broker URL, topic name, and other relevant parameters.
Next, you need to create a new connection in Airbyte by clicking on the ""New Connection"" button on the dashboard.
Select the Kafka source connector from the list of available connectors and provide the necessary details such as the connector name, version, and configuration settings.
After providing the required details, click on the ""Test Connection"" button to ensure that the connection is established successfully.
If the connection is successful, you can proceed to create a new pipeline by clicking on the ""New Pipeline"" button on the dashboard.
Select the Kafka source connector as the source and choose the destination connector where you want to send the data.
Configure the pipeline settings such as the data mapping, transformation, and other relevant parameters.
Once you have configured the pipeline, click on the ""Run"" button to start the data transfer process.
Monitor the pipeline progress and ensure that the data is transferred successfully from the Kafka source connector to the destination connector.

Configuration fields

Kafka configuration fields
Field	Type	Description
bootstrap_servers*	string	A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping—this list only impacts the initial hosts used to discover the full set of servers. This list should be in the form host1:port1,host2:port2,.... Since these servers are just used for the initial connection to discover the full cluster membership (which may change dynamically), this list need not contain the full set of servers (you may want more than one, though, in case a server is down).
protocol*	object	The Protocol used to communicate with brokers.
subscription*	object	You can choose to manually assign a list of partitions, or subscribe to all topics matching specified pattern to get dynamically assigned partitions.
auto_commit_interval_ms	integer	The frequency in milliseconds that the consumer offsets are auto-committed to Kafka if enable.auto.commit is set to true.
auto_offset_reset	string	What to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server - earliest: automatically reset the offset to the earliest offset, latest: automatically reset the offset to the latest offset, none: throw exception to the consumer if no previous offset is found for the consumer's group, anything else: throw exception to the consumer.
client_dns_lookup	string	Controls how the client uses DNS lookups. If set to use_all_dns_ips, connect to each returned IP address in sequence until a successful connection is established. After a disconnection, the next IP is used. Once all IPs have been used once, the client resolves the IP(s) from the hostname again. If set to resolve_canonical_bootstrap_servers_only, resolve each bootstrap address into a list of canonical names. After the bootstrap phase, this behaves the same as use_all_dns_ips. If set to default (deprecated), attempt to connect to the first IP address returned by the lookup, even if the lookup returns multiple IP addresses.
client_id	string	An ID string to pass to the server when making requests. The purpose of this is to be able to track the source of requests beyond just ip/port by allowing a logical application name to be included in server-side request logging.
enable_auto_commit	boolean	If true, the consumer's offset will be periodically committed in the background.