Connect Kafka data with any data source.

Replicate any source into Kafka without custom pipelines.

Load data into Kafka Read the docs

Sources: 590+
Standard support: 0.1.11

Overview

What is the Airbyte Kafka connector?

The Airbyte Kafka connector loads data from 590+ Airbyte sources into Kafka, with incremental sync on a schedule you control.

01
Load from any source
Move data into Kafka from 590+ Airbyte sources on a schedule you control.
02
Incremental syncs
Pull only the records that changed since the last run instead of reloading everything.
03
Cloud or self-hosted
Run the Kafka connector on OSS.

Setup

Live in nine steps

First, you need to have an Apache Kafka destination connector installed on your system. If you don't have it, you can download it from the Apache Kafka website.
Once you have the Apache Kafka destination connector installed, you need to create a new connection in Airbyte. To do this, go to the Connections tab and click on the "New Connection" button.
In the "New Connection" window, select "Apache Kafka" as the destination connector and enter the required connection details, such as the Kafka broker URL, topic name, and authentication credentials.
After entering the connection details, click on the "Test Connection" button to ensure that the connection is working properly.
If the connection test is successful, click on the "Save" button to save the connection.
Once the connection is saved, you can create a new pipeline in Airbyte and select the Apache Kafka destination connector as the destination for your data.
In the pipeline configuration, select the connection you created in step 3 as the destination connection.
Configure the pipeline to map the source data to the appropriate Kafka topic and fields.
Once the pipeline is configured, you can run it to start sending data to your Apache Kafka destination.

Configuration fields

Kafka configuration fields
Field	Type	Description
acks*	string	The number of acknowledgments the producer requires the leader to have received before considering a request complete. This controls the durability of records that are sent.
batch_size*	integer	The producer will attempt to batch records together into fewer requests whenever multiple records are being sent to the same partition.
bootstrap_servers*	string	A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping—this list only impacts the initial hosts used to discover the full set of servers. This list should be in the form host1:port1,host2:port2,.... Since these servers are just used for the initial connection to discover the full cluster membership (which may change dynamically), this list need not contain the full set of servers (you may want more than one, though, in case a server is down).
buffer_memory*	string	The total bytes of memory the producer can use to buffer records waiting to be sent to the server.
client_dns_lookup*	string	Controls how the client uses DNS lookups. If set to use_all_dns_ips, connect to each returned IP address in sequence until a successful connection is established. After a disconnection, the next IP is used. Once all IPs have been used once, the client resolves the IP(s) from the hostname again. If set to resolve_canonical_bootstrap_servers_only, resolve each bootstrap address into a list of canonical names. After the bootstrap phase, this behaves the same as use_all_dns_ips. If set to default (deprecated), attempt to connect to the first IP address returned by the lookup, even if the lookup returns multiple IP addresses.
compression_type*	string	The compression type for all data generated by the producer.
delivery_timeout_ms*	integer	An upper bound on the time to report success or failure after a call to 'send()' returns.
enable_idempotence*	boolean	When set to 'true', the producer will ensure that exactly one copy of each message is written in the stream. If 'false', producer retries due to broker failures, etc., may write duplicates of the retried message in the stream.

Hybrid deployment

Sovereign by architecture, not by policy.

The data plane runs in your environment. Records, credentials and keys stay with you; the control plane only receives configuration and job metadata.

Your environment

DataConnectorsCredentialsKeys

↕ metadata only

Airbyte control plane

OrchestrationMonitoringUI

FAQ

Kafka connector questions

What data can I load into Kafka with Airbyte?

Airbyte moves data from any supported source into Kafka on a schedule you control, with full refresh and incremental sync where the destination supports it.

Does the Kafka connector support incremental sync?

Yes. The Kafka destination supports incremental sync, so Airbyte can load only the records that changed since the last sync.

Where can I run the Kafka connector?

Run the Kafka connector on OSS.

Didn't find your answer? Talk to sales