Datadog to Kafka: How to Move Your Data
Move Datadog into Kafka with Airbyte. Why your query is a capacity decision, how volume spikes break topics rather than tables, and when to skip the bus.
Moving Datadog into Kafka puts a slice of your observability data somewhere several systems can consume it. Datadog holds everything and shows it beautifully, but it retains data for a bounded period and its own integrations are built for alerting rather than for handing the same records to four different services.
This guide covers the managed path with Airbyte. One thing shapes this build more than anything else: the query you write decides how much data moves, and observability data has no natural size, so an unbounded query lands on a topic rather than in a table and the consequences arrive faster.
Datadog to Kafka at a glance:
Why move data from Datadog to Kafka?
Two situations account for most of these pipelines, and it is worth saying plainly that this is a narrower pairing than most.
The first is distributing a deliberately narrow signal. Deployment markers, a specific class of monitor event, a bounded log query: something small and well defined that an incident tool, a reporting warehouse and an internal service all want. Each of them polling Datadog separately means three sets of keys and three implementations of the same query, and publishing once is tidier.
The second is outliving Datadog's retention, moving a chosen subset somewhere it can sit beside business data for longer than the platform keeps it. Outside those two, this is often the wrong tool. Datadog already has webhooks and integrations built for pushing events to other systems quickly, and this pipeline is scheduled batches rather than a live feed. If the destination is a single warehouse, the bus is pure overhead and Datadog to BigQuery gets you there with fewer moving parts.
What do you need before you start?
Four things, and the last one is the one people skip:
Two keys, which are not the same thing. An API key identifies your organisation, and an application key carries the permissions of the user who created it. You need both, and the application key is the one that decides what the pipeline can actually see. The Datadog source documentation covers generating each.
Your Datadog site. Datadog runs regional instances and your data lives in exactly one of them. Point the connector at the wrong site and the failure presents as an authentication error, which sends people off rotating perfectly good keys for an afternoon.
Topics, created in advance and sized for the volume. The destination writes to topics that already exist. Partition count and retention deserve more thought here than on a typical pipeline, because the volume is whatever your query returns rather than the size of some fixed table.
A bounded query, and a measurement of what it returns. Run it in Datadog first and look at how many records come back over a representative day. This takes ten minutes and is the difference between a pipeline that behaves and one that fills a broker.
If your organisation restricts access by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.
How do you build a Datadog to Kafka pipeline in Airbyte?
Step 1: Bound the query and measure what it returns
Write the query in Datadog, restrict it by service, environment, status or whatever else narrows it honestly, and count what comes back over a normal day. Then consider an abnormal one, because the day a service starts erroring is the day your log query returns fifty times its usual volume, and that is precisely when you least want a pipeline misbehaving. Everything downstream is sized from this number.
Step 2: Configure the Datadog source
Click Sources in the left navigation, then New Source, and select Datadog, following adding a source. Supply the API key, the application key and your site, then the query and streams you settled on. If authentication fails with credentials you are confident about, check the site before you touch the keys.
Step 3: Configure the Kafka destination
Click Destinations, then New Destination, and select Kafka, following adding a destination. Supply the bootstrap servers, security protocol and topic configuration. Output is JSON only, and each message wraps the record alongside its identifier, the extraction timestamp and the stream name, so consumers read through an envelope rather than finding Datadog's own shape at the top level.
Step 4: Create the connection and schedule against the volume
Click Connections, then New connection, select your streams and a sync mode. Frequent syncs of a small bounded query are fine. Frequent syncs of a broad one produce a great deal of traffic and give consumers no time to catch up between batches, so the interval and the query have to be chosen together rather than one after the other.
Set an alert on consumer lag from the start. This is the pipeline most likely to outrun the things reading from it, and lag is the earliest honest signal that it has.
What decides how much data this pipeline moves?
Your query, and nothing else. Most sources have a natural size: a customer table holds however many customers you have, and a campaign list is as long as your marketing team has been busy. Observability data has no such ceiling. It is generated continuously by machines, and the only thing standing between a pipeline and an unbounded firehose is how narrowly the query is written.
That makes the query a capacity decision rather than a filter. Narrowing by service, environment and status is not tidiness, it is the mechanism that determines broker storage, partition counts and whether consumers keep up. A query written loosely because it was easier to reason about will happily return millions of records, and nothing in the configuration will warn you.
The volume is also not stable, which is the part that catches people. Observability data spikes exactly when systems misbehave, so the pipeline's heaviest day is an incident day. Size for the spike rather than the average, and write the query so that a noisy service cannot quietly multiply what it returns.
Why does that volume hurt more on a topic than in a table?
Because a warehouse absorbs an unexpected volume and hands you a larger bill, while a topic has fixed retention and consumers that must keep pace. Ten times the expected records arrives as consumer lag, then as eviction: retention starts discarding the oldest messages while a slow consumer is still working through them, and that consumer misses data permanently with nothing in its own logs to say so.
Partition count matters more here than on a slower pipeline, because consumers scale by partition and a topic provisioned for a modest signal cannot be widened usefully once the messages are already arriving. Deciding this from your measured spike, rather than from the average day, is the difference between adding a consumer instance and rebuilding the topic.
There is also a circularity worth naming. You are using infrastructure to carry the telemetry that tells you how your infrastructure is doing, so a bad day for the cluster is a bad day for the data describing it, exactly when you want that data most. Keep the genuinely operational path in Datadog's own alerting, and let this pipeline serve consumers that can tolerate arriving late, which is what a scheduled batch pipeline can honestly promise.
Frequently asked questions
Why do I need two different keys?
They do different jobs. The API key identifies the organisation and the application key carries the permissions of the user who created it, so the second one determines what the pipeline can read.
My credentials are correct but authentication fails. Why?
Check the site. Datadog is regional and querying the wrong instance produces an error that reads like an authentication problem, so confirm the region before regenerating anything.
How much data will this actually move?
Entirely up to your query, because observability data has no natural size. Run the query in Datadog and count the results over a normal day and a busy one before sizing anything downstream.
Can I use this to trigger alerts in other systems?
Not well. Syncs run on a schedule, so latency is your sync interval, and Datadog's own webhooks and integrations exist precisely for pushing events out promptly. Use those for anything operational.
Can I do this without writing code?
The pipeline, yes, and it is entirely UI-driven. The work that is not configuration is writing a query narrow enough to be safe and building consumers that cope when a busy day multiplies the volume.
Get your Datadog data into Kafka
Be honest about whether you want a bus at all, because Datadog's own integrations handle the operational cases better and a single warehouse consumer does not need one. If you do, the query is the whole game: bound it, measure it on a busy day, size partitions and retention from that number rather than the average, and watch consumer lag from the first sync. Leave anything that must arrive quickly to Datadog itself.
Airbyte's connector catalog includes 600+ pre-built connectors, so operational data can reach the systems that need it without bespoke collectors. For the same source into a warehouse, see Datadog to Snowflake, and for comparable engineering telemetry onto the same bus, Jenkins to Kafka.
Integrate with 700+ apps using Airbyte
Move data from 700+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.