Datadog to Elasticsearch: How to Move Your Data

Moving Datadog into Elasticsearch gives you searchable observability history beyond what your retention period covers. Datadog is excellent while data is fresh and expires it on a schedule you pay to extend, which is awkward when an investigation reaches back further than the platform still holds.

This guide covers the managed path with Airbyte. Two things shape the build, and the first is a blocker: this destination exists on Airbyte Core and PyAirbyte only. The second is that observability data has no natural size, so your query decides how large the index becomes.

Datadog to Elasticsearch at a glance:

Why move data from Datadog to Elasticsearch?

Two situations account for most of these pipelines.

The first is retention at a price you choose. Keeping observability data longer in Datadog is a subscription conversation, and for a narrow slice you genuinely need to keep, an index you operate yourself is often considerably cheaper than extending retention across everything.

The second is that this is a natural pairing, unusually. Search over log and event text is exactly what Elasticsearch is built for, so unlike most destinations this one does the same kind of work as the source. If you want aggregation and joins against business data rather than search, a warehouse suits that better and Datadog to BigQuery is the route.

What do you need before you start?

Four things, and the first decides whether you can proceed at all:

An Airbyte Core or PyAirbyte deployment. The Elasticsearch destination is not offered on Standard, Plus, Pro or Enterprise Flex. The Elasticsearch destination documentation sets out the support position, and no configuration works around it.

Two Datadog keys, which do different jobs. An API key identifies the organisation and an application key carries the permissions of the user who created it, so the second decides what the pipeline can read. Both are required together.

Your Datadog site. Datadog runs regional instances and your data lives in exactly one. Pointing at the wrong one produces an error that reads like an authentication failure and sends people rotating perfectly good keys.

An index mapping and a lifecycle policy. Both matter more here than on a modest dataset, since this is the kind of data that grows quickly and never stops. Deciding them before the first sync saves reindexing and saves a disk.

If your cluster restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.

How do you build a Datadog to Elasticsearch pipeline in Airbyte?

Step 1: Design the mapping and the lifecycle policy together

Decide which fields are analysed text and which are keywords, then decide how long indexes live and when they roll over. Those two decisions belong together for observability data because the volume is continuous, and an index designed without a lifecycle policy is a disk filling up on a schedule nobody chose. Time-based indexes with rollover are the conventional answer and they exist precisely for this shape of data.

Step 2: Configure the Datadog source

Click Sources in the left navigation, then New Source, and select Datadog, following adding a source. Supply the API key, application key and site, then the query and streams you want. If authentication fails with credentials you trust, check the site before touching the keys.

Step 3: Configure the Elasticsearch destination

Click Destinations, then New Destination, and select Elasticsearch, following adding a destination. Supply the endpoint and authentication, and point it at indexes whose mapping you have already created rather than letting Elasticsearch guess types from the first documents that happen to arrive.

Step 4: Create the connection and watch the index grow

Click Connections, then New connection, select your streams and a sync mode. Then watch index size and shard count over the first fortnight rather than assuming the initial sizing was right, because the volume this pipeline delivers is whatever your query happens to match on a given day.

Keep Datadog's own alerting as the operational path. This pipeline is a searchable archive, not a monitoring system, and scheduled batches cannot be either.

How large will the index actually get?

Entirely as large as your query allows. Most sources have a natural size, because a customer table holds however many customers you have. Observability data is generated continuously by machines and has no such ceiling, so the query is not a filter on a fixed dataset, it is the thing that defines how much data exists to move.

For an index, that makes sizing a capacity decision rather than a detail. Shard count, index size and how quickly you roll over all follow from the daily volume, and an index sized for a quiet week behaves badly when a service starts erroring and produces fifty times its usual output. That spike arrives on exactly the day somebody most wants to search the data.

So narrow the query by service, environment and status until it covers what you genuinely need to keep, then size for the busy day rather than the average. Time-based indexes help considerably here, since a rollover policy lets a bad week produce more indexes rather than one enormous one, and a retention policy deletes the oldest without anybody having to remember.

How should log and event fields be mapped?

By what people do with them, which for observability data divides cleanly. The message body is analysed text, because engineers search it for words they half remember from an error. Service name, host, environment and status are keywords, because those are filters applied to narrow a search rather than terms anybody types partially.

Getting it backwards produces behaviour that looks like a bug. A service name mapped as analysed text matches partially, so searching for one service returns results from three others whose names share a word. A message body mapped as a keyword matches nothing unless somebody reproduces it exactly, which during an incident nobody will. Both fail quietly by returning the wrong results rather than no results.

Use multi-field mapping where a value needs both behaviours, which for this data usually means the service or host name, searchable in part and matchable exactly when somebody filters a dashboard. Decide these during the mapping design, since adding them later means reindexing, and reindexing an observability dataset is an operation rather than an afternoon.

Frequently asked questions

Can I use this destination on any Airbyte plan?

No. Elasticsearch is available on Airbyte Core and PyAirbyte only, so confirm your deployment before planning around it.

My credentials are correct but authentication fails.

Check the site, since Datadog is regional and querying the wrong instance produces an error that reads like an authentication problem.

How do I stop the index growing without limit?

Narrow the query first, then use time-based indexes with rollover and a retention policy. The volume is whatever your query matches, so both halves matter.

Can I alert from this index?

Not usefully. Syncs are scheduled batches, so latency is your sync interval. Keep operational alerting in Datadog and treat this as a searchable archive.

Can I do this without writing code?

The pipeline, yes. The index mapping and lifecycle policy are configuration you write in Elasticsearch, and they are what keeps this manageable as the data accumulates.

Get your Datadog data into Elasticsearch

Confirm your deployment supports this destination first, since Core and PyAirbyte are the only options. Check the site before blaming the keys. Then treat the query as a capacity decision, because observability data has no natural ceiling, and pair a narrow query with time-based indexes and a retention policy. Map message bodies as analysed text and service, host and status as keywords, and leave operational alerting where it belongs.

Airbyte's connector catalog includes 600+ pre-built connectors, so operational history can be searched long after the platform expired it. For the same source into a lakehouse, see Datadog to Databricks, and for comparable audit data into the same destination, AWS CloudTrail to Elasticsearch.

Start syncing now →