Datadog to PostgreSQL: How to Move Your Data

Moving Datadog into PostgreSQL works for a narrow slice and nothing wider. A tool that needs to know which services errored overnight, or a record of particular incidents kept beyond your Datadog retention, are reasonable; a copy of your observability data is not.

This guide covers the managed path with Airbyte. Two things shape the build: the query you write is the only thing standing between this database and an unmanageable volume, and the copy has to do something Datadog itself will not.

Datadog to PostgreSQL at a glance:

Why move data from Datadog to PostgreSQL?

One situation genuinely suits this, and the other is worth naming.

The good case is feeding software a specific signal. A deployment gate checking error rates, a status page reading a handful of metrics, or a service that needs recent incident data beside your own records all want a small, current table nearby.

The poor case is analysis at any scale, because observability volumes overwhelm this destination quickly. For trend analysis or modelling over logs and metrics, Datadog to Databricks is the right home and will cost you far less trouble.

What do you need before you start?

Four things, and the first is the one that keeps this viable:

A query narrow enough to be sensible. Queries become streams, each producing its own table, and the query string is where you filter. Written broadly, it will fill this database and keep going. The Datadog source documentation covers the configuration.

Both an API key and an application key. These are different things in Datadog, created in different places, and the API key alone is not sufficient for read endpoints.

The correct site for your account. Datadog is regional, and a wrong site setting presents as an authentication failure rather than as anything mentioning regions.

A named consumer and an index plan. If software is not reading this, the copy has no purpose that Datadog's own interface does not already serve.

If your database restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.

How do you build a Datadog to PostgreSQL pipeline in Airbyte?

Step 1: Write the query before anything else

Compose and test your query in Datadog first, checking how many records it returns over a day, because that number is what this pipeline will produce every day forever. A query returning a few thousand rows is a comfortable table; one returning a few million is a database nobody wants in six weeks. This is the only filter you get.

Step 2: Configure the Datadog source

Click Sources in the left navigation, then New Source, and select Datadog, following adding a source. Supply both keys, the site and a start date, then define your queries with a name, a data source and the query string. Each becomes a stream, so name them for what they mean rather than what they filter.

Step 3: Configure the PostgreSQL destination

Click Destinations, then New Destination, and select PostgreSQL, following adding a destination. Supply the host, port, database and credentials, and give this its own schema. Expect metadata columns to be added, so application code should request fields by name.

Step 4: Create the connection and watch the row count

Click Connections, then New connection, select your streams and a sync mode. Then check the table size after a week rather than after a quarter, since a query that seemed narrow in testing behaves differently against a busy production week.

Agree a retention rule while you are there, because nothing removes old rows unless somebody writes that job.

Why is the query the whole pipeline?

Because it is the only place volume is decided, and the volumes involved are unlike anything else in this catalogue. Observability data is generated continuously by machines rather than occasionally by people, and a moderately busy platform produces more log lines in a day than most business systems produce records in a year.

Against a destination sized for around ten gigabytes, that arithmetic is unforgiving. A query matching all errors across all services is not a filter in any meaningful sense, and the resulting table will be unusable long before anybody decides to do something about it. Narrow means one service, one condition, or one metric.

The queries-as-streams design helps if you use it deliberately. Several narrow queries, each producing its own table, is a better arrangement than one broad query somebody filters afterwards, because the filtering happens before the data moves rather than after it lands. And check what your application key can reach while testing, since users have reported the connector attempting the audit logs endpoint even when only logs were configured.

What is this copy actually for?

Something Datadog will not do, which is a shorter list than people assume. Datadog queries its own data faster than this database ever will, visualises it better and alerts on it natively, so a copy that merely duplicates those capabilities is a pipeline with no return.

Two things genuinely qualify. Joining observability signals to your own application data, so a service can act on both without calling two APIs, and keeping a narrow record past your Datadog retention, which is plan-dependent and rarely as long as an auditor or a post-incident review would like.

Both of those imply a small table serving software rather than a dataset people browse. So index for the lookups that software makes, keep the schema separate from your application's own, and resist the request to add one more query each time somebody wants a number, since that is how a deliberate slice becomes an accidental observability platform nobody chose to build.

Frequently asked questions

Is PostgreSQL a sensible destination for Datadog?

For a narrow query feeding software, yes. For observability data generally, no, since the volumes exceed what this destination is comfortable holding.

My table is growing alarmingly.

The query is too broad, since that is the only filter in this pipeline. Narrow it to one service or condition, and add a retention job to remove old rows.

Authentication fails but my key is right.

Check the site setting, since Datadog is regional and a mismatch looks like a credentials problem. Also confirm you supplied both an API key and an application key.

A stream fails mentioning audit logs.

Users have reported the connector querying that endpoint even when only logs were configured, which fails without audit log access. Check what your application key can reach.

Can I do this without writing code?

The pipeline, yes, though the query is written in Datadog's syntax. The indexes and retention job are yours and both matter here.

Get your Datadog data into PostgreSQL

Write and test the query before configuring anything, counting what it returns in a day, because that is the only filter between this database and observability volumes. Supply both keys and the right site. Then be clear what the copy does that Datadog will not, index for the software reading it, and write a retention job so a deliberate slice stays one.

Airbyte's connector catalog includes 600+ pre-built connectors, so operational signals can reach the systems that act on them. For the same source into a warehouse, see Datadog to BigQuery, and for the same source into a warehouse with fine-grained controls, Datadog to Snowflake.

Start syncing now →