Datadog to BigQuery: How to Move Your Data

Moving data from Datadog to BigQuery is usually driven by one of two pressures: retention or joins. Datadog charges for how long it keeps your logs and metrics, and those windows are shorter than most compliance teams would like. Meanwhile the questions people actually want answered tend to involve joining observability data against things Datadog has never heard of, like customer tier, contract value, or deployment history.

BigQuery solves both. Long-term storage costs a fraction of extended monitoring retention, and once the data is a table it joins against anything else in your warehouse. This guide covers which Datadog streams are available, how to build custom query streams for the data the standard ones miss, and what to plan around.

Which Datadog streams can you sync?

The Datadog source reads from the Datadog API and exposes twelve streams. Knowing what each one holds saves you selecting everything and paying to move data nobody asked for.

Two things to note. Only Logs and AuditLogs respect the query, start date, and end date filters, so those settings do nothing for the other streams. And the connector does not support namespaces, meaning everything lands in the dataset you nominate rather than being organised by source structure.

How do you build custom query streams?

This is the most useful part of the connector and the least obvious. Beyond the twelve fixed streams, the Queries setting lets you define your own, and each one becomes a separate stream in the destination.

Each query needs a name, a data source, and a query string. Supported data sources are metrics, cloud_cost, logs, and rum. A metrics query looks like kubernetes_state.node.count{*}, while a RUM query filtering failed requests looks like @type:resource @resource.status_code:>=400.

The cloud_cost data source deserves particular attention, because it is how you get Datadog's cloud cost data into the same warehouse as your finance tables. That combination, infrastructure spend joined to revenue or customer data, is difficult to assemble any other way and is often the strongest business case for building this pipeline at all.

Define one query per question you want answered rather than one broad query you filter later. Each becomes its own BigQuery table, and narrow streams are cheaper to move and easier to model.

How do you set up the pipeline?

Step 1: Create Datadog credentials

You need two things, and they are not the same thing. An API key and an application key are both required. The API key identifies your organisation; the application key carries the permissions of the user who created it.

That second detail causes real outages. Because the application key inherits its creator's role, a permission change or an offboarded employee can break the connection with an authorization error that looks nothing like a permissions problem. Create the key under a service account with a stable role rather than a person's login.

Step 2: Configure the BigQuery destination

The BigQuery destination needs a service account holding the BigQuery User and BigQuery Data Editor roles, with its key in JSON format. Set a Project ID, a Dataset Location, and a Default Dataset ID. The location is permanent, so if you plan to join Datadog data against existing datasets, create everything in the same location now rather than discovering the constraint later.

For loading, Batched Standard Inserts is the simpler choice and lets BigQuery manage its own staging files. GCS Staging gives you control over where staging files live, but the bucket must use Google-managed encryption, since customer-managed keys are not supported.

Step 3: Select streams and run

Choose your streams, set incremental where the stream supports it, and pick a frequency. If your organisation restricts outbound access by IP, add the Airbyte Cloud IP addresses to the relevant allow lists first.

What should you plan around?

Log volume is the thing that will surprise you. Datadog holds a great deal of it, and a first sync with a wide query and no start date can pull far more than you expected into BigQuery. Set a start date, use the query filter to narrow to what you actually need, and let the first run finish before scheduling anything frequent.

It is also worth knowing this is a Marketplace connector, community-maintained rather than certified. It is actively updated, but treat monitoring as part of the build rather than an afterthought.

Frequently asked questions

Why do I need both an API key and an application key?

Datadog uses them together. The API key identifies your organisation and the application key carries the permissions of whoever created it. Create the application key under a service account, because a key tied to an individual will break when their role changes or they leave.

Can I sync Datadog cloud cost data?

Yes, through a custom query with cloud_cost as the data source. It is not one of the twelve fixed streams, so you have to define it yourself. It is also one of the most valuable things you can put in a warehouse, because infrastructure spend becomes joinable to revenue and customer data.

Why is my date filter not working on some streams?

The query, start date, and end date settings only apply to the Logs and AuditLogs streams. Other streams ignore them, so narrow those through stream selection rather than expecting the filters to apply.

How do I keep the first sync from pulling everything?

Set a start date and use the query filter to scope logs to what you need. Log streams are where volume gets away from people, and a broad unfiltered first run can move far more than intended.

Does the connector support namespaces?

No. All streams land in the dataset you nominate on the destination. Use stream prefixes if you need to distinguish Datadog tables from others sharing that dataset.

Get your Datadog data into BigQuery

Create the application key under a service account, scope your log queries before the first run, and define custom queries for the data the fixed streams miss. Consolidating onto Snowflake instead? See our guide to Datadog to Snowflake. Replicating a database into the same warehouse? See PostgreSQL to BigQuery.

Start syncing now →