Datadog to Amazon Redshift: How to Move Your Data
Move Datadog into Amazon Redshift with Airbyte. Why the query decides your volume, the two key types, mandatory S3 staging and syncing inside retention.
Moving Datadog into Amazon Redshift is usually about outliving retention. Datadog keeps metrics and logs for a period that suits operating a system rather than analysing one, and questions about how reliability has changed across a year, or how incidents map onto releases, need data the platform no longer holds.
This guide covers the managed path with Airbyte. Two things shape it: the query you write on the source side decides what arrives, and Redshift cannot be loaded without a staging bucket, which adds an S3 prerequisite to the project.
Datadog to Amazon Redshift at a glance:
Why move data from Datadog to Amazon Redshift?
Two situations account for most of these pipelines.
The first is retention. Observability platforms price on ingestion and retention, so keeping years of history inside Datadog is expensive in a way that keeping it in a warehouse is not. A pipeline accumulating what matters gives you the long view at storage prices.
The second is joining operational data to everything else. Incident volume against release frequency, error rates against customer churn, infrastructure cost against usage: none of these are questions Datadog can answer, because the other half lives in your warehouse.
What do you need before you start?
Two credentials that are easy to confuse, and an AWS prerequisite:
An API key and an application key. Both are needed and they are different things: the API key identifies your organisation while the application key is associated with the user who created it and carries their permissions. The Datadog source documentation covers where to find them.
Your Datadog site. Datadog runs regional sites and the API host differs accordingly, so the site setting has to match the one your organisation uses. Getting it wrong produces an authentication error rather than anything mentioning geography.
A query, and a realistic view of what it returns. The logs and metrics streams take a query defining what to retrieve, and that is the single biggest lever on volume. An unfiltered log query against a busy estate is an enormous amount of data to move and store.
An S3 staging bucket and IAM permissions. The Redshift destination writes data to S3 as files with a manifest, then issues a COPY command, which is Redshift's own recommended approach. You need put, get, delete and list on the bucket, in the same region as the cluster.
Note also that an SSH tunnel covers the SQL connection only. Staging traffic reaches S3 over HTTPS regardless, so a network design assuming everything passes through a bastion is wrong.
How do you build a Datadog to Redshift pipeline in Airbyte?
Step 1: Write the query before anything else
Decide precisely what you want to keep for years, then write the query that returns it. Errors from your production services is a reasonable scope; all logs from everything is not, and on a warehouse that charges for storage the difference compounds every day the pipeline runs.
Step 2: Configure the Datadog source
Click Sources in the left navigation, then New Source, and select Datadog, following adding a source. Supply the API key, application key, site, your query and a start date. If the connection test fails, check the site before regenerating either key.
Step 3: Configure the Redshift destination
Click Destinations, then New Destination, and select Redshift, following adding a destination. Supply the cluster host, database, schema and credentials, along with your S3 bucket and IAM key. Staging files are removed after a successful COPY by default, and can be retained if you need them for debugging.
Step 4: Create the connection and sync inside retention
Click Connections, then New connection, select your streams and an incremental sync mode where available. Frequency here is governed by Datadog's retention rather than by dashboard freshness, since anything aged out before a sync reads it is gone from both systems.
Set sort and distribution keys on the resulting tables, because observability data is queried by time almost exclusively and on Redshift that choice matters more than anything in the pipeline.
Why is the query the whole design?
Because observability data has no natural size. A busy estate generates as much log volume as you are willing to keep, so unlike most sources there is no fixed dataset waiting to be copied. What you write in the query field is what exists downstream, and nothing else constrains it.
That makes an unfiltered query the most expensive mistake available here, and one that compounds. Every day the pipeline runs adds to a table you are paying to store, and the day somebody notices is the day you also have a deletion exercise rather than just a configuration change.
So the useful discipline is starting from the question. If the purpose is tracking error rates by service over time, query errors by service rather than everything and aggregating later, because the aggregate is what you want to keep and the raw volume is what you are trying to avoid.
Why is S3 staging not optional?
Because it is how Redshift is designed to be loaded. The destination writes data to S3 as files with a manifest, then issues a COPY command pointing at them, which is Redshift's own recommended path for bulk loading rather than something Airbyte invented. There is no direct insert alternative.
On this source that matters more than usual, because observability volumes are large and every row passes through the bucket. Put it in the same region as your cluster to avoid cross-region transfer charges, which on a log pipeline is a real number rather than a rounding error.
It also affects your network design, since an SSH tunnel covers the SQL connection to the cluster and not the staging traffic. If your security model assumes all traffic passes through a bastion, that assumption needs revisiting before somebody discovers it in a review.
Frequently asked questions
Why do I need two Datadog keys?
They do different jobs. The API key identifies your organisation, while the application key is tied to the user who created it and carries their permissions.
My keys are correct but authentication fails. What now?
Check the site setting, since Datadog runs regional sites and the wrong one produces an error that looks exactly like bad credentials.
Do I need an S3 bucket to load into Redshift?
Yes. The destination stages data in S3 and loads it with COPY, which is Redshift's recommended approach. Put the bucket in the same region as your cluster.
How do I keep the volume manageable?
Through the query. Start from the question you want answered in three years and retrieve only that, because an unfiltered log query grows every day the pipeline runs.
Can I do this without writing code?
Yes. The setup above is UI-driven, though the Datadog query itself is the part worth thinking hardest about.
Get your Datadog data into Amazon Redshift
Write the query first and scope it to the question you want answered years from now, because nothing else limits what this pipeline moves. Set up the staging bucket in the right region, check the site before blaming the keys, and sync comfortably inside Datadog's retention.
Airbyte's connector catalog includes 600+ pre-built connectors, so operational data can sit beside the delivery and commercial sources that explain it. For the same source in a warehouse without staging, see Datadog to Snowflake, and for error tracking alongside it, Sentry to BigQuery.
Integrate with 700+ apps using Airbyte
Move data from 700+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.