Datadog to Databricks: How to Move Your Data
Replicate Datadog to Databricks with Airbyte. Defining custom query streams, the two keys and site setting, Unity Catalog prerequisites, and modelling logs.
Replicating Datadog to Databricks moves observability data out of a monitoring tool with a short retention window and into a lakehouse where it can be kept indefinitely and joined against deployment records, incident tickets, and business metrics. That is the main reason teams build this pipeline: Datadog is excellent at showing you what is happening now and expensive for asking what happened last year. Airbyte handles the extraction with no code.
The distinctive thing about this connector is that you largely define your own streams. Rather than a fixed catalogue, it lets you supply queries, and each one becomes a stream.
Defining what you actually want to sync
The Datadog source takes an optional Queries setting where multiple queries produce multiple streams. Each entry has a name, a data source chosen from metrics, cloud cost, logs, and RUM, and the query string itself, written in Datadog's own query syntax.
This is the part to get right, because it determines everything about the volume and usefulness of what lands. Datadog holds an enormous amount of data and almost none of it is worth keeping forever. Define narrow queries answering questions you know you will ask, rather than syncing broadly and filtering later, since filtering later means paying to extract, store, and scan data you never wanted.
There are also top-level settings for a query, a start date, and an end date, which filter records collected from the Logs and AuditLogs streams, along with a limit controlling how many records are collected per request.
One reported behaviour is worth knowing before you debug it yourself. Users have found the connector attempting to query the audit logs endpoint even when only logs were configured, which fails for anyone whose Datadog account does not include audit log access. If your sync fails at the check step with a permissions error mentioning an endpoint you did not ask for, that is the pattern, and the workaround is to ensure your application key has access or to raise it with support rather than reconfiguring repeatedly.
On credentials, you need both an API key and an application key, which are different things in Datadog and are created in different places. You also set the site, since Datadog runs regional instances and pointing at the wrong one produces failures that look like bad credentials.
What Databricks requires
The Databricks destination requires a workspace with Unity Catalog enabled. This is a hard prerequisite and the most common reason a first attempt fails.
You also need a SQL warehouse or all-purpose compute cluster, and permission to create schemas, tables, and Unity Catalog Volumes in the target catalog. Volumes are load-bearing rather than incidental: the connector stages Avro files in one before loading them into tables, so without that permission the sync cannot stage data at all.
Authenticate with OAuth2 through a service principal, which is the recommended path, or with a personal access token. Collect the Server Hostname, HTTP Path, and Port from the SQL warehouse Connection Details tab, with 443 as the default, and take the Unity Catalog name from the workspace sidebar, remembering it is the top-level catalog rather than a schema. Accept the JDBC driver terms in the configuration.
Since version 4.0.0 the destination uses direct load, writing straight to final tables, and the underscore-prefixed raw tables are no longer produced. If you are upgrading from 3.x with models reading raw tables, rewrite them first.
Designing for observability data
Two properties of this data should shape your model. Log records are deeply nested, carrying attributes, tags, and host metadata as structured objects, and those arrive in Databricks as STRING columns holding serialised JSON. If your analysis depends on a particular tag or attribute, extract it into a typed column in a downstream model rather than asking every query to parse JSON.
And observability data is append-heavy by nature. A log line or a metric point does not get updated, which means incremental append is usually the right sync mode and the deduped modes are solving a problem you do not have. Since the tables grow indefinitely, partition and cluster them on the event timestamp in a downstream table so queries scanning a single week do not read years.
Frequently asked questions
How do I control what gets synced?
Through the Queries setting, where each query you define becomes a stream. Choose a data source from metrics, cloud cost, logs, or RUM, and write the query narrowly.
My sync fails on an endpoint I did not configure.
Users have reported the connector querying the audit logs endpoint even when only logs were configured, which fails without audit log access. Check that your application key has the necessary access.
Why do my credentials fail?
Check the site setting first, since Datadog is regional and a mismatch presents as an authentication error. Also confirm you have supplied both an API key and an application key.
Which sync mode suits log data?
Incremental append. Observability records are written once and never updated, so deduplication solves a problem this data does not have.
Get your Datadog data into Databricks
Define narrow queries rather than syncing broadly, get the site right before debugging credentials, enable Unity Catalog with Volume permissions, and extract the tags you query into typed columns. For the same source into a warehouse, see our guides to Datadog to BigQuery and Datadog to Snowflake.
Integrate with 600+ apps using Airbyte
Move data from 600+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.