Microsoft Dataverse to Databricks: How to Move Your Data
Move Microsoft Dataverse data into Databricks with Airbyte. Change tracking, Unity Catalog Volumes permissions, and how Dataverse types land in Delta tables.
Moving Microsoft Dataverse to Databricks brings your Dynamics 365 and Power Platform records into a governed lakehouse, where Unity Catalog handles access control and lineage across CRM data and everything else at once. For organisations already standardised on Databricks, it puts business application data under the same governance model as the rest of the estate.
Both ends have a prerequisite that fails quietly rather than loudly. Working through them before you build saves a confusing afternoon.
What to check before you build
Four checks, two on each side. Each one produces a failure that does not obviously point at its cause.
Why change tracking is the important one
The Dataverse source discovers every table in your environment automatically through the EntityDefinitions endpoint, so each one appears as a stream without configuration. That convenience hides the problem.
Incremental sync relies on Dataverse's change tracking feature. The connector sends a track-changes request header and Dataverse returns a delta link on the last page of results, which Airbyte saves as state and reuses next time to fetch only what changed. Where change tracking is not enabled on a table, that mechanism is simply unavailable and the stream falls back to full refresh.
It is off by default and set table by table in Power Apps. Combined with automatic discovery, that produces a specific failure mode: someone selects thirty tables, sets an hourly schedule, and re-reads most of the environment every hour without realising. Because Dataverse enforces service protection limits and answers with HTTP 429 when they are exceeded, the punishment for this is throttling rather than just wasted compute.
Enable tracking on every table you want incrementally, and give any table that cannot have it a deliberately slower schedule.
Setting up the Databricks destination
Unity Catalog is required. You also need a SQL warehouse or all-purpose cluster, and from its Connection Details tab you take the Server Hostname, HTTP Path, and Port. The catalog name is the top-level name in the workspace sidebar.
Grant the connector permission to create schemas, tables, and Unity Catalog Volumes. Airbyte stages Avro files in Volumes before loading them into Delta tables, so a service principal that can write tables but not create Volumes fails with an error that looks nothing like the actual problem. OAuth2 with a service principal is recommended; a personal access token also works.
Create the Databricks Lakehouse destination, enter the connection details and catalog name, accept the JDBC driver terms, and set a default schema. From version 4.0.0 the destination uses Direct Load, writing each stream straight to a final Delta table with no raw tables. Upgrading from 3.x is a breaking change covered in the migration guide.
How Dataverse data lands in Delta tables
Numbers become DECIMAL(38, 10), which allows 28 integer digits and 10 fractional. That comfortably covers Dataverse Money fields, so unlike some warehouse destinations there is no precision setting to get wrong here.
Three mappings are worth knowing. DateTime fields arrive as timestamps with timezone preserved. Status and State fields arrive as integer option set codes rather than the labels users see in Dynamics, so you will want a lookup to make them readable. And virtual fields are excluded from the schema entirely and never synced, so any calculated column you depend on has to be reproduced in Databricks.
Schema and table names are lowercased automatically, though column names keep their source casing. Check what was created before writing queries against the name you expected.
On deletions, the connector replicates incremental deletes, but not every entity supports it and deleted records carry only the ID. Databricks learns a record disappeared without learning what it held. If the contents of deleted opportunities or cases matter to your reporting, build a history table downstream rather than relying on the delete event.
Frequently asked questions
Why is every sync a full refresh?
Change tracking is disabled on those tables. It is a per-table setting in Power Apps, off by default, and without it incremental sync is not available to the connector.
My table permissions are correct but the sync still fails. Why?
Check whether the principal can create Unity Catalog Volumes. Airbyte stages Avro files there before writing any Delta table, so table-level permissions alone are not enough.
Do I need to worry about Money field precision?
Less than with some destinations. Databricks maps numbers to DECIMAL(38, 10), an exact type with ten fractional digits, which covers financial values from Dataverse without extra configuration.
Why are my status fields showing numbers?
Status and State arrive as integer option set codes rather than display labels. Maintain a lookup table in Databricks to translate them, since the labels live in Dataverse metadata rather than the records themselves.
What causes HTTP 429 errors?
Dataverse service protection limits. Enable change tracking so you pull deltas instead of whole tables, reduce sync frequency, and deselect tables you do not need.
Get your Dataverse data into Databricks
Work the four checks before you build, and this is a short setup with good governance on the other side. Consolidating onto a cloud warehouse instead? See our guide to Microsoft Dataverse to Snowflake. Moving files into the same lakehouse? See SFTP to Databricks.
Integrate with 600+ apps using Airbyte
Move data from 600+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.