Microsoft Dataverse to Amazon S3 with AWS Glue: How to Move Your Data

Moving Microsoft Dataverse into Amazon S3 with AWS Glue takes business application data out of the Power Platform and puts it in an open table format your whole AWS estate can query. Athena, Glue jobs, EMR and Redshift Spectrum all read the same tables, without a licence tied to each reader.

Two things are worth establishing first. Airbyte's route here is the S3 Data Lake connector, which writes Iceberg tables registered in a catalog such as Glue. And the Dataverse source offers genuine change tracking rather than a cursor, which is unusual enough to shape how you design the pipeline.

Microsoft Dataverse to Amazon S3 with AWS Glue at a glance:

Why move data from Dataverse to S3 with AWS Glue?

Two situations account for most of these pipelines.

The first is getting business data out of a licensed platform. Dataverse underpins Dynamics 365 and the Power Platform, and access is tied to licensing in ways that make it awkward to give every analyst a seat. A lake copy removes that constraint entirely.

The second is a cross-cloud reality. Plenty of organisations run Microsoft business applications and an AWS analytical estate, and this pipeline is the bridge. Landing in Iceberg rather than a proprietary warehouse also keeps the data readable by whatever engine you use next.

What do you need before you start?

The Microsoft side has a two-part setup where the second half is commonly missed:

An application registered in Entra ID. With a client identifier and secret, and permissions for the Dataverse API. This is the half most people complete, and on its own it is not enough.

A matching application user inside Dataverse. Created in the Power Platform admin centre and assigned a security role. Without it your registration authenticates successfully and then cannot see anything, which produces a confusing failure that looks like a permissions bug rather than a missing step.

An S3 bucket, a bucket policy and a Glue database. The S3 Data Lake destination documentation includes the policy to paste in with your bucket name substituted. AWS Glue is one of the supported Iceberg catalogs, alongside REST, Nessie and Polaris.

A list of the entities you need, checked for change tracking. Not every Dataverse entity supports it, and the ones that do not will behave differently. Establish which is which before designing around incremental sync.

Finally, look at your primary key types, because of how the destination handles numeric keys in Iceberg. That detail is covered below and it is the one most likely to produce a surprising result.

How do you build a Dataverse to S3 pipeline in Airbyte?

Step 1: Complete both halves of the Microsoft setup

Register the application in Entra ID, then go to the Power Platform admin centre and create the corresponding application user with a security role. Doing both before touching Airbyte saves you diagnosing an authentication that succeeds and returns nothing, which is the single most common way this connector wastes an afternoon.

Step 2: Configure the Dataverse source

Click Sources in the left navigation, then New Source, and select Microsoft Dataverse, following adding a source. Supply your environment URL, tenant identifier, client identifier and secret. The environment URL is specific to your Dataverse instance rather than a shared endpoint.

Step 3: Configure the S3 Data Lake destination

Click Destinations, then New Destination, and select S3 Data Lake, following adding a destination. Supply your bucket, region and credentials, then select AWS Glue as the catalog and give it the database name. The connector writes Iceberg tables and registers them in Glue, which is what makes them queryable from Athena without further work.

Step 4: Create the connection and verify from Athena

Click Connections, then New connection, select your entities and a sync mode. Where change tracking is available, incremental gives you updates and deletions. Where it is not, you are on full refresh for that entity, which is worth knowing per entity rather than assuming uniformly.

After the first sync, query a table from Athena to confirm the Glue registration worked end to end, and check the registered names while you are there.

What does genuine change tracking give you?

Deletions, which most API sources cannot offer. Dataverse issues a delta link that the connector stores and presents on the next sync, and the API returns what changed since that point rather than everything matching a timestamp filter. That includes records that were removed.

Delete events carry the record identifier and nothing else, which is enough to act on and not enough to reconstruct what the record contained. So decide what a deletion should mean in your lake: removing the row, or flagging it and keeping the history, which on a compliance-minded dataset is often the better answer.

The caveat is coverage. Not all entities support change tracking, so an entity that does not gets full refresh instead, with no delete detection and a heavier read each time. Check per entity rather than assuming the whole environment behaves the same way, because a mixed design is normal here.

Why do numeric primary keys sort strangely?

This comes from Iceberg rather than from Airbyte. When a primary key field has the Airbyte Number type, the connector stores it as an Iceberg string rather than a double, because Iceberg identifier fields do not support the double type and deduplication correctness depends on the key being a valid identifier column.

The trade-off is that ordering and comparison on those columns become lexicographic rather than numeric, so the value 9 sorts after 10. Deduplication stays correct, which is what the choice protects, and any query that sorts or filters on the key needs to account for it.

Dataverse leans on GUIDs for its primary keys, which sidesteps this for the main record identifiers. It is still worth checking any entity where an auto-numbered field is doing the work, and casting explicitly in downstream views where the key is used analytically rather than only for deduplication.

Frequently asked questions

My credentials authenticate but no data comes back. Why?

You have probably completed the Entra ID registration without creating the matching application user inside Dataverse and assigning it a security role. Both halves are required.

Are deletions captured?

On entities that support change tracking, yes. Delete events carry the record identifier only, so decide whether your lake removes the row or flags it.

Why can I not sync one entity incrementally?

Not every Dataverse entity supports change tracking. Those that do not fall back to full refresh, so check per entity rather than assuming the environment is uniform.

Why did my table names change?

When using AWS Glue, the connector replaces non-alphanumeric characters in table names and namespaces with underscores for Athena compatibility. Check the registered names after the first sync.

Can I do this without writing code?

The Airbyte setup is UI-driven, and the Microsoft and AWS sides are portal configuration plus a bucket policy the documentation provides.

Get your Dataverse data into S3

Complete both halves of the Microsoft setup before anything else, check change tracking support per entity rather than assuming, decide what a deletion should mean in your lake, and verify the registered table names from Athena after the first sync.

Airbyte's connector catalog includes 600+ pre-built connectors, so business application data can sit beside your other sources in one lake. For the same source feeding a streaming platform, see Microsoft Dataverse to Kafka, and for a relational source landing in the same lake, PostgreSQL to Amazon S3 with AWS Glue.

Start syncing now →