Microsoft Dataverse to Kafka: How to Move Your Data

Streaming Microsoft Dataverse to Kafka is the most technically interesting pairing in this series, because Dataverse is one of the few SaaS sources with genuine change tracking rather than a timestamp cursor. Real change events, including deletes, arriving on a bus that several systems can subscribe to.

That makes the destination's behaviour matter more here than it does on a cursor-based source. This guide covers the managed path with Airbyte, and it spends most of its time on the two places where the strength of the source meets a limit in the pipeline: what a delete event actually contains, and whether your consumers see events in the right order.

Microsoft Dataverse to Kafka at a glance:

Why move data from Microsoft Dataverse to Kafka?

Two situations account for most of these pipelines.

The first is fan-out. Dataverse sits underneath Dynamics 365 and Power Apps, which means it holds records several systems want to react to: a provisioning service, an entitlement check, a billing trigger, a search index. Giving each its own Dataverse integration means several app registrations and several things to fix when Microsoft changes something.

The second is decoupling with replay. A consumer that breaks can be fixed and re-run from an offset rather than losing whatever arrived while it was down. Combined with genuine change tracking on the source, that gets you closer to a real change stream than most SaaS sources allow.

What do you need before you start?

The Microsoft side has a two-part permissions model that catches people out:

An app registration in Microsoft Entra ID. With a client secret. Note the application and directory identifiers from the overview page, since you need both. Authentication uses the client credentials flow rather than a user login, which is what you want for a pipeline.

That app added as an application user in Dataverse. This is the step people miss. Creating the registration in Entra ID is not enough: it must also be added as an application user in your Dataverse environment with at least read access to the tables you want to sync. Without that, you have valid credentials and no visible data. The Dataverse source documentation covers both halves.

A check on which entities support change tracking. Not all of them do. For those that do not, you fall back to full refresh, which changes what arrives in your topic and how often it makes sense to sync.

Topics created in advance. The destination fails on an unknown topic unless the broker creates topics automatically, which is discouraged in production where partition counts and retention should be chosen deliberately.

If your Kafka cluster restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.

How do you build a Dataverse to Kafka pipeline in Airbyte?

Step 1: Register the app and grant it Dataverse access

In the Azure portal, go to Microsoft Entra ID, then App registrations, then New registration. Name it, register it, and note the application and directory identifiers along with a client secret. Then go into your Dataverse environment and add that app as an application user with read access to the tables you need. Both halves are required, and the second is the one that gets forgotten.

Step 2: Configure the Dataverse source

Click Sources in the left navigation, then New Source, and select Microsoft Dataverse, following adding a source. Supply your environment URL, tenant, client identifier and secret. The connector discovers the schema automatically from your environment's entity definitions, so the catalogue reflects your customisations rather than a fixed list.

Step 3: Configure the Kafka destination

Click Destinations, then New Destination, and select Kafka, following adding a destination. Supply the bootstrap servers, security protocol and credentials, and a topic pattern. One topic per entity is usually what consumers want, since a provisioning service reacting to accounts has no interest in every other table in Dataverse.

Step 4: Create the connection with incremental sync

Click Connections, then New connection, select your entities and an incremental sync mode so change tracking is used rather than a full re-read. On the first sync the connector reads everything and stores a delta link returned by the API, then uses that on subsequent syncs to retrieve only what changed.

Check the topic names after the first sync, because a naming transformation replaces certain special characters and Dataverse entity names do not always survive unchanged.

How does Dataverse change tracking actually work?

The connector asks the Dataverse Web API to track changes using a dedicated request header. The API returns a delta link on the last page of results, which the connector stores as its state and sends on the next sync to retrieve only what has changed since.

This is meaningfully better than a timestamp cursor. A cursor cannot see deletions and misses updates that leave the cursor column untouched. Change tracking sees both, because the platform is telling you what changed rather than you inferring it from a column. If you have worked with Salesforce or another cursor-based CRM source, this is the capability you have been missing.

The limitation is coverage. Not all entities support change tracking, and for those that do not you are on full refresh, re-reading everything each sync. Work out which of your entities fall on which side before you design your topics and schedules, because the two behave very differently in a streaming context.

What does a delete event actually contain?

Only the identifier. This is documented plainly and it is the detail most likely to surprise a consumer author: deleted data have the ID and nothing else. You learn that a record went away and you learn which one, and you do not learn what it contained.

For a consumer maintaining its own copy of the data, that is entirely sufficient, because it already holds the record and can remove it by key. For a consumer that reacts to content, an alert that includes the customer name for instance, it is not, because the name is not in the delete event. Those consumers need to look the record up from their own state before it is gone.

Which leads to the interaction that matters most on this pipeline. Because the Kafka destination keys each message with an Airbyte-generated UUID rather than the record identifier, change events for the same record land in different partitions and can be consumed out of order. On a cursor-based source that is a nuisance. Here, with real deletes in the stream, an update consumed after a delete resurrects a record that no longer exists. If any consumer maintains state, repartition on the record identifier before it sees anything.

Frequently asked questions

Does the Dataverse connector capture deletes?

Yes, through Dataverse change tracking. Note that deleted records carry only the identifier, so consumers that need the record's contents must hold their own copy.

I have valid credentials but no data. What is wrong?

The app registration almost certainly has not been added as an application user in your Dataverse environment with read access to the tables. Creating it in Entra ID is only half the setup.

Why is one of my entities syncing in full every time?

Not all Dataverse entities support change tracking. Those that do not fall back to full refresh, so check which of your entities are affected before designing schedules around them.

Are change events ordered correctly in Kafka?

Not per record. Messages are keyed by an Airbyte UUID rather than the record identifier, so an update can be consumed after a delete. Repartition on the record identifier if any consumer maintains state.

Can I do this without writing code?

The Airbyte setup is entirely UI-driven, and the Microsoft side is portal configuration. If you need per-record ordering, the repartitioning step is code in whatever stream processing you already run.

Get your Dataverse data into Kafka

Complete both halves of the Microsoft permissions setup, check which entities support change tracking before designing topics, and tell your consumers two things: a delete event carries only the identifier, and per-record ordering needs a repartition. This source gives you a better change stream than most, and the value of that depends on handling it properly downstream.

Airbyte's connector catalog includes 600+ pre-built connectors, so the same Dataverse source can feed a warehouse alongside the topic. For that pairing, see Microsoft Dataverse to BigQuery, and for a cursor-based CRM feeding the same destination, Salesforce to Kafka.

Start syncing now →