Salesforce to Kafka: How to Stream Your Data

Getting Salesforce onto Kafka is usually attempted by having each consuming team call the Salesforce API themselves. That means several sets of credentials, several integrations to maintain, and all of them drawing from the same daily API budget until one of them exhausts it and breaks the others. Platform Events are the other route, and they cover only the objects and changes somebody remembered to configure.

This guide covers the managed path: replicating Salesforce objects into Kafka topics with Airbyte, so one pipeline feeds every consumer. It also covers a behaviour that is sensible against a warehouse and genuinely awkward against an append-only log, which is what happens when a sync stops early.

Salesforce to Kafka at a glance:

Why move data from Salesforce to Kafka?

Two situations account for most of these pipelines.

The first is shared consumption. Sales operations wants opportunity changes, finance wants closed deals, the product team wants account attributes, and support wants case activity. One pipeline into topics they each subscribe to replaces four integrations, four credential sets and four claims on the same daily API budget.

The second is decoupling. Systems that react to CRM changes, a provisioning service, an entitlement check, a billing trigger, should not each hold Salesforce credentials and should not fail when Salesforce is slow. A topic in between gives them a buffer and lets them replay from an offset after a bug rather than losing whatever arrived while they were down.

What do you need before you start?

Two of these are configuration and two are agreements with other people:

Salesforce credentials with the right visibility. The connector sees what the authenticating user sees, so field-level security and sharing rules on that account determine what reaches your topics. Use a dedicated integration user with deliberately scoped access rather than an administrator's login.

A share of the daily API budget. Salesforce enforces a daily request limit on the org, and everything connected to it draws from the same pool. Agree your share with whoever owns the org before choosing a sync frequency.

Idempotent consumers. This one is unusual as a prerequisite and it belongs here rather than as an afterthought, for reasons covered below. Deduplicating on the Salesforce record ID and its last-modified timestamp is the usual approach.

Topics created in advance. The destination fails on an unknown topic unless the broker creates topics automatically, which is discouraged in production. Salesforce orgs gain custom objects over time, so agree a process for adding a topic when someone adds an object.

If your Kafka cluster restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.

How do you build a Salesforce to Kafka pipeline in Airbyte?

Step 1: Prepare the integration user and topics

Create the Salesforce integration user and confirm it can see the objects and fields you need, since anything hidden by sharing rules will be quietly absent rather than reported as an error. Then create your topics on the broker with partition counts and retention chosen for the volume you expect.

Step 2: Configure the Salesforce source

Click Sources in the left navigation, then New Source, and select Salesforce, following adding a source. Authenticate, set a start date, and set the lookback window deliberately rather than leaving the ten minute default in place. Salesforce's API is eventually consistent, so a record modified shortly before a sync may not be visible to it, and the lookback re-reads a short window behind the last cursor position to catch those.

Step 3: Configure the Kafka destination

Click Destinations, then New Destination, and select Kafka, following adding a destination. Supply the bootstrap servers, security protocol and credentials, and a topic pattern. One topic per Salesforce object is usually what consumers want, since a team interested in opportunities rarely wants cases in the same feed.

Step 4: Create the connection with incremental sync

Click Connections, then New connection, pick your source and destination, and select the objects you need. Choose an incremental sync mode rather than full refresh, and treat that as mandatory rather than a preference on any org of size. The reason is the next section. Set a replication frequency that fits your share of the API budget, save, and run the first sync.

Check the topic names afterwards. A naming transformation replaces certain special characters, so custom object names ending in the usual suffix will not appear exactly as you wrote them.

Why does a successful sync sometimes publish less data?

When the connector reaches your org's daily API limit it does not fail. It stops where it is, reports the run as successful, and picks up from that position next time. That is the right behaviour, because failing the whole sync would discard work already done, and over a day or two the data catches up on its own.

The self-correction only works on incremental sync. Configure full refresh against a large org and every run truncates somewhere arbitrary and starts from the beginning again next time, never reaching the end. That is why the sync mode choice above is not really a choice.

Against Kafka there is a second implication that does not exist against a warehouse. Consumers see a continuous log with no sync boundaries, so a partial day is invisible to them. If anything downstream computes daily totals or triggers on completeness, it needs a signal from outside the topic, typically a check on the sync status in Airbyte. Alerting on a sudden drop in message volume is a crude but effective substitute.

What goes wrong silently?

The lookback window is the first thing. In a warehouse, deduplication on the way in makes overlapping re-reads free. Kafka has no such mechanism, so re-read records are simply published again and a wider lookback means more duplicate messages. That is not a reason to keep it narrow, since missing records are worse than duplicate ones, but it is why idempotent consumers are a prerequisite rather than a nicety.

Formula fields are the second and the more insidious. The connector syncs the output of a formula, not the formula itself, and editing a formula does not change any record's modification timestamp. Incremental sync therefore never re-reads those records, so the recalculated values are never published and consumers that built state from the stream keep the old numbers indefinitely with nothing to indicate they are wrong. The fix is to reset the stream and backfill, which republishes them, so make sure consumers can absorb a sudden replay of historical records without double-counting.

The third is a data quirk. Bulk API results arrive as CSV, which cannot distinguish an empty cell from the literal text in it, so values such as NA, N/A, NULL, None and NaN become null by default. If your org legitimately uses those as values, a country code or a status, enable the option to preserve them. It affects Bulk API streams only, since REST-synced streams keep them as they are.

Frequently asked questions

Why did my Salesforce sync succeed but publish fewer records?

It hit your org's daily API limit. The connector stops there, reports success, and resumes from that point next run. On incremental sync this is self-correcting.

Will widening the lookback window create duplicate messages?

Yes. Kafka has no deduplication on write, so re-read records are republished. Make consumers idempotent on the record ID and its last-modified timestamp first.

How do I get updated formula field values into the topic?

Reset the stream and backfill. Changing a formula does not update record timestamps, so incremental sync will never republish those records on its own.

Are updates to the same record ordered?

Not by default. Messages are keyed by an Airbyte UUID rather than the record ID, so repartition on the record ID if a consumer tracks state transitions in sequence.

Can I do this without writing code?

Yes. The setup above is entirely UI-driven, from creating the source through to scheduling the connection. If you would rather manage pipelines as infrastructure, the same source, destination and connection can be provisioned from configuration files instead.

Get your Salesforce data into Kafka

Use incremental sync so partial runs resume, make consumers idempotent before you touch the lookback window, and give someone outside the topic the job of noticing when a sync stopped short, because the topic itself will never tell you. Then write down for your Salesforce admins that changing a formula requires a stream reset.

Airbyte's connector catalog includes 600+ pre-built connectors, so the same Salesforce source can feed a warehouse alongside the topic without rebuilding anything. For the same destination fed by a database with real change capture, see PostgreSQL to Kafka, and for Microsoft's CRM stack and its change tracking approach, see Microsoft Dataverse to BigQuery.

Start syncing now →