HubSpot to Kafka: How to Stream Your Data

Getting HubSpot data onto Kafka usually starts with each team building their own integration. Marketing ops has one, the data team has another, someone wrote a script two years ago that still runs. They all work, and they all draw from the same daily API budget, which belongs to the HubSpot account rather than to any one of them. Eventually one exhausts it and breaks the others.

This guide covers the managed path: replicating HubSpot objects and associations into Kafka topics with Airbyte, so one pipeline feeds everyone. Most of the design work here is budgeting API calls rather than configuring Airbyte, and there is one distinction in how the connector syncs incrementally that determines whether that budget goes further or not at all.

HubSpot to Kafka at a glance:

Why move data from HubSpot to Kafka?

Two situations account for most of these pipelines.

The first is consolidating API usage. One pipeline reading HubSpot and publishing to topics replaces several integrations competing for a shared budget, and it makes that consumption visible in one place rather than spread across tools nobody has inventoried.

The second is reacting to changes. Systems that respond to CRM activity, a provisioning job, an onboarding trigger, a support routing rule, get a buffer between them and HubSpot, and can replay from an offset after a bug rather than losing whatever arrived while they were broken. Webhooks deliver once and assume you were listening; a topic does not.

What do you need before you start?

Two of these are configuration and two are conversations:

A Private App or OAuth application. With read scopes on every object you intend to sync. Association streams need read scopes on both objects in the relationship, and custom objects need the custom object read scope.

An agreed share of the daily API budget. The burst limit is per app, so a surge from your sync will not immediately throttle anyone else. The daily limit belongs to the account and every connected application draws from it. Find out what else talks to HubSpot before choosing a schedule.

Idempotent consumers. Deduplicating on the object ID and its last-modified timestamp. This belongs in the prerequisites rather than as a later caveat, because the lookback window discussed below is not safe to widen without it.

Topics created in advance. The destination fails on an unknown topic unless the broker creates topics automatically, which is discouraged in production. HubSpot accounts gain custom objects and new association pairs over time, so agree a process for adding topics.

If your Kafka cluster restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.

How do you build a HubSpot to Kafka pipeline in Airbyte?

Step 1: Create the app and prune your properties

Create the Private App with the read scopes you need, then take ten minutes to look at your custom properties. The more properties an object carries, the more data comes back on every record on every request, and HubSpot instances accumulate properties the way garages accumulate paint tins. Removing ones nobody queries is a genuine performance improvement rather than housekeeping, and it is easier now than after consumers depend on the payload shape.

Step 2: Configure the HubSpot source

Click Sources in the left navigation, then New Source, and select HubSpot, following adding a source. Authenticate, set a start date, and define the association streams you need by naming a from object and a to object for each. Then refresh the source schema, which is the step people forget when a stream they configured stubbornly fails to appear in the catalogue.

Step 3: Configure the Kafka destination

Click Destinations, then New Destination, and select Kafka, following adding a destination. Supply the bootstrap servers, security protocol and credentials, and a topic pattern. One topic per object and one per association stream is usually what consumers want, since someone reacting to ticket activity has no interest in marketing email events in the same feed.

Step 4: Create the connection and set the lookback

Click Connections, then New connection, pick your source and destination, and select streams and a sync mode for each. Set a replication frequency that fits your share of the API budget, and set a lookback window if you are syncing the search-backed streams, which include deals, companies, contacts, tickets, leads and the engagement streams. Those can miss records because of irregularities in HubSpot's API, and the lookback re-fetches a configurable number of minutes before the previous sync state to catch them.

Check the topic names after the first sync. A naming transformation replaces certain special characters, and generated association stream names in particular may not appear exactly as you expect.

Why is my incremental sync not saving any API calls?

Because incremental means two different things depending on the stream. Server-side incremental asks HubSpot for records changed since the last sync, and the API returns only those, so both the request volume and the transfer shrink. Client-side incremental asks for everything, receives everything, and discards what the connector has already seen.

Both produce correct data in your topic. Only the first reduces what you spend. Which one a stream uses is a property of that stream rather than a setting you choose, so the value of knowing this is diagnostic: when an incremental sync is unexpectedly slow or expensive, this is usually why, and the fix is dropping streams or reducing frequency rather than hunting for a configuration error that does not exist.

The lookback window interacts with this in a way that is specific to Kafka. In a warehouse, deduplication on the way in makes the overlap free. Kafka has no such mechanism, so re-read records are republished as duplicates and a wider lookback directly increases message volume. Missing records are worse than duplicate ones, so widen it when you need to, but only once consumers can absorb it.

How do you get the relationships between objects?

Usually the relationships are the point. Which deals belong to which company, which tickets belong to which contact. These do not fall out of the object streams on their own. You define association streams explicitly by picking a from object and a to object, optionally naming the stream, and the connector emits one record per relationship.

Two things make this pleasant. Association streams sync incrementally, fetching associations only for records modified since the last sync, so they are cheap against your rate limit. And they map naturally onto their own topics, which suits consumers that care about relationship changes without wanting the full object payloads alongside.

One thing to tell your consumers about the payload itself. Each message is keyed by a UUID Airbyte generates rather than the HubSpot object ID, so two updates to the same deal can land in different partitions and be consumed out of order. For most CRM consumers that is fine. For anything tracking deal stage transitions in sequence, repartition on the object ID first.

Frequently asked questions

Why is my incremental HubSpot sync still slow?

Some streams use client-side incremental, where the API returns everything and the connector filters locally. The data is correct but there is no reduction in API usage or transfer time.

Does my sync compete with other HubSpot integrations?

For the daily limit, yes. It is shared across every app on the account. The burst limit is per app, so bursts are isolated but daily budget is not.

How do I get deal and company relationships into Kafka?

Define an association stream with the from and to objects, grant read scopes on both, then refresh the source schema so the generated stream appears in the catalogue.

Records are missing from my topic. What should I change?

Set a lookback window to compensate for HubSpot search API irregularities, and make consumers idempotent first, because Kafka republishes the re-read records as duplicates.

Can I do this without writing code?

Yes. The setup above is entirely UI-driven, from creating the source through to scheduling the connection. If you would rather manage pipelines as infrastructure, the same source, destination and connection can be provisioned from configuration files instead.

Get your HubSpot data into Kafka

Agree your share of the account's daily API budget before choosing a schedule, define the association streams you need rather than hoping relationships appear, prune custom properties nobody queries, and make consumers idempotent before you touch the lookback window. Nearly every problem people report on this pipeline is a rate limit shared with somebody they had not thought about.

Airbyte's connector catalog includes 600+ pre-built connectors, so the same HubSpot source can feed a warehouse alongside the topic without rebuilding anything. For another workflow source feeding the same destination, see n8n to Kafka, and for Microsoft's CRM stack and its change tracking approach, Microsoft Dataverse to BigQuery.

Start syncing now →