Hubspot to Amazon S3 with AWS Glue: How to Move Your Data
Move HubSpot into S3 with AWS Glue using Airbyte. Why search-backed streams miss records, why associations must be defined, and what upserts cost in Iceberg.
Moving HubSpot into Amazon S3 with AWS Glue puts CRM history into an open table format several engines can read. HubSpot answers questions about the current state of a pipeline well and is a poor place to keep years of it, which is exactly what anybody analysing conversion or retention needs.
This guide covers the managed path with Airbyte. Two things shape the build: several important streams are search-backed and can miss records without a lookback, and CRM records change constantly, which is the pattern Iceberg handles least cheaply.
Hubspot to Amazon S3 with AWS Glue at a glance:
Why move data from Hubspot to Amazon S3 with AWS Glue?
Two situations account for most of these pipelines.
The first is keeping commercial history somewhere cheap and open. Years of deals, contacts and engagements are valuable for analysis and expensive to keep queryable in a warehouse, and Iceberg tables on S3 hold them at storage prices while staying readable by Athena, Spark and anything else in your estate.
The second is feeding data science rather than dashboards, where a lake suits the work better than a warehouse does. If what you actually want is fast interactive reporting with no table maintenance to think about, that is an argument for a warehouse instead, and Hubspot to BigQuery asks considerably less of you operationally.
What do you need before you start?
Four things, and the third is a decision most people make by accident:
HubSpot credentials with the right scopes. A private app token or OAuth, with read scopes covering every object you intend to sync. Association streams need scopes on both objects involved, which is easy to overlook. The HubSpot source documentation lists what each stream requires.
An S3 bucket and a Glue catalog. The pipeline writes data files to the bucket and registers tables in Glue, so the credentials need both. Keep namespace and table names alphanumeric with underscores, since Glue rewrites anything else.
A list of the associations you need. Associations are not automatic. You define them by naming the object at each end, and without them you have contacts and deals with no way to connect the two, which is most of what a CRM dataset is for.
Awareness that the daily limit is account-wide. It is shared with every other integration touching your HubSpot account, so a marketing tool having a busy day is capable of affecting your sync. The burst limit is per app and less likely to trouble you.
If your network restricts traffic by IP, add the Airbyte Cloud IP addresses to the relevant allow list before you begin.
How do you build a Hubspot to Amazon S3 with AWS Glue pipeline in Airbyte?
Step 1: Define the associations you actually need
Work out which relationships your analysis depends on, such as contacts to companies or deals to contacts, and define an association stream for each. They sync incrementally and are cheap to carry, so the cost of including one you end up not using is small compared with discovering months later that you cannot join two objects. Check the read scopes cover both ends of every pair you define.
Step 2: Configure the HubSpot source
Click Sources in the left navigation, then New Source, and select HubSpot, following adding a source. Supply your credentials and start date, add your association streams, and set a lookback window. That last setting is not optional decoration, for reasons the next section explains.
Step 3: Configure the S3 Data Lake destination
Click Destinations, then New Destination, and select the S3 Data Lake, following adding a destination. Choose AWS Glue as the catalog and supply the bucket, region and credentials. Name your namespace and tables in plain letters, digits and underscores, because Glue converts anything else into underscores and a renamed table is one nobody will guess.
Step 4: Create the connection and choose the interval carefully
Click Connections, then New connection, select your streams and a sync mode. Resist the temptation to sync every fifteen minutes. CRM records update constantly, every update is an upsert, and in Iceberg an upsert is more expensive than it looks, which makes frequency a design decision rather than a preference.
Schedule compaction and snapshot expiry from the start. Both are yours to arrange, and on this particular pairing they matter more than on most.
Why do your most important streams miss records?
Because deals, companies, contacts, tickets, leads and engagements are served by HubSpot's search endpoint, and search-backed streams can miss records during incremental syncs. That list is not a collection of minor streams; it is essentially the whole CRM, which makes this the single most important thing to know about the connector.
The remedy is the lookback window, which re-reads a period you have already covered so anything missed the first time is picked up on a later pass. It costs a little duplication, which deduplication handles, and it buys you completeness. Leaving it unset is the default path to a dataset that is quietly short of records nobody can identify.
The related choice is between server-side and client-side incremental. Server-side asks HubSpot for changes. Client-side pulls everything from the API and filters afterwards, which produces correct data and saves nothing at all against the account-wide daily limit. If your syncs are consuming more of that limit than expected, this is usually why, and it becomes visible only when another integration starts failing.
Why does a frequently updated CRM cost more in Iceberg?
Because Iceberg has no update in place. An upsert is an equality delete followed by an insert, so changing one field on a contact writes a delete file and a new row rather than modifying anything. That is a reasonable design for a format built on immutable files, and it means the cost of a record depends on how often it changes rather than on how large it is.
CRM data is close to the worst case. Contacts acquire properties, deals move stages, companies get enriched, and a sales team working normally produces a steady stream of small updates to existing records. Sync that every fifteen minutes and delete files accumulate quickly, which gradually slows reads because every query has to reconcile them before returning anything.
So pick the interval from how fresh the data genuinely needs to be, which for commercial analysis is usually daily rather than hourly, and schedule compaction to fold those delete files back into the data. A lake nobody maintains degrades quietly, and this is the pairing where that shows up soonest, so budget for the maintenance jobs at the same time as the pipeline rather than after somebody complains about query times.
Frequently asked questions
Why are some deals missing from my table?
Deals are a search-backed stream and those can miss records on incremental syncs. Set a lookback window so later passes pick up anything the first one skipped.
Can I join contacts to companies without association streams?
No. Associations must be defined explicitly by naming the objects at each end, and they need read scopes on both. They sync incrementally and cost little.
Does client-side incremental reduce API usage?
No. It still pulls everything from the API and filters afterwards, so the data is correct and your account-wide daily limit is consumed as though you had synced in full.
Why are reads getting slower over time?
Upserts write equality deletes, and frequently updated CRM records accumulate them quickly. Lengthen the sync interval and schedule compaction, since neither happens on its own.
Can I do this without writing code?
The pipeline, yes. Table maintenance is not, and on this pairing the compaction schedule is closer to a requirement than an optimisation.
Get your Hubspot data into Amazon S3 with AWS Glue
Define your association streams first, because without them you have objects and no relationships. Set a lookback window, since the streams that can miss records are essentially your whole CRM. Prefer server-side incremental, as the client-side variant spends your account-wide daily limit for no saving. Then choose a sync interval that respects how upserts work in Iceberg, and schedule compaction alongside the pipeline rather than after query times degrade.
Airbyte's connector catalog includes 600+ pre-built connectors, so commercial history can live in open formats rather than expensive storage. For the same source into a column store, see Hubspot to ClickHouse, and for another business platform into the same destination, Microsoft Dataverse to Amazon S3 with AWS Glue.
Integrate with 700+ apps using Airbyte
Move data from 700+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.