Zendesk Support to Databricks: How to Move Your Data

Moving Zendesk Support into Databricks gives a large support operation somewhere to keep and model years of tickets. Zendesk answers questions about this week competently and becomes slow and awkward once somebody asks about eighteen months of comments, audits and satisfaction ratings across several brands.

This guide covers the managed path with Airbyte. Two things shape the build: the first sync is governed by a rate limit floor that no subscription raises, and ticket records nest deeply enough that the destination you choose decides how much flattening you have to do.

Zendesk Support to Databricks at a glance:

Why move data from Zendesk Support to Databricks?

Two situations account for most of these pipelines.

The first is scale. A busy instance produces vastly more comments and audit events than tickets, and years of that is more than an operational database is comfortable holding. A lakehouse has no practical ceiling, which makes it the right home for a support history nobody wants to start deleting.

The second is modelling, since support data genuinely benefits from transformation into cohorts, first response times and resolution paths that no reporting screen computes. If your instance is modest and the point is serving current ticket state to an internal tool, the lakehouse is heavier than the job requires and Zendesk Support to MySQL is a better fit.

What do you need before you start?

Four things, and the last one is a permission people rarely anticipate:

An admin account, or a note of what you are losing. Several streams require administrator rights and an account without them returns less rather than failing. Use an admin service account where you can. The Zendesk Support source documentation covers credentials and streams.

Your ticket count, roughly. Because incremental export endpoints are limited to ten requests per minute regardless of plan, the size of your instance is the length of your first sync. Knowing the number turns an unexplained wait into a planned one.

A catalogue and schema decided in advance. Each stream becomes a table in Unity Catalog, so where those live and who can read them is a governance decision. Support data carries customer names and message content, which argues for narrow grants.

Permission to create Volumes in Unity Catalog. Staging happens through Avro files written into a Volume, and that permission is separate from being able to create tables. Where Unity Catalog is administered centrally, request it early.

If your workspace restricts traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.

How do you build a Zendesk Support to Databricks pipeline in Airbyte?

Step 1: Estimate the backfill before you promise a date

Take your ticket count, remember that comments and audits multiply it considerably, and set that against ten requests per minute on the incremental export endpoints. The arithmetic is unglamorous and it is the difference between a stakeholder expecting data on Tuesday and a stakeholder discovering on Thursday that the first sync is still running. Nothing you configure and nothing you pay Zendesk changes that floor.

Step 2: Configure the Zendesk Support source

Click Sources in the left navigation, then New Source, and select Zendesk Support, following adding a source. Supply the subdomain, credentials and a start date. Check the discovered streams against what Zendesk documents, since a non-admin account removes streams quietly, and select the deleted tickets stream if you are on version 5.2.0 or later and want them.

Step 3: Configure the Databricks destination

Click Destinations, then New Destination, and select Databricks, following adding a destination. Supply the workspace details, warehouse or cluster, catalogue and schema. Ticket records nest heavily and Spark reads that natively, so resist any urge to flatten on the way in.

Step 4: Create the connection and let the first sync run

Click Connections, then New connection, select your streams and a sync mode. Use incremental wherever offered, since support history only grows. Then leave the backfill alone rather than restarting it because it looks stuck, because slow is what ten requests per minute looks like from the outside.

Once caught up, daily or hourly syncs sit comfortably within both of Zendesk's rate limit tiers and the pipeline stops being interesting, which is the goal.

Why is the first sync slow whatever you pay Zendesk?

Because there are two rate limit regimes and only one of them responds to your plan. The general limits scale with your subscription and concurrency helps there. The incremental export endpoints, which are what a backfill actually uses, sit at a floor of ten requests per minute for everybody.

That distinction catches out teams on premium plans, who reasonably expect to have bought their way past this sort of thing. The symptom is a first sync that runs for days on a large instance while the Airbyte interface reports everything as healthy, which it is. Restarting makes it worse rather than better, since progress is lost and the floor is unchanged.

So plan around it rather than against it. Start the pipeline earlier than the deadline that motivated it, set a start date that covers the history you genuinely need rather than everything ever recorded, and tell whoever is waiting that the backfill is a one-off cost. Steady state is comfortable; it is only the first pass that hurts.

How should Databricks hold a ticket?

Whole, in bronze, then shaped above it. A Zendesk ticket is not a flat row: it carries custom fields, a via object describing how it arrived, satisfaction ratings, tags and references to users and organisations. Spark handles that structure natively, so the landing table can hold exactly what the API returned with nothing discarded.

That fidelity is the real argument for this destination over a relational one. Elsewhere, nested fields are flattened into column names that grow long and awkward, and a custom field added next quarter becomes a schema change. Here the raw record sits beside the typed columns, so a field nobody modelled is still recoverable when somebody finally asks about it.

Build the silver layer for the questions support leaders actually ask, which means resolution times, first response, reopen rates and volume by channel, with custom fields given readable names. Label the bronze tables clearly as a landing layer and restrict them, both because somebody will otherwise build a dashboard on them and because ticket comments contain whatever customers chose to write, including things nobody intended to warehouse.

Frequently asked questions

We are on a premium Zendesk plan. Why is the backfill still slow?

The incremental export endpoints have a floor of ten requests per minute for every plan. Other limits scale with your subscription, but that one does not.

Why are some streams missing?

Almost always the authenticating account's role, since admin-required streams are skipped silently rather than raising an error. Compare the discovered streams against the documented list.

My ticket count dropped after an upgrade.

Version 5.2.0 moved deleted tickets into their own stream, so the tickets stream legitimately holds fewer records. Select the new stream to restore the total.

Should I flatten custom fields before loading?

No. Spark handles the nesting natively, so let it land intact and shape it in a silver table where the logic is visible and changeable.

Can I do this without writing code?

The pipeline, yes. The silver layer turning nested tickets into support metrics is real modelling work, and it is where the value of this arrangement appears.

Get your Zendesk Support data into Databricks

Estimate the backfill from your ticket count and ten requests per minute, then start earlier than the deadline that prompted this. Check the authenticating account's role, since a non-admin dataset arrives looking complete. Secure Volume permission early. Then let tickets land nested, because Spark reads them happily and a lakehouse has no volume ceiling, which is the whole reason to choose this destination for a large instance.

Airbyte's connector catalog includes 600+ pre-built connectors, so support history can be modelled beside the commercial data that explains it. For the same source into a warehouse, see Zendesk Support to BigQuery, and for another customer-facing system into the same destination, Zoho CRM to Databricks.

Start syncing now →