Launchdarkly to Databricks: How to Move Your Data

Moving LaunchDarkly into Databricks gives you a record of how your feature management estate changes. LaunchDarkly shows the current state of every flag, which is exactly right for operating it and useless for asking how many flags have been on for two years or which ones nobody ever cleaned up.

This guide covers the managed path with Airbyte. Two things shape the build: every stream is full refresh only, and because the source describes the present rather than the past, the history you want is something the destination accumulates for you.

Launchdarkly to Databricks at a glance:

Why move data from Launchdarkly to Databricks?

Two situations account for most of these pipelines.

The first is flag hygiene, which every organisation using feature management eventually needs. Flags accumulate, temporary ones become permanent, and nobody can say which are safe to remove. A history of flag state makes that answerable rather than a matter of asking around.

The second is joining release activity to outcomes, setting flag changes against incidents, performance or product metrics. If your aim is simply a current inventory rather than a history, a lakehouse is more machinery than the job needs and a small warehouse table would do, since the value here comes from accumulating snapshots over time.

What do you need before you start?

Four things, and the second decides how heavy this pipeline becomes:

An access token. A personal or service token from your LaunchDarkly authorisation settings, and a Reader role covers everything this connector reads. A service token is preferable, since a personal one leaves with its owner. The LaunchDarkly source documentation lists the streams.

A decision about the audit log. It is the one stream that grows without bound, and since every stream is full refresh only, keeping it means re-reading the entire log on every sync. That is the difference between a light pipeline and a heavy one.

Permission to create Volumes in Unity Catalog. Staging happens through Avro files written into a Volume, which is a separate permission from creating tables and worth requesting early.

A view on how long you want history. Since the source only ever describes the present, your tables become the only record of how things looked before, which makes retention a decision rather than a default.

If your workspace restricts traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.

How do you build a Launchdarkly to Databricks pipeline in Airbyte?

Step 1: Decide whether you need the audit log

Ask what questions the audit log answers that the other streams do not, because it is the stream with a cost attached. Who toggled a flag and when is genuinely useful for incident review, and if nobody will use it, leaving it out makes this a trivial pipeline instead of a demanding one. The other five streams describe a configuration estate and are small by any measure.

Step 2: Configure the LaunchDarkly source

Click Sources in the left navigation, then New Source, and select LaunchDarkly, following adding a source. Supply the access token, which is the whole configuration, then select from projects, environments, flags, metrics, members and the audit log. Environments, metrics and flags are fetched per project, so a large estate multiplies the requests involved.

Step 3: Configure the Databricks destination

Click Destinations, then New Destination, and select Databricks, following adding a destination. Supply the workspace details, warehouse or cluster, catalogue and schema. Flag definitions carry nested targeting rules and variations, and Spark reads those natively, so nothing needs flattening on the way in.

Step 4: Create the connection and choose append over overwrite

Click Connections, then New connection, select your streams and a sync mode. Only full refresh is available, and the variant matters: overwrite gives you a current inventory, while append accumulates a snapshot per sync and is what turns this into the history you actually wanted.

Daily suits a configuration estate, and a gentler schedule also keeps you clear of rate limits that respond to excess with a 429 rather than a queue.

Why does the audit log make this expensive?

Because every stream is full refresh only, and the audit log is the one that never stops growing. The other five describe a configuration estate whose size is bounded by how many projects and flags you have. The audit log records every change anybody has ever made, and re-reading all of it on every sync is a cost that increases for as long as people keep using LaunchDarkly.

Rate limits make that sharper. They apply per key per second and exceeding them returns an error rather than throttling politely, so a heavy sync against a large estate is more likely to run into them than a light one. Fetching environments, metrics and flags per project compounds this, since each project is its own set of requests.

The sensible response is proportion. Sync the configuration streams frequently, because they are small and their current state is what most questions need. If you want the audit log, give it its own connection on a slower schedule, so a daily flag inventory does not drag a complete audit history behind it every morning. Separate connections cost a little configuration and save a great deal of unnecessary work.

How do you turn current state into history?

By appending, because the source will never give you the past. LaunchDarkly tells you what a flag looks like now, so questions about how long something has been enabled or when a targeting rule changed can only be answered from records you kept yourself. Full refresh overwrite discards exactly that, leaving a table that is permanently current and permanently without memory.

Full refresh append builds the series instead, adding a complete snapshot of your flags on every sync. Run it daily and you accumulate daily observations of every flag's state, which is precisely the raw material for the flag hygiene question. The absence of a volume ceiling here is what makes that affordable, since a snapshot per day of a large estate would be uncomfortable in a destination that charged by the row.

Then build the silver layer that makes it legible. A table deriving when each flag first appeared, when its state last changed and how long it has been in that state turns thousands of snapshots into the short list somebody can act on. Keep the raw snapshots in bronze, because a new question about flag history is then a query rather than a request to start collecting something.

Frequently asked questions

Why is there no incremental sync?

The connector supports full refresh only, so each sync reads everything from scratch. Choosing between overwrite and append decides whether you keep a current inventory or accumulate history.

Can I see how long a flag has been enabled?

Only from snapshots you have accumulated, since the source describes current state. Use append and derive the duration in a silver table.

My syncs are returning 429 errors.

Rate limits apply per key per second. Reduce frequency, split the audit log onto its own slower connection, and remember that per-project streams multiply requests across a large estate.

What role does the token need?

A Reader role grants read access to everything this connector syncs. Prefer a service token over a personal one so the pipeline survives staff changes.

Can I do this without writing code?

The pipeline, yes. Deriving flag lifetimes from accumulated snapshots is modelling work, and it is the part that answers the question people actually asked.

Get your Launchdarkly data into Databricks

Decide about the audit log first, since it is the only stream that grows without limit and full refresh means re-reading all of it every time. Give it a separate, slower connection if you want it. Choose append rather than overwrite, because the source only describes the present and your snapshots are the only history that will ever exist. Then derive flag lifetimes in a silver table, which is the question that prompted this in the first place.

Airbyte's connector catalog includes 600+ pre-built connectors, so release tooling can be analysed alongside the systems it changes. For issue tracking into the same destination, see Jira to Databricks, and for another full-refresh operational source into the same destination, n8n to Databricks.

Start syncing now →