Github to Databricks: How to Move Your Data

Moving GitHub into Databricks gives engineering leaders the delivery history their tooling never keeps in one place. Questions about review latency, how long pull requests wait, and whether a process change helped are all trend questions across many repositories, and GitHub answers none of them.

This guide covers the managed path with Airbyte. Two things shape the build: storage is not your constraint here, so the usual advice about trimming streams inverts, and pull request records nest deeply enough that the destination choice matters.

Github to Databricks at a glance:

Why move data from Github to Databricks?

Two situations account for most of these pipelines.

The first is measuring delivery properly. Cycle time, review latency and rework are derived metrics that need the raw events kept and modelled, not counted once and discarded. That is exactly the work a lakehouse suits, and it benefits from having everything rather than a summary somebody chose two years ago.

The second is joining engineering activity to incidents, releases or cost. If what you want is a straightforward count of merged pull requests per team per quarter, that is a warehouse report and Github to BigQuery gets you there with less to operate.

What do you need before you start?

Four things, and the second is the one that actually limits you:

A GitHub token and a repository list. Leaving the repository field blank gives you everything the token can see, which on an organisation account is usually far more than intended. The GitHub source documentation covers the options.

An understanding of the two rate limit budgets. REST counts requests and GraphQL calculates points based on what a query asks for, and this connector uses both. That, rather than storage, is what bounds how much you can collect.

Permission to create Volumes in Unity Catalog. Staging goes through Avro files written into a Volume, which is separate from creating tables and worth requesting early.

A list of the metrics you intend to derive. Review latency and cycle time are computed from timestamps across several streams, so knowing which metrics matter tells you which streams are genuinely load bearing.

If your workspace restricts traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.

How do you build a Github to Databricks pipeline in Airbyte?

Step 1: Work out which budget constrains you

Because the destination imposes no practical volume limit, the question is not what you can afford to store but what you can afford to fetch. Pull request stats, reviews, releases, projects and the reaction streams come through GraphQL and spend points; everything else spends requests. Knowing which of those two budgets your selection loads tells you whether an ambitious stream list is actually achievable.

Step 2: Configure the GitHub source

Click Sources in the left navigation, then New Source, and select GitHub, following adding a source. Supply the token, your repository list and a start date, then select streams. Name the repositories explicitly rather than leaving the field blank, since scope here is a fetching cost rather than a storage one.

Step 3: Configure the Databricks destination

Click Destinations, then New Destination, and select Databricks, following adding a destination. Supply the workspace details, warehouse or cluster, catalogue and schema. Pull request records carry nested users, labels and requested reviewers, and Spark reads those natively, so nothing needs flattening on the way in.

Step 4: Create the connection and keep the raw events

Click Connections, then New connection, select your streams and a sync mode. Use incremental where offered, since engineering history only grows. Daily is ample, because nobody needs a review indexed within minutes and a gentler schedule keeps both rate limit budgets comfortable.

Then build the silver layer, because the landing tables describe GitHub's objects rather than your delivery process.

What can you afford to keep here?

More than you would elsewhere, which inverts the usual advice about this source. In a search index or a warehouse, high-volume streams like reactions are the obvious thing to drop because they cost storage and nobody searches them. Here storage is effectively free, so the case for dropping them rests entirely on the API budget they consume.

That changes the calculation rather than reversing it. Reactions genuinely do tell you something about engagement, and a lakehouse can hold years of them without anybody minding. What they cost is points on the GraphQL budget, alongside reviews and pull request stats, which are the streams your delivery metrics actually depend on.

So spend the budget on what you will model. Reviews and pull request stats earn their points because review latency and cycle time are computed from them. Reactions earn theirs only if somebody has a question about engagement, and if they do, this is the one destination where keeping them indefinitely is not an argument. Decide from the metrics list rather than from a general instinct to collect everything.

How should Databricks hold a pull request?

Whole in bronze, reshaped in silver. A pull request record is not a flat row: it carries a nested author, an array of labels, a list of requested reviewers and references to branches and repositories. Spark handles that natively, so the landing table can hold exactly what the API returned with nothing discarded or renamed.

That fidelity matters because delivery metrics are derived from timestamps scattered across several streams. Review latency is the gap between a pull request opening and its first review; cycle time spans commits, reviews and the merge. None of those is a field you can select, and all of them need the raw events preserved rather than summarised at load time.

Build the silver layer around those metrics, with one row per pull request carrying the derived durations and the reviewers unnested into something joinable. Then treat the definitions as the deliverable rather than the tables: teams argue about what counts as the start of cycle time, and having that written down in one modelled table is what stops two dashboards disagreeing about the same quarter.

Frequently asked questions

Should I drop the reaction streams?

Not necessarily here. Storage is not the constraint, so the question is whether they are worth the GraphQL points they consume, which depends on whether anybody will analyse engagement.

Why is my sync slow when one rate limit looks healthy?

Check the other one. REST counts requests and GraphQL calculates points, so one can be exhausted while the other appears untouched.

Do I need to flatten pull request records?

No. Spark reads the nesting natively, so let it land intact and unnest reviewers and labels in a silver table where the logic is visible.

Can I compute cycle time from one stream?

No, it spans commits, reviews and merges, which is why the raw events need keeping rather than summarising at load time.

Can I do this without writing code?

The pipeline, yes. Deriving delivery metrics from timestamps across several streams is modelling work, and it is the reason to choose this destination.

Get your Github data into Databricks

Work out which rate limit budget your stream selection loads, because here the API bounds you rather than the destination. Name your repositories. Then let pull request records land nested, since Spark reads them happily, and put the effort into a silver layer carrying agreed definitions of review latency and cycle time, which is the thing two dashboards will otherwise disagree about.

Airbyte's connector catalog includes 600+ pre-built connectors, so engineering activity can be modelled beside everything it affects. For the same source into a search engine, see Github to Elasticsearch, and for issue tracking into the same destination, Jira to Databricks.

Start syncing now →