GitHub to BigQuery: How to Move Your Data
Replicate GitHub to BigQuery with Airbyte. Rate limit handling and multiple tokens, repository wildcards, silently skipped repos, and stream limits to know.
Replicating GitHub to BigQuery gives you engineering data in a warehouse: pull requests, issues, commits, reviews, and workflow runs, joinable against everything else and queryable with SQL. It is how teams build cycle-time dashboards and DORA metrics without paying for a tool that computes them for you. Airbyte handles the extraction with no code.
One thing shapes the pipeline more than any other, and it is not the warehouse side: GitHub's API rate limit. How the connector handles exhaustion, and what you can do to widen the budget, determines how long your syncs take and whether they complete at all.
How does the connector handle GitHub's rate limits?
The GitHub source behaves sensibly when quota runs out. Rather than failing immediately, it waits for the limit to reset. That wait is bounded by a Max Waiting Time setting, which defaults to 120 minutes and can be raised to 240. Rate-limit exhaustion is classified as a transient error, so if the connector does exceed that ceiling, Airbyte retries the sync according to your connection's retry behaviour.
The consequence is that a sync across a large organisation does not usually fail. It stalls, sometimes for hours, which looks like a hung job unless you know to expect it. If a sync is running far longer than the data volume suggests, check whether it is parked waiting for a quota reset before assuming something is broken.
The lever that actually helps is supplying more than one token. The connector accepts multiple personal access tokens separated by commas, and load balances quota consumption across them. For an organisation with many repositories, this is the single most effective configuration change available, and it is easy to miss because it looks like an ordinary credential field.
Consult GitHub's rate limit documentation for the limits that apply to your account type, since they differ between personal tokens, apps, and enterprise arrangements.
Selecting repositories and scopes
You list repositories in the form of organisation and repository name, separating multiple entries with spaces. To cover every repository an organisation owns, use the organisation followed by a wildcard rather than enumerating them.
One behaviour is worth knowing before it costs you a week of missing data. Repositories with the wrong name, or that do not exist, or that use the wrong name format, are skipped with a warning in the logs rather than failing the sync. A typo produces a successful sync that is silently missing a repository. After the first run, check the log warnings against your intended list rather than trusting the green tick.
On authentication, Airbyte Open Source uses a personal access token, while OAuth is available on Cloud. The OAuth application requests a defined set of scopes covering repositories, organisation membership, repository hooks, users, discussions, projects, and workflows. If a stream you expected is absent, an insufficient scope is the first thing to check, since permission gaps here tend to present as missing streams rather than as explicit errors.
An optional start date limits how far back streams replicate. Setting it thoughtfully is the easiest way to cut the size of your initial sync, which is also the sync most likely to run into quota limits.
Which stream limits should you know about?
The Releases stream uses GitHub's GraphQL API and fetches up to 100 assets per release. Releases carrying more than 100 assets include only the first 100, because sub-pagination for release assets is not currently supported. If you publish builds for many platforms and architectures, your asset data is truncated, and nothing in the sync signals it.
This is a good example of a general principle for this connector: the streams are numerous and their individual behaviours vary, so select only the streams you will actually query. Every additional stream consumes quota that the streams you care about could have used, and on a large organisation that trade is real rather than theoretical.
Landing the data in BigQuery
The BigQuery destination uses direct load, writing straight to final tables. Batched Standard Inserts is the default loading method and suits GitHub volumes comfortably. GCS staging adds a bucket, an HMAC key, and a Storage Object Admin grant for little benefit at this scale, and does not support buckets using customer-managed encryption keys.
Your service account needs BigQuery User and BigQuery Data Editor. Set the dataset location to match the datasets you will join against, because BigQuery cannot query across physical locations and the location is fixed once the dataset exists.
GitHub API responses are deeply nested, with user objects, label arrays, and review structures inside a single pull request record. Those arrive as structured JSON rather than flat columns. Since the whole point of this pipeline is usually a metric such as time to first review or merge frequency, plan a downstream model that extracts the handful of fields those calculations need. Tables are partitioned daily on the extraction timestamp rather than on event time, so filtering on a pull request's creation date will not prune partitions.
Frequently asked questions
My sync seems to hang for hours. Is it broken?
Probably not. When quota is exhausted the connector waits for the reset rather than failing, bounded by Max Waiting Time, which defaults to 120 minutes and can go to 240. Supplying multiple tokens is the fix.
A repository is missing but the sync succeeded.
Repositories with an incorrect name, an incorrect format, or that do not exist are skipped with a warning in the logs rather than failing the run. Check the warnings against your intended list.
Can I sync every repository in my organisation?
Yes, using the organisation wildcard form rather than listing repositories individually. Expect quota pressure, and supply multiple tokens if the organisation is large.
Why is my release asset data incomplete?
The Releases stream fetches up to 100 assets per release and does not sub-paginate, so releases with more than 100 assets include only the first 100.
Get your GitHub data into BigQuery
Supply multiple tokens, select only the streams you will query, check the logs for skipped repositories, and model the nested JSON into the metrics you actually want. For another API source into the same warehouse, see our guide to Datadog to BigQuery. For the general pattern of loading API data, see API to database.
Integrate with 600+ apps using Airbyte
Move data from 600+ sources into warehouses, lakes, and beyond. Set up pipelines in minutes with pre-built connectors and the Connector Builder.