GitHub to MySQL: How to Sync Your Data

Syncing GitHub to MySQL is mostly an exercise in rate limit budgeting, and it is unusual because you are budgeting against two limits at once that have nothing to do with each other.

The GitHub source reads from both of GitHub's APIs. Most streams use the REST API, which counts requests. A handful use the GraphQL API, which counts points, where each query costs an amount GitHub calculates from what you asked for. The two budgets are tracked separately, so you can exhaust one while the other is untouched, and a sync that looks fine on paper can stop halfway because six of your forty streams were quietly spending a different currency.

Know which streams cost points

The streams that go through GraphQL rather than REST include pull request stats, reviews, pull request comment reactions, issue reactions, releases and the newer projects endpoint. If you are syncing an organisation with a lot of activity, the reaction streams in particular are worth thinking about before you select them: they are high-volume, they are usually the least analytically valuable thing in the catalogue, and they are drawing down the point budget that the streams you actually care about also need.

The general principle is worth stating plainly. Select streams because someone will query them, not because they were ticked by default. On a source with two independent rate limits, an unused stream is not free, it is competing for capacity with a used one.

Three fixes when you run out

When limits are reached before every stream has been read, the recommended remedies are incremental sync, a longer interval between syncs, and splitting the work into separate connections carrying fewer streams each. They compound rather than compete, and the third one is the most useful and least used.

Splitting by API is the natural cut. Put the GraphQL-backed streams in their own connection on a slower schedule, and let the REST streams run more often. Each connection then has its own budget, its own schedule and its own failure boundary, so a reaction stream exhausting the point budget stops affecting whether your issues and commits arrived. It is more objects to manage and it is worth it on any organisation big enough to hit the limits in the first place.

Landing GitHub data in MySQL

The destination side is simple: a database, a schema and a user with permission to create and write tables. Each stream becomes its own table with Airbyte's metadata columns alongside your data.

The modelling is where the work is. GitHub API responses nest heavily, and a pull request record carries user objects, label arrays, milestone objects and repository references inside it. In MySQL those land as JSON rather than as columns, which is correct behaviour and mildly awkward to query. Build a small set of downstream views that extract the fields your team actually uses, an author login, a merge timestamp, a label list, and point people at those rather than at the raw tables. It is fifteen minutes of work that determines whether anyone uses this dataset.

One thing to decide early: whether you are syncing one repository, several, or an entire organisation. The cost difference is not linear, because the high-volume streams scale with activity rather than with repository count, and one busy monorepo can consume more budget than twenty quiet services.

Frequently asked questions

Why did my GitHub sync stop before reading every stream?

You exhausted one of GitHub's two rate limits. REST and GraphQL are budgeted separately, so check which streams were still pending and which API they use.

Which streams use the GraphQL API?

Pull request stats, reviews, pull request comment reactions, issue reactions, releases and the projects endpoint. These consume calculated query points rather than request counts.

Should I split my GitHub sync into multiple connections?

On any sizeable organisation, yes. Separating the GraphQL-backed streams onto their own slower schedule gives each connection its own budget and its own failure boundary.

Why is my pull request data full of JSON columns?

GitHub responses nest objects and arrays inside each record, and Airbyte preserves that. Build downstream views that extract the specific fields your team queries.

Get your GitHub data into MySQL

Select streams deliberately, split the GraphQL-backed ones onto their own connection and schedule, run incrementally, and put a thin modelling layer over the nested payloads before anyone else sees them. Do those four things and this connector is undramatic. Skip them and you get partial syncs nobody can explain.

For another source where an API limit ends syncs early and the fix is a sync mode rather than a schedule, see Salesforce to PostgreSQL. For a maintained commerce connector with its own bulk API behaviour, see Shopify to BigQuery.

Start syncing now →