Github to Elasticsearch: How to Move Your Data

Moving GitHub into Elasticsearch gives you search across issues, pull requests and discussions that GitHub's own search struggles with at organisation scale. Finding the conversation where somebody explained a decision two years ago, across forty repositories, is a search problem rather than a browsing one.

This guide covers the managed path with Airbyte. Two things shape the build, and the first is a blocker: this destination exists on Airbyte Core and PyAirbyte only. The second is that GitHub text is full of code, which standard text analysis handles poorly.

Github to Elasticsearch at a glance:

Why move data from Github to Elasticsearch?

Two situations account for most of these pipelines.

The first is institutional memory. Engineering decisions get made in pull request comments and issue threads, and finding them later across an organisation's repositories is genuinely hard. An index you control, with your own ranking and filters, does that better than searching each repository in turn.

The second is retaining discussion from repositories that get archived or transferred. If your questions are about counts rather than content, such as how many pull requests each team merged per quarter, that is aggregation and a warehouse answers it better, so Github to BigQuery is the more suitable destination.

What do you need before you start?

Four things, and the first decides whether you can proceed at all:

An Airbyte Core or PyAirbyte deployment. The Elasticsearch destination is not offered on Standard, Plus, Pro or Enterprise Flex, and no configuration works around that. The Elasticsearch destination documentation sets out the position.

A GitHub token and a list of repositories. Leaving the repository field blank gives you everything the token can see, which on an organisation account is a great deal more than most projects need. The GitHub source documentation covers the options.

A decision about reaction streams. Reactions are high volume and almost nobody searches them. Dropping them is usually the single most effective thing you can do for the cost of this pipeline.

An index mapping that expects code. Issue bodies and review comments contain identifiers, stack traces and snippets, and the default text handling treats those less helpfully than it treats prose.

If your cluster restricts inbound traffic by IP, add the Airbyte Cloud IP addresses to the allow list before you begin.

How do you build a Github to Elasticsearch pipeline in Airbyte?

Step 1: Decide what you can do without

Go through the stream list and drop anything nobody will search. Reactions are the obvious candidate, since they are numerous and carry no text worth finding. Name your repositories rather than taking everything the token can see. This is not tidiness: the streams you keep determine which of GitHub's two separate rate limit budgets you spend, and an over-broad selection turns a comfortable sync into a slow one.

Step 2: Configure the GitHub source

Click Sources in the left navigation, then New Source, and select GitHub, following adding a source. Supply the token, your repository list and a start date, then select the streams you settled on. Issues, pull requests, comments and reviews are the ones carrying searchable text and the ones this pipeline exists for.

Step 3: Configure the Elasticsearch destination

Click Destinations, then New Destination, and select Elasticsearch, following adding a destination. Supply the endpoint and authentication, and point it at indexes whose mapping you created deliberately rather than letting Elasticsearch infer one from whichever documents arrive first.

Step 4: Create the connection and test the searches you care about

Click Connections, then New connection, select your streams and a sync mode. Then search for something you know exists, ideally a function name or an error message rather than an ordinary English phrase, because that is the case this dataset will actually be used for and the case most likely to disappoint.

Daily is plenty. Nobody needs a discussion indexed within minutes of it happening, and a gentler schedule keeps both rate limit budgets comfortable.

Why do you have two rate limit budgets?

Because GitHub runs two APIs that are metered differently, and this connector uses both. The REST API counts requests, which is straightforward to reason about. The GraphQL API calculates points based on how much a query asks for, so two calls can cost very different amounts depending on what they retrieve.

Knowing which streams sit on which side is what makes a slow sync explicable. Pull request stats, reviews, pull request comment reactions, issue reactions, releases and projects come through GraphQL and spend points. Everything else spends requests. A sync can therefore be constrained by one budget while the other sits largely untouched, which is confusing if you assumed there was a single limit.

The reaction streams deserve singling out. They are high volume, they sit on the points-based budget, and they hold no text anybody will ever search for. Dropping them is the cheapest improvement available on this pipeline, and if somebody genuinely wants reaction counts later, that is a question for a warehouse rather than a search index.

Why does the default analyser struggle with GitHub text?

Because this text is not prose. Issue bodies and review comments are full of function names, file paths, stack traces and pasted snippets, and standard text analysis was designed for sentences. Identifiers written in camel case or with underscores tokenise in ways that make searching for them unreliable, which is precisely the search engineers will try first.

The practical symptom is that ordinary searches work and technical ones do not. Looking for a phrase somebody wrote returns sensible results, while looking for the function at the centre of the incident returns nothing or returns everything, and people conclude the index is broken when the mapping simply was not built for this content.

Two remedies are worth applying together. Give the body fields an analyser that splits identifiers sensibly, which usually means adding word delimiter handling so camel case and underscores produce useful tokens. And keep repository name, author, label and state as keywords with multi-field where a name also needs partial matching, since those are filters rather than search terms. Decide both before the first sync, because changing a mapping means reindexing.

Frequently asked questions

Can I use this destination on any Airbyte plan?

No. Elasticsearch is available on Airbyte Core and PyAirbyte only, so confirm your deployment before planning around it.

Why is my sync slow when I have plenty of rate limit left?

Check the other budget. REST counts requests and GraphQL calculates points, so one can be exhausted while the other looks healthy. Streams like reviews and reactions spend points.

Should I sync reaction streams?

Rarely. They are high volume, they consume the points budget, and they contain nothing searchable. Dropping them is the easiest saving available here.

Why can I not find a function name I know is in an issue?

The analyser is probably splitting identifiers unhelpfully. Add word delimiter handling to the body fields so camel case and underscored names produce tokens people can search for.

Can I do this without writing code?

The pipeline, yes. The index mapping and any custom analyser are configuration you write in Elasticsearch, and they are what decides whether the search is worth having.

Get your Github data into Elasticsearch

Confirm your deployment supports this destination, since Core and PyAirbyte are the only options. Name your repositories, drop the reaction streams, and understand that GitHub meters REST and GraphQL separately so one budget can run out while the other looks fine. Then build a mapping that expects code rather than prose, and test it with a function name rather than a sentence, because that is the search this index will live or die by.

Airbyte's connector catalog includes 600+ pre-built connectors, so engineering discussion can be searched long after the repository was archived. For the same source into a warehouse, see Github to Snowflake, and for team conversation into the same destination, Slack to Elasticsearch.

Start syncing now →