Building an Open-Source Engineering Operations Platform

About Thomas Gerber

Thomas leads the Faros Community Edition at Faros AI. He was previously Director of Engineering at Salesforce Einstein, in charge of the Einstein ML Data Lake.

Thomas shared how Faros Community Edition leverages Airbyte to integrate engineering operations data. He covers why Faros CE choose Airbyte for ELT and how they contribute back with custom connectors and a TypeScript Connector Development Kit.

You can also watch the Airbyte Community call with Faros AI.

Why Faros CE needs ELT

‍Faros Community Edition (CE) is an open-source engineering operations (EngOps) platform that connects the dots between all your operational data sources for a single-pane view across the software development life cycle.

Faros CE provides API, BI, and automation layers for engineering operations data (from Pull Requests to Incidents). With Faros CE, you can answer questions about bottlenecks in your CI/CD pipelines, issues in your onboarding processes, progress towards your goals, with data that is fresh, connected, queryable, and most importantly, all in one place.

For example, lead time for changes - or how long it takes for code committed to be running in production - is notoriously difficult to compute because its underlying data spans engineering systems. And yet it is one of the DORA metrics, which most engineering organizations are starting (or trying) to track.

But before any of this can be achieved, data needs to be ingested into the Faros CE platform from many different sources (GitHub, Jira, CircleCI, PagerDuty, …) in order to properly connect and analyze that data. Faros CE decided to leverage Airbyte for that ELT.

Why we chose Airbyte

We didn’t want to spend our energy working on data movement. Instead, we want to focus on our data schema and its useful applications for our community. We hence had to pick a data integration platform.

There are 3 reasons Airbyte stood out:

1. Airbyte has a vibrant open-source community; just check those stats out (powered by Faros CE)!
2. Airbyte’s specification has become the de-facto standard for data movement
3. Airbyte already has a rich catalog of connectors that we can leverage (GitHub for example)

How Faros CE integrates Airbyte and how we contribute back

To integrate Airbyte, we first created the Faros destination connector. It knows how to send data from sources into the platform, and takes care of mapping the various entities being extracted against our rich, connected data schema. Because of the great Airbyte specification, that code is compact and focused on the value Faros CE provides (i.e. that mapping).

Second, we contributed to source connectors important to Faros CE (Jira) or new source connectors that didn’t exist yet (Jenkins, Customer.io, Harness, VictorOps, PagerDuty). Since we use Typescript, we decided to contribute our Typescript CDK. It is our way to strengthen the community while benefiting from it!

Finally, whether it is through docker-compose or plural.sh, Airbyte is deployed as part of Faros CE, with a pre-configured Faros destination, and some pre-configured sources and connections. Our users can have data ingested in the platform in a matter of minutes!

Our users have direct access to the Airbyte instance that is launched as part of their Faros CE platform. They have therefore access to its full feature set! In particular, they can create custom sources or contribute new sources to the Airbyte community with minimal effort! 

Airbyte is the bedrock on top of which Faros CE is built. The product and its community enables us to focus on our core value. We are proud members of the Airbyte community.