Article

Introducing Airbyte Hack Days

•

July 6, 2022

•

Summarize this article with:

48 hours, 31 Airbyters, and 16 outstanding projects later, Airbyte wrapped up its first internal hackathon. We opted for a two-day format in a concerted effort to find balance between maintaining our standard velocity and creating space to build new and novel things. It felt like just the right amount of time to create an MVP of each idea without getting too caught up in the details. Airbyters across the organization came together to pitch creative ideas, build new workflows and processes, and support each other with feedback along the way.

The official Hack Days began with elevator pitches on Monday morning to recruit team members and add a little competitive flex to the proceedings. Once the participants found a project they wanted to work on, the teams got busy hacking for two days straight. We canceled our regular meetings and our company Slack channel was nothing but crickets as the team worked away on their ideas.

48 hours later, the company got together to watch the highly anticipated presentations. In total, 31 people presented 16 different projects to the team. We voted on our favorite projects–overall favorite, hottest idea and highest impact project.

Now I bet you are curious what exactly we hacked together! Here is a summary of the 16 projects our team took on:

Connector Schema diagrams by Chris D., Marcos, Simon and Sherif

Our goal was to automatically generate an ERD (Entity Relationship Diagrams) within our connector docs to make it easier for our users to understand what they get for every connector. Instead of reverse-engineering the columns and tables generated through Airbyte, we provide users with a diagram ahead of time. We managed to develop the first versions by generating a DBML file with dbt that we uploaded to dbdocs.io. As an example of how this looks, you can check out Sendgrid.

Adaptive Connector by Liren and Davin

This project aims to remove the variants and simplify the development cycle when publishing connectors on Airbyte Open Source vs. Airbyte Cloud. The underlying idea of the solution is to make a connector adaptive based on the deployment environment. For example, our Postgres connector has different variants in Open Source and Airbyte Cloud, with Airbyte Cloud having higher security restrictions. Now, instead of publishing the Postgres connector twice, the connector can simply listen to an environment variable telling it where it is deployed (Open Source versus Airbyte Cloud), and automatically change its behavior as needed.

Airbuild (Workstation init script) by Benoit, Jimmy, Malik and Topher

Our goal was to install system dependencies for working at Airbyte in a simple and idempotent way. Using a simple structure, Airbuild checks for the state of your system and installs needed dependencies without the need for user intervention. Because we check for dependencies for every installed item, the current state of the system isn’t as brittle as when using generic Make or Bash scripts.

Cassettes by Alexandre and Pedro

We prototyped speeding up regression testing for API sources by leveraging VCR.py to record all HTTP interactions that take place during an initial test run, and replay them in subsequent runs. This allows for faster test runs as we don't need to hit the endpoint, and stronger assertions because we can assert we're sending exactly the HTTP requests that were recorded. As a bonus, this also allows us to run integration tests offline.

Octavia CLI metabase dashboard by Augustin

Our CLI was released in April. We built a dashboard to monitor its usage and have a glimpse into the hurdles and errors our users are encountering. Metabase, our dashboard tooling, along with our great internal analytics platform, allowed us to quickly design a basic dashboard.

Redesign documentation UX by Amruta, Sophia and Grace

We reimagined our documentation homepage UI by creating a clickable prototype using Figma. The goal was to design an intuitive and user-centric experience to help users navigate our various resource centers (documentation, tutorials, videos) and find the information they need.

Faster Connector Docker builds by Edward

This project refactored our Gradle build to reduce wasted build time with tightly-scoped dependencies and more reliable build caching.

Connector manager by Andy and Nico

We mocked up a user interface that allows both internal teams and external contributors to easily make changes to a connector’s metadata without submitting a pull request, and see a single source of truth for the information about each connector.

Automatic Airbyte releases by Lake and Charles

We wrote a Github action to automatically generate a new release of Airbyte Open Source whenever a “bump version” PR is merged into master. This removes some manual action that was required when releasing Airbyte, and reduces the likelihood of human error causing issues for later releases.

Connectors data service by Evan

This project prototyped what creating an API to expose all of our connector data in one place would look like. Deploying this would mean faster releases to both our Open Source and Cloud users, and opens the door for future projects like searching for connectors and easier releasing for third-party Airbyte connectors.

Airbyte Audit Logs by Subodh and Terence

We prototyped what an Audit Logging system could look like in Airbyte. Although Audit Logging is already on the product roadmap for the future, we used this opportunity to try out technologies we were less familiar with. By utilizing PubSub, Dataflow, and Elasticsearch, we ended up with a searchable log of all calls to any Airbyte API endpoint.

Connector CDK UI by Brian L.

This project experimented with what it might be like to be able use a UI-based tool to develop an API connector. It looked like Postman where you can explore the relevant API. It was able to extract the schema and had inputs to teach it how the API handled authentication, pagination, and parsing. Brian built a new connector within 5 minutes during his demo of the prototype. This is VERY exciting for our community. Imagine being able to build a connector in 5 minutes for most APIs! We’ll get there as we’ll productize this prototype.

Analytics team documentation by Alyssa

Many groups at Airbyte are leveraging the analytics team more and more, so we put together a robust Notion page outlining how to best work with us and how to get projects scoped and slotted into upcoming sprints.

Manager 101 by Jerri, Meghan and Anita

Airbyte is a small, fast-growing company with a lot of folks in managing or team lead positions for the first time. We created a Notion guide with tips and tricks to make managing easier and more transparent.

Standalone catalog by Harshith

Today, a connector in the Airbyte UI is only shown in the dropdown, which lacks the real estate to add more information which may be useful to our users and customers. This project aims to improve connector information in two ways:

Visibility for users: more information shown to them such as connector quality, supported features, ownership, performance indicators and more.
Discoverability for users: when users arrive at Airbyte, they might not always know which connector they want and hope to use Airbyte as a directory for them to find the most appropriate vendor to add to their stack.

Design language CSS improvements by Edmundo

The goal of this project was to update our design system for our web application across our webapp.

We were able to create SCSS variables to establish an updated color palette and define the common spacing, shadows and rounded corner as SCSS variables.

Additionally, we experimented with the idea of creating a utility React component that can be generated by assigning css classes instead of styles. For example, a card component could be defined as a <div /> with classes that give it a white background, a drop shadow, and rounded corners.

—

We’re already looking for the next session!

‍

Limitless data movement with free Alpha and Beta connectors

Introducing: our Free Connector Program ->

The data movement infrastructure for the modern data teams.

Try a 30-day free trial