How to add custom source to PyAirbyte using the no-code builder

Intro

PyAirbyte is an open-source library that packages Airbyte connectors and makes them available in Python, while removing the need for hosted services. This library is built for developers, providing them with the capability to construct customized data pipelines that extract data from multiple sources.

You can explore all pre-built connectors, ready to be utilized in your data pipelines, through various methods. One way is by accessing the official repository of Airbyte connectors in the master branch of airbytehq/airbyte.

‍

‍

You can also obtain a list of all available connectors by installing Airbyte in a Jupyter Notebook using pip, then execute the following line of code:

# Importing PyAirbyte 
import airbyte as ab
# Show all available connectors
ab.get_available_connectors()

Building Airbyte connectors

There are several methods for building new connectors in Airbyte. Many of the connectors included in the PyAirbyte source are defined with a manifest.yaml file. This file defines all the specifications required for the connector to function, including the streams that define the data schema for syncing in the Airbyte platform. A stream often corresponds to a resource within your API.

Below is a sample YAML file that defines the source-pokeapi connector, which is available in the official repository of Airbyte connectors.\

# Importing PyAirbyte
version: 0.78.5
type: DeclarativeSource
check:
  type: CheckStream
  stream_names:
    - pokemon
definitions:
  streams:
    pokemon:
      type: DeclarativeStream
      name: pokemon
      primary_key:
        - id

base_requester:
    type: HttpRequester
    url_base: https://pokeapi.co/api/v2/pokemon

Defining your custom connector

To define our custom connector, we will utilize the platform provided by Airbyte for this purpose, known as the Connector Builder in Airbyte Cloud.

Shown below is the home page of the connector Builder. You can import a YAML file that you have developed on your own, load an existing connector (one of the manifest.yml available in the airbyte-integration GitHub page for example), or start a connector from scratch which we will show later in this blog post.

‍

‍

Here are some mandatory fields that you will need to fill in the Global configuration of your connector:

• The Connector Name
• The API Base URL
• The Authentication Method (if it exists, as there are APIs that do not need auth methods like API keys or API-tokens)

‍

‍

If you're experienced with APIs, you're likely aware that every API responds to each call made to its endpoints by providing a response. Here we are doing a basic API call to our Jokes API endpoint https://api.chucknorris.io/jokes/ , which returns a successful status (200). We can validate this by just clicking the Test button in the right-side panel.

‍

‍

Next, you can define some Streams that your connector will connect to. To set this up, you should have read in detail the corresponding API documentation.

‍

‍

Using custom connectors in PyAirbyte

As developers, we aim to reuse the code we've developed across various platforms and environments. This is also achievable in Airbyte; whether we're managing data in Airbyte OSS, Airbyte Cloud, or within a Jupyter Notebook using PyAirbyte.

I've prepared a notebook to demonstrate this process. Once we're satisfied with our connector's output, we can export a YAML file.

‍

Following is the definition we will utilize in the Notebook, leveraging the new feature introduced by the Airbyte Team to execute declarative YAML sources with PyAirbyte. First, we will define the YAML specs in the Notebook (this is the YAML definition that we have downloaded from the Connector Builder UI), and then save our source_manifest_yaml source_manifest_dict: dict = yaml.safe_load(source_manifest_yaml) as a dictionary in the Notebook.

‍

‍

This is how to use the get_source() implementation provided by the Airbyte Team with the airbyte.experimental method. This method includes the new source_manifest input argument, which we use to pass our defined source_manifest_dict. This allows us to read the YAML defined in the Connector Builder UI.

The first thing we need to call the Airbyte library in Python together with the get_source() method.

import airbyte as ab
from airbyte.experimental import get_source

‍

‍

The result of the above code segment is that we have connected to our custom connector using a YAML file passed as an argument to the newly created get_source() method.

Conclusion

The Airbyte Connector Builder is a powerful tool for developers, enabling direct interaction with data and the creation of custom API integrations from scratch. It allows us to tailor connections to our specific needs and share our discoveries with the broader data community.

The notebook containing the code developed for this tutorial is available on Google Colab.

References

• https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/yaml-overview
• https://www.youtube.com/watch?v=g_NZPHVNB38&ab_channel=Airbyte
• https://docs.airbyte.com/connector-development/connector-builder-ui/overview
• https://github.com/airbytehq/PyAirbyte/pull/175