Article

Validate Connector Configurations with the new PyAirbyte CLI

•

October 14, 2024

•

10 min read

Summarize with ChatGPT

V0.18.0 of PyAirbyte just dropped, bringing a number of new features such as helper functions for testing and benchmarking, and a new execute_sql() method on caches. What I was most interested in trying out though was an experimental CLI, pyairbyte, that you can use to validate connections. This would be super helpful as part of a continuous integration script for open-source installations to ensure your connectors are configured properly.

‍

To get started, make sure you install or update the airbyte library. As a best practice, when testing new features, I always use a virtual environment to keep my local machine neat. Here’s a quick run down of the steps you need, if you are not familiar with virtual environments in Python.

‍

Set up your virtual environment for testing

As a precursor, we need to ensure that we are using python v3.10 or v3.11. This is required by Airbyte due to existing dependencies. If you don’t have it already, on a mac, execute the following:

‍$ brew install python@3.11

‍

Then, I’ll create a new folder to use the experimental features, and set up your virtual environment.

‍$ mkdir pyairbyte-v0.18.0

$ ~/code/pyairbyte-v0.18.0 python3.11 -m venv .venv

‍

Now I can go ahead and activate my virtual environment and grab the latest version of airbyte if you don’t already have a running installation.

‍

$ source .venv/bin/activate

$ pip3 install airbyte

‍

Let’s get to validating!

Alright, with our virtual environment set up, now it is time to test things out. Typing pyairbyte validate –help shows me the parameters I need; specifically the connector name, and either a path or inline yaml snippets. We’ll come back to the –config option shortly.

‍

Now,, of course, at this point, we are going to need a connector to validate against in your environment. In my demo environment, I have a connection configured that uses the faker source. I’ve creatively called source-faker. This is the name (TEXT) that I need to pass into the —connector option on the command line.

‍

Jumping back to the command line, I can now attempt to validate my connection with the following command:

$ pyairbyte validate –connector source-faker

‍

If source-faker exists in my environment, I see the successful results outputted to the terminal.

‍

Using –config option

As you can see from the output above, this returns the connector spec, which I could use a diff on a reference file if I wanted. As a best practice however, you should use the –config option. –config allow you to either provide a path to a YAML file containing values to validate, or pass key/values inline. Inline elements work great for one or two keys. For anything more, I would highly recommend keeping them in a YAML file, which would you could check into version control for consistent CI processes. Just remember, do not store credentials in this YAML file. Always use a secret store and reference that from the YAML.

‍

Let’s say I wanted to validate that count element in my source-faker is configured to have a default value of 1000, I could use the CLI with the following command:

‍

$ pyairbyte validate --connector source-faker --config='{count: {default: 1000}}'

‍

Running this command will succeed, confirming that my connector is configured correctly. Nice! If I actually wanted my default count to be 2000 however, the command would fail by throwing a validation error, which could trigger a notification to team to resolve the issue.

‍

$ pyairbyte validate --connector source-faker --config='{count: {default: 2000}}'

‍

Summary

The new PyAirbyte CLI is still in the experimental phase, but already the addition of the validation function can make a huge impact on continuous integration processes, and ensure that your data pipelines continue to perform as designed. Give it a try and let us know what you think.

‍

Limitless data movement with free Alpha and Beta connectors

Introducing: our Free Connector Program ->

The data movement infrastructure for the modern data teams.

Try a 30-day free trial