Blog

Engineering

Validate Connector Configurations with the new PyAirbyte CLI

Validate Airbyte connector configurations with the PyAirbyte CLI to catch issues earlier and improve local development workflows.

ENGINEERING

October 14, 2024

10 min read

Quinton Wall

Summarize with AI:

Validate Connector Configurations with the new PyAirbyte CLI

V0.18.0 of PyAirbyte just dropped, bringing a number of new features such as helper functions for testing and benchmarking, and a new execute_sql() method on caches. What I was most interested in trying out though was an experimental CLI, pyairbyte, that you can use to validate connections. This would be super helpful as part of a continuous integration script for open-source installations to ensure your connectors are configured properly.

‍

To get started, make sure you install or update the airbyte library. As a best practice, when testing new features, I always use a virtual environment to keep my local machine neat. Here’s a quick run down of the steps you need, if you are not familiar with virtual environments in Python.

‍

Set up your virtual environment for testing

As a precursor, we need to ensure that we are using python v3.10 or v3.11. This is required by Airbyte due to existing dependencies. If you don’t have it already, on a mac, execute the following:

‍$ brew install python@3.11

‍

Then, I’ll create a new folder to use the experimental features, and set up your virtual environment.

‍$ mkdir pyairbyte-v0.18.0

$ ~/code/pyairbyte-v0.18.0 python3.11 -m venv .venv

‍

Now I can go ahead and activate my virtual environment and grab the latest version of airbyte if you don’t already have a running installation.

‍

$ source .venv/bin/activate

$ pip3 install airbyte

‍

Let’s get to validating!

Alright, with our virtual environment set up, now it is time to test things out. Typing pyairbyte validate –help shows me the parameters I need; specifically the connector name, and either a path or inline yaml snippets. We’ll come back to the –config option shortly.

‍

Now,, of course, at this point, we are going to need a connector to validate against in your environment. In my demo environment, I have a connection configured that uses the faker source. I’ve creatively called source-faker. This is the name (TEXT) that I need to pass into the —connector option on the command line.

‍

Jumping back to the command line, I can now attempt to validate my connection with the following command:

$ pyairbyte validate –connector source-faker

‍

If source-faker exists in my environment, I see the successful results outputted to the terminal.

‍

Using –config option

As you can see from the output above, this returns the connector spec, which I could use a diff on a reference file if I wanted. As a best practice however, you should use the –config option. –config allow you to either provide a path to a YAML file containing values to validate, or pass key/values inline. Inline elements work great for one or two keys. For anything more, I would highly recommend keeping them in a YAML file, which would you could check into version control for consistent CI processes. Just remember, do not store credentials in this YAML file. Always use a secret store and reference that from the YAML.

‍

Let’s say I wanted to validate that count element in my source-faker is configured to have a default value of 1000, I could use the CLI with the following command:

‍

$ pyairbyte validate --connector source-faker --config='{count: {default: 1000}}'

‍

Running this command will succeed, confirming that my connector is configured correctly. Nice! If I actually wanted my default count to be 2000 however, the command would fail by throwing a validation error, which could trigger a notification to team to resolve the issue.

‍

$ pyairbyte validate --connector source-faker --config='{count: {default: 2000}}'

‍

Summary

The new PyAirbyte CLI is still in the experimental phase, but already the addition of the validation function can make a huge impact on continuous integration processes, and ensure that your data pipelines continue to perform as designed. Give it a try and let us know what you think.

‍

About the Author

Quinton Wall

Quinton is the Head of Developer Relations at Airbyte

Try Airbyte Agents

Be among the first to explore our new platform and get access to our latest features.

Try it free

Validate Connector Configurations with the new PyAirbyte CLI

Try Airbyte Agents

Related posts