Blog

Data

Data contracts and Airbyte: A partnership for maintaining data consistency

Data Contracts and Airbyte - Exploring the partnership between data contracts and Airbyte for maintaining data consistency.

DATA

December 20, 2023

10 min read

Madison Schott

Summarize with AI:

Data contracts and Airbyte: A partnership for maintaining data consistency

Although data contracts are a new buzzword in the data industry, they attempt to solve an age-old data problem. A problem that centers around the disconnect between engineers generating and using data.

Data producers write the code that generates a company's data. However, they rarely consult with data consumers about the data that is actually needed. Then, when the consumers think they know what to expect from a data source, the code changes without anyone being notified. Now the data infrastructure and models are breaking and nobody knows why!

We’ve always experienced this- it’s nothing new. We now finally have something to call it that everyone universally understands.

While we finally have somewhat of an answer to this problem, it’s the actual implementation of the solution that we struggle with. Data contracts are key to a strong stream of communication between data producers and data consumers. However, it’s building the right systems of communication that is the most challenging part.

With data contracts in place, specifically for data ingestion, we can minimize data fires, pipeline breakages, and failing tests.

What is a data contract?

If you’re new to the term data contract, you’re probably wondering what this means. The term doesn’t make much sense without the right context.

A data contract is an agreement between the engineers who produce the data (think software engineers) and the data or analytics team that is then using this data.

Data contracts are a way to set expectations about the data. Typically, analytics engineers or data engineers should be setting expectations with software engineers on how they want/need the data to look. For example, if analytics engineers expect all primary keys to be integers, software engineers need to ensure they are generating these fields as integers in the source system.

Setting these expectations ensures that the data coming in doesn’t alter from what was agreed upon, preventing models from breaking in the analytics pipeline. Because who wants to deal with unexpected breakages due to issues in the source data?

Why are data contracts important for data ingestion?

Data contracts are arguably one of the most important aspects when it comes to data ingestion. When data is ingested into the data warehouse, analytics engineers expect it to look a certain way. They most likely have tests in place that check the source data for the agreed-upon expectations.

How this data looks when it’s ingested determines how downstream data models are built. If the source data is different than expected, then the data models could break or behave unpredictably. For example, if you’re expecting an integer field that you cast to a boolean in a staging model, a simple data type change would break the view created by the staging model. With data ingestion and data modeling, all of the small details about your data matter.

The most important aspects of data contracts

When implementing data contracts on your data ingestion pipeline, it’s important to consider two main factors- data types and ingestion frequency. Each of these has the power to affect downstream data models when standards for them are not properly set.

Defining data types

Data type is an important piece of metadata that is typically defined in a data contract. Data types often determine how analytics engineers code data models and how data analysts derive metrics. If an analytics engineer expects a field to be one data type and it’s ingested as something completely different, this has the power to break an entire data pipeline.

For example, if every dataset has a field that’s a JSON object and contains all of the key information about a record, this JSON object must be properly parsed to be used in downstream data models. The analytics engineer may be using a function specific to the data warehouse that parses that JSON object into a usable form.

The staging models could look something like this:

WITH customers AS (
 SELECT
   id,
   JSON_PARSE(metadata) AS metadata
 FROM {{ source ('raw_data', 'customers')}}
),


customers_json_parsed AS (
 SELECT
   id AS customer_id,
   metadata.name::VARCHAR AS customer_name,
   metadata.email::VARCHAR AS customer_email,
   metadata.state::VARCHAR AS customer_state,
 FROM customers
)


select * from customers_json_parsed

Imagine these staging models form the first layer of the entire data transformation ecosystem. All downstream data models reference these staging models.

Then, let’s say the engineering team decides they are no longer going to use JSON objects to store this key information. Instead, they will format this data as pre-parsed fields readily available in the dataset. While this is great, it’s not how the data models are currently coded.

This may not break all of the data models, but it will make the data go stale. Or, if the engineers decide to stop sending this JSON object field entirely, the pipeline will be set on fire. Obviously, we don’t want this happening and this is exactly why we form data contracts to prevent this.

Ingestion frequency

In addition to data type, knowing the frequency at which new data is being generated is important in setting expectations for when your data ingestion processes and data models need to run.

For example, if the system generates new data in the source daily, it wouldn’t make sense for you to ingest data hourly. You may even trigger alerts if your dbt project, for example, expects data hourly but only receives it daily.

This is why it’s also important to set true expectations via tests and contracts in your project. Data freshness tests are determined by how frequently the source systems are receiving new data. Make sure these are accurate from the very beginning and not changing to avoid false alarms.

Airbyte Schema Propagation

While managing ever-changing data types and ingestion frequency can be a challenge, modern data stack tools like Airbyte have done a great job at assisting with this. Airbyte helps you maintain contracts between data and software engineers by allowing you to choose how to handle changes in source schemas.

This way, you won’t be caught off-guard if a contract isn’t maintained, preventing data from suddenly changing downstream.

Airbyte offers four different options for how you can handle schema changes in your source:

Propagate all field and stream changes
Propagate field changes only (this is the default option for new connections)
Detect changes and manually approve
Detect changes and pause connection

For more on how Airbyte’s schema propagation feature works and how you can set it up for your won connectors, check out this recent article.

Applying data contracts in dbt

If you use dbt for data transformation, you may be surprised to learn that they offer a data contracts feature. Together with Airbyte’s schema propagation, this can help set you up for success and minimize breakages in your data pipeline.

While dbt’s contract feature doesn’t look at source tables specifically, you can use it to ensure your data is being ingested as specified before your models build. This is opposite to how testing works where the models are already fully built before the tests are applied.

You can think of testing as a reactive approach and contracts as a proactive approach.

With dbt, you must first apply a contract to a model, setting enforced to be true. Note that this is only supported on tables and views. Let’s apply a contract to the model we wrote above.

models:
 - name: customer_segments
   config:
     contract:
       enforced: true

Once you have enforced a contract on a model, you must specify the datatype on every field within the model.

models:
 - name: customer_segments
   config:
     contract:
       enforced: true
   columns:
     - name: customer_id
       data_type: int
     - name: customer_name
       data_type: string
     - name: customer_email
       data_type: string
     - name: customer_state
       data_type: string

You also have the option to set a “constraint” on specific fields within that model. dbt contracts support a few different types of constraints:Not_null Unique primary_keyforeign_key

Not_null is the only constraint supported and enforced across all platforms. Postgres is the only data platform that supports all four of these constraints. The status of the others depends on the specific data warehouse that you use.

You can define a constraint on a field in your model like so:

models:
 - name: customer_segments
   config:
     contract:
       enforced: true
   columns:
     - name: customer_id
       data_type: int
       constraints: 
        - type: not_null
        - type: primary_key
     - name: customer_name
       data_type: string
     - name: customer_email
       data_type: string
       constraints:
        - type: not_null
     - name: customer_state
       data_type: string

While dbt still has a long way to go with building out contracts and constraints (since most data warehouses do not support all of them), it is a powerful feature to move towards data contracts as a normal practice. Data contracts are still fairly new but an important practice that won’t be going away. Integrate them into your ecosystem now so you can continue to build on them with future data sources.

Conclusion

Data contracts are a powerful tool to ensure data is being ingested into your data warehouse as expected. You can specify them directly within your dbt project to ensure data being ingested by Airbyte matches your expectations before you use this data downstream in models. This ensures your models always contain accurate and reliable data!

You can also take advantage of Airbyte’s schema propagation feature to ensure engineering is maintaining their part of the data contract. This way, analytics engineers can hold software engineers accountable without the pipeline breaking being the thing that spurs conversation. Implementing and maintaining contracts is a proactive feature that will help prevent pipeline failures, ensuring the business has the data it needs to make strategic decisions in real time.

About the Author

Madison Schott

Madison Schott is an Analytics Engineer, author of The ABCs of Analytics Engineering book. Madison blogs about the modern data stack on Medium and writes a newsletter on Substack.