Article

How Airbyte Aligns with Software & Data Engineering Best Practices

•

April 22, 2024

•

10 min read

A few years back, when I was hired as the first analytics engineer on a brand-new data team, I discovered a minefield of issues created by the previous data consultants. They didn’t use GitHub for version control, models were built using all of the worst SQL practices, and engineers were making changes to the data without the data team knowing. When best practices aren’t followed, it causes a host of issues and makes them impossible to discover, let alone debug.

While analytics teams often don’t employ the same best practices as software engineering teams, they must do so. Following software engineering best practices in all areas of your data stack is the key to producing high-quality data that the business can trust. Without these best practices in place, tech debt builds up while trust in the business diminishes. Unfortunately, once you lose trust, it is hard to get it back.

Luckily, with the rise of the modern data stack, we see software engineering best practices being applied to analytics more and more often. Soon enough, it will become the new norm! Tools like Airbyte are making it easier for data teams to build things the right way, in a way where the stack can scale.

In this article, we will discuss the software engineering best practices embedded throughout Airbyte that allow you to build a solid, stable, scalable analytics foundation.

Best Practice #1: Handling schema changes

On every data team I’ve been on, there’s been a huge disconnect between backend engineers (data producers) and analytics engineers (data consumers). It’s been extremely common for data producers to make schema changes, sometimes breaking ones, without letting analytics engineers know.

This then breaks downstream data models, causing a cascade of failures that the analytics team is responsible for fixing. We typically need to stop everything we are doing, alert data producers of the breaking change, and do everything in our power to fix it without sacrificing data quality.

These sudden, urgent changes put us in a bad spot as we don’t want to act too fast for fear of removing important tests or changing expectations of the data, but we also need to consider business stakeholders whose trust diminishes as errors persist.

Luckily, if you are using Airbyte, this disconnect causes fewer issues. Airbyte actually protects against breaking and non-breaking changes by allowing you to review changes before they are made to the connector. This means that even if an engineer changes the schema of a data table, that change will not manifest until you’ve approved the change in your data connector.

Airbyte has four different settings for the detection and propagation of schema changes:

Propagate all field and stream changes- This setting allows all changes to manifest in your tables whether there are changes in the stream, the addition or deletion of fields, or a change in data types. Keep in mind that stream changes include changes in sync behavior, tables being synced from a source, and propagation itself.

This setting wouldn’t protect against the issue I mentioned before but it may be a good option if you have a dependable change process set up between engineering and analytics.

Propagate field changes only- This setting will only allow changes in fields in your table to propagate downstream, including additions, deletions, and data type changes. The only difference here is that changes in the stream will not be automatically handled.

Detect changes and manually approve- This setting detects all of the changes to both your fields and streams, making you aware of them when you look at the connector in the UI. I highly recommend using this setting, as you need to manually approve any changes that are made to your tables. This gives you a chance to check all downstream dependencies and ensure the changes being made will not break any of your data models, tests, or analytics reporting.

It essentially acts as a safeguard to any changes that engineering may make without your knowledge. And, even if they tell you about them, maybe after the fact, it allows you to evaluate the changes without a sense of fear and urgency.

Detect changes and pause connection- Lastly, this setting will detect changes and immediately pause the connection. While this wouldn’t be ideal if you have downstream dependencies on a data source, it could be good for data sources that you are still exploring and testing.

A simple freshness test on the data source could also help you detect when a connection has been paused and needs to be evaluated for schema changes. This is just an extra way to protect your data pipeline from breaking changes.

Best Practice #2: Typing and deduping

Software and data engineers ensure they handle duplication at the source when producing any kind of data. Best practices include incrementing primary keys, utilizing staging tables, hashing keys, and different types of testing to ensure duplicates don’t make their way into production data. In the case that duplicates do make it in, it’s another best practice to ensure monitoring and logs are properly set up to help detect any problems.

Airbyte utilizes per-row error handling to ensure all of your data makes it to your destination, rather than failing your entire sync. They simply store details about any failures in a field called _airbyte_meta that you can then query for deeper insights. In addition, failures only occur when there are issues moving your data rather than with the content of the data.

To ensure you always have the highest quality data, Airbyte utilizes something called a “soft reset”. This essentially means dropping and rebuilding your tables from time to time, in a way that is nearly invisible to the user. Soft resets allow Airbyte to have full control over different data characteristics like data type and field order while keeping your tables fully populated.

All of these “behind the scenes” processes allow your data to consistently meet your expectations.

Best Practice #3: Transparency in errors and logs

Transparency is necessary in any software engineering environment. Without transparency into errors and logs, you have no insight into the issues in your code and how you can fix them. Having a black box environment is one of the worst things possible for software code but also data environments. You need full transparency into errors and what is happening at all times to make for easy debugging.

Airbyte’s sync history view gives you insight into everything you need to know about your past and current syncs. It shares important key data points like the amount of data moved, when the sync occurred, the number of records read and committed, and the total sync time. Each of these data points is always available for you to look back on, along with the logs.

I’ve personally used the sync logs to understand the different stages of my sync and how each table created relates to the finished process. They contain detailed records of the different tables generated, the number of records moved, when normalization occurred, and any potential errors that the sync has run into. The logs are also extremely searchable, allowing you to look for key terms to help find what you’re looking for.

Best Practice #4: Version Control

One of the most important software engineering practices is finally making its way over to analytics workflows. And that is version control. Version control allows you to keep track of changes in your code over time, giving insight into the state of your project at any point in time.

It’s especially powerful because it allows you to revert changes in the case of a bug being pushed to production or review a teammate’s code before merging it into the main codebase.

Airbyte offers a Terraform provider that allows you to automate and version-control your Airbyte configuration as code. In other words, it allows you to implement Infrastructure-as-Code (IaC). This way, you can keep track of the changes you make to your connections and their streams over time. Similar to version-controlling software or data model code, this gives you a clear picture of how your connector code has changed over time.

This is ideal for debugging any problems you may see in source data. For example, let’s say you notice some missing records in a source table. One of the first things you want to do is see if anything with your connector configuration has changed since the missing records started appearing.

With the Terraform provider, you can look at all of the changes to your connector, correlating any problems that have arisen to specific changes that you’ve made. This is extremely powerful and allows you to have full ownership over your change logs.

Best Practice #5: Alerting

Lastly, you need to have alerts properly configured to send to the right locations. Software engineers often have entire platforms dedicated to monitoring the state of their infrastructure and how stable it is. This allows them to be alerted of any failures, immediately, in the place they are going to see it.

I’ve found that many analytics tools can easily hide key failures and alerts if you don’t have them properly configured. The failures can live in the UI of the tool, which you don’t end up checking on a daily basis. However, Airbyte makes it easy to set up notifications to the platforms that you are using most often (email and Slack).

Airbyte makes it easy to customize notifications to ensure they are most helpful to you and what you care about. While there are eight different types of failure notifications, here are the few that I find most helpful:

Failed Syncs- This sends an alert every time a sync fails to run. I currently have these types of notifications set up for my syncs to send to an #analytics-monitoring channel. Whenever I see a sync failure, I tag the owner responsible so they can look into it.

Connection Updates Requiring Action- This notification updates you when there is an update to a connection that requires action on your part. This could be a breaking schema change or a data table that gets removed.

Warning on Repeated Failures- This notification will appear when you’ve ignored your failed sync notifications one too many times. When you receive this notification, it’s usually a sign that you need to debug or disable the connector.

Conclusion

Choosing a tool that follows best practices is necessary for creating a scalable data environment. You don’t want to look back years from now and wish you chose a tool that allowed you to build out your data stack the right way.

When it comes to data ingestion, Airbyte follows all of the software engineering best practices like version control, Infrastructure-as-Code (IaC), deduplication, and schema detection to ensure you are producing the highest quality data.

The right tool will help reduce the number of errors, resolve issues faster, and most importantly, allow you to build something that will last for years to come with minimal issues.

Limitless data movement with free Alpha and Beta connectors

Introducing: our Free Connector Program ->

The data movement infrastructure for the modern data teams.

Try a 14-day free trial

About the Author

Madison Schott is an Analytics Engineer, author of The ABCs of Analytics Engineering book. Madison blogs about the modern data stack on Medium and writes a newsletter on Substack.