To continue our community appreciation blog series, we're bringing you our next ink-credible data person: Daniel Diamond, an Airbyte open source contributor based out of Melbourne Australia, who is currently leading a multidisciplinary data platform team at 99designs and fostering a data-driven culture across the organization. Keep reading for some tips on submitting your first PR, a few book recommendations and a pic of his favorite watch.
I try to keep up with the ever-evolving data space and in particular, the open source community. I first heard about Airbyte through the dbt slack community at the end of 2020. This was roughly the same time that I launched a large-scale initiative to integrate the data platform from one acquired company into another. We needed a DMS alternative, we needed connectors for 3rd party data sources and we had to move a lot of data. There were various ELT tools out there, however Airbyte’s open-source strategy, long-term vision and particularly their interest in the CDC approach was sufficient to consider.
Airbyte leverages debezium under the hood, which means you can implement CDC incremental syncing capabilities. A far more efficient approach than the standard approach.
Specifically relating to Airbyte, I’ve helped extend the Octavia CLI project, which allows users to leverage IaC when working with Airbyte. Instead of updating your Airbyte connections manually via the UI, users can now import all existing sources, destinations and connections and update them programmatically. Users can deploy version-controlled yaml definitions of their Airbyte instances through CI/CD.
Probably #good-reads. The data space has been evolving rapidly and so having a tether to the latest and greatest ideas or projects helps one feel like it’s not entirely impossible to keep up.
Contributing to Airbyte has been a win-win strategy. As I run into additional requirements or desirable features, I have the freedom to explore and extend the product and have it reviewed by the team.
Open source fosters a community of like-minded and passionate individuals. For the individual, you benefit from exposure to other ways of thinking and coding styles. For the product, you benefit from continuous engagement and user feedback from a diverse set of individuals not to mention the actual code contributions.
Keep your contributions small in scope and get your feet wet. Don’t worry about breaking things, you should feel comfortable to rely on Airbyte’s process: PR review, CICD and testing frameworks.
The Modern Data Stack has turned into quite a controversial topic. Every day more tools arise whilst adjacent to this, enterprise players (Snowflake, Databricks etc.) are extending their capabilities e.g. snowflake x salesforce. At some point I anticipate consolidation and I’m curious if 3rd party applications will get involved. Related, we still haven’t unlocked source application-led event-streaming capabilities across the data space.
Autumn. The water’s warm, the days aren’t 40C and the streets are beautiful.
It’s mostly been work-related book clubs and thus been pretty technical/managerial. Turn The Ship Around is a great one if you’re interested in building leaders. A more recent read was American Prometheus, a thorough biography into Oppenheimer’s life. Highly recommended if you’re interested in this complex individual or the making of the atomic bomb. Also a timely recommendation with the upcoming Nolan film 🎥.
The halios tropik B rounds out the collection nicely. Made by an awesome microbrand out of Canada, this clean automatic bronze diver watch develops a beautiful patina over time. Or you can rogue, pull out the ammonia and force patina the thing.
If someone has helped you in the community, we want to hear about it. Ping me in slack @Karen Bajza-Terlouw.
🌊 you out there!
Get all your ELT data pipelines running in minutes with Airbyte.