About Jins
Jins Kadwood is the Chief Technology Officer at AgriDigital. He is an experienced, successful technology entrepreneur and Executive Product and Technology leader. Awarded Australian Fintech CIO/CTO of the year by Fintech Australia in 2018, Jins brings together behavioral sciences to bridge the gap between technology and business to build high performing cross-functional business culture.
Jins gave us a detailed look at how his team migrated their database and systems without disrupting their clients. As the Agridigital business grew, the old technical architecture couldn't handle the growing number of transactions fast enough. Jins and his team discovered Airbyte as a solution to restructure their problematic architecture, resolve performance issues, and meet their industry’s strict regulatory compliance requirements.
About AgriDigital
AgriDigital is a cloud-hosted multi-tenant supply chain platform that transforms the way farmers stay on top of contracts, prices, deliveries, orders, payments, and invoices at scale. Today, with more than 8000 clients, we are Australia’s fastest-growing agricultural supply chain innovator. In addition, we are one of the world’s first companies to use blockchain to build digital trust across agriculture supply chains.
What was the problem?
As we grew rapidly, various components in our old architecture made it hard to manage and scale. Even though our platform was built with a microservice architecture in mind from the get-go, several key challenges needed to be addressed:
Complex and interconnected data architecture
For our business, it was getting difficult to handle the increasing amount of data synchronization. With a centralized global cache for important lookup data such as currency and pricing information, existing scripts had to be carefully handcrafted to ensure that new fields were reflected in the cache. In addition, with one microservice flowing into another microservice to fetch data, even a small change in one of the microservices required modifications to other dependent microservices.
“Synchronizing data from one database to another was ridiculously hard, and especially when you've got it hosted on AWS, which doesn't help with doing synchronizations nicely. The net result was extremely long cycles for development and deployment that took several months to complete and was costly to maintain.”
Performance bottlenecks
Over time, it became hard to anticipate the number of incoming service requests and determine how big or small the cache should be sized to service the requests successfully. Without appropriate cache and network sizing, the constant cache API calls would easily choke the network, causing bottlenecks.
“The fact that every microservice needed to make an HTTP request call created a performance issue – sending a lookup request and receiving the massive payload back just wasn't feasible – we needed something more scalable and reliable.”
Slow Analytics
Our data architecture became highly siloed, and it was tough to understand where all the data was. In addition, there was an extra load on the underlying database, making it difficult to provide fast insights to each tenant and run global reporting across all the tenants.
Meeting regulatory compliance needs
Our platform operates in a highly regulated industry where compliance is critical. We had to ensure that our data remained within the geographical service boundaries, and we were concerned about using uncertified or risky third-party services.
How did we discover Airbyte?
At first, our team explored the possibility of building our own dbt-style custom ETL tool. However, we were deliberating over the best solution, concerned about potentially interrupting our extensive network of supply chains. Fortunately, I stumbled across Airbyte in a blog post. Our team was impressed by Airbyte’s ability to move data from one database to another quickly, and we decided to utilize it to resolve our growing issues.
“We had to synchronize data from one database to the other, and it was ridiculously hard, especially since our solution was hosted on AWS. However, what would typically take us more than 30 minutes on average was complete in just a few minutes using Airbyte, and maintaining data freshness was a matter of simply adjusting the sync frequency.”
How did Airbyte solve our problem?
Simplified architecture and shorter release cycles
Our new architecture removed the tight coupling between different microservices and eliminated the centralized reference cache. This streamlined our platform debugging and made development cycles more efficient. Moreover, it became more convenient than ever to support growing customer needs. Our team could add new platform features without rebuilding and redeploying microservices because Airbyte automatically managed schema changes and syncs.
“Airbyte is a critical tool for our company because we use it in the operations of our core platform to move data from point A to point B. In addition, Airbyte made it possible to release and fund our microservices independently, shortening the release cycle time from multiple months to only a few days.”
Enhanced performance
With the new architecture in place, the performance of core platform operations increased. In addition, our microservice code footprints became lighter by pushing the complex sync and transformation logic to Airbyte. For example, we eliminated about 200 lines that managed the synchronization and transformation in our services. With the lightweight design, we can now scale out to many more microservices than before without saturating our server resources.
Richer and fresher global analytical dashboards
All the local microservice caches are closely synced with Airbyte. Near real-time data is cached at every microservice, and new data flows in whenever the syncs are triggered.
“We regularly go through all of our single-tenant customer data and consolidate it into a single data lake to generate global reports. Using Airbyte, there isn’t a single connector that we miss — everything from all the ads to marketing, CRM, stripe, payments, everything — we now run straight through Airbyte into our data lake. Airbyte solves this problem instantaneously and is the glue that pins everything together.”
Meeting compliance with security regulations
We easily worked with all compliance and security constraints thanks to Airbyte’s open-source software. Moreover, Airbyte operates entirely within the trust boundaries of the platform itself, removing potential security risks for our company and network of clients.
How do we feel about using Airbyte?
Our team was able to speed up its release cycle with Airbyte massively. With our new architecture in place, we can devote our energy and resources to our main purpose: digitizing and growing grain supply chains. We are sure that Airbyte can help us in our journey to enable all participants on our platform to operate with confidence.
