Accelerating AgriDigital's Supply Chains with Airbyte
Agridigital, a digital supply chain platform, massively sped up their release cycle with Airbyte.
AgriDigital, based out of Australia, is a cloud-hosted multi-tenant supply chain platform that transforms the way farmers, buyers, and consumers perform digital commodity transactions. It is among the world's first companies to use blockchain to build digital trust across agriculture supply chains — empowering a network of close to 8000 customers to effortlessly stay on-top-of contracts, prices, deliveries, orders, payments, and invoices at scale.
With billions of dollars worth of transactions annually flowing across the AgriDigital platform, AgriDigital quickly established itself as a reliable broker, inventory management system, and data analytics provider focusing on three initial business problems -
Without a platform like AgriDigital, farmers would typically not get paid for their commodities until months after delivery. Additionally, the lack of trust and financial liquidity in the supply chain made access to finance a challenge for buyers. If people need to get paid on time, quick invoice and payment processing are critical.
With the cost of food security becoming a global challenge, AgriDigital promised more transparency for consumers. By leveraging digitally verifiable smart contracts instead of paper trails, commodity ownership could be more accurately tracked, and counterfeit goods could be quickly identified and eliminated from the supply chain. But, of course, this had to work across the billions of transactions running across that AgriDigital platform.
Typically, individuals and organizations along supply chains lack trust and the incentive to globally share data such as the status of goods and open market prices. AgriDigital wanted to provide 360-degree view analytics to boost cooperation and trust among participants, enabling customers to make informed decisions whenever they wanted.
AgriDigital’s old architecture leveraged a mix of different components that made it complex, hard to manage, and scale.
With an increased demand in AgriDigital’s disruptive technology, demand for the platform surged, which meant more transactions needed to be completed in lesser time. And, even though the platform was built with a microservice architecture in mind from the get-go, it had several key challenges that needed to be addressed -
As more and more features were requested by AgriDigital’s customers, adding newer data tables and attributes became a regular task for every code sprint.
With a centralized global cache for important lookup data such as currency and pricing information, existing scripts had to be carefully handcrafted to ensure that new fields were reflected in the cache. Additionally, tighter coupling among the different microservices added another layer of architectural complexity — as one microservice had to go through another microservice to fetch data. Even a tiny change in one of the microservices would result in modifications to other dependent microservices.
“Synchronizing data from one data database to another was ridiculously hard, and especially when you've got it hosted on AWS, which doesn't help with doing synchronizations nicely. The net result was extremely long cycles for development and deployment that took several months to complete and was costly to maintain.”
As the platform’s popularity grew, it became hard to anticipate the number of incoming service requests and determine how big or small the cache should be sized to service the requests without failing. Without appropriate cache and network sizing, the constant cache API calls would easily choke the network, causing bottlenecks. Additionally, the microservices would need to manage the sync and transformation of data themselves, putting additional work pressure on the servers instead of useful request processing.
“The fact that every microservice needed to make an HTTP request call created a performance issue – sending a lookup request and receiving the massive payload back just wasn't feasible – we needed something more scalable and reliable.”
AgriDigital’s data architecture was highly siloed, and it became tough to understand where all the data was. Since several microservices were fighting for reference cache data, hot aggregated data was not always available in the cache when it was most needed. As a result, there was extra load on the underlying database, resulting in longer response times. In addition, with multiple tenants at play, it became challenging to provide faster time to insight for each tenant and run global reporting across all the tenants.
“With Airbyte, the release cycle time shrunk from over multiple months to only a few days.”
AgriDigital operates in a highly regulated industry where compliance is critical. To satisfy the regulatory compliance requirements, it is vital to ensure that data is strictly contained within the geographical service boundary. As a result, the business risk was elevated if AgriDigital leveraged any third-party services that were proprietary, un-certified, or running outside the allowed geo boundaries.
AgriDigital had several issues with their old architecture. So how did they solve it?
After exploring the possibility of building their own DBT-style custom ETL tool, they stumbled across Airbyte in a blog post and figured that it could move data from A to B with just a click, so they decided to give it a shot. Re-platforming the architecture using Airbyte made it possible to solve the caching situation, the tight coupling situation, and the performance issues. All in one.
Here are some of the results -
By removing tight coupling between different microservices and eliminating the centralized reference cache, development cycles became more efficient, and platform debugging was streamlined. In addition, to support growing customer needs, adding new platform features with additional tables and attributes became more manageable than ever — and because Airbyte automatically managed schema changes and syncs, rebuilding and redeploying microservices was not needed.
“Airbyte is a critical tool for our company because we use it in the operations of our core platform to move data from point A to point B. Airbyte made it possible to release and fund our microservices independently, shortening the release cycle time from multiple months to only a few days.”
With the new architecture in place, the performance of core platform operations increased — microservice code footprints became lighter, pushing the complex sync and transformation logic to Airbyte. Due to the lightweight design, it was now possible to spin up and scale out to many more microservices than before, without saturating server resources, including the network. For example, we've now eliminated about 200 lines of code that managed the synchronization and transformation in our services. So that’s 200 fewer lines of code to manage.
“We had to synchronize data from one data database to the other, and it was ridiculously hard, especially since our solution was hosted on AWS. However, what would typically take us more than 30 minutes on average was complete in just a few minutes using Airbyte, and maintaining data freshness was a matter of simply adjusting the sync frequency.”
As the new platform architecture rolled out, slow analytical dashboards were a thing of the past. With Airbyte, all the local microservice caches were closely synced — near real-time data was cached at every microservice, and fresher data flowed in whenever the syncs were triggered.
“We regularly go through all of our single-tenant customer data and consolidate it into a single data lake to generate global reports. Using Airbyte, there isn’t a single connector that we miss — everything from all the ads to marketing, CRM, stripe, payments, everything — we now run straight through Airbyte into our data lake. Airbyte solves this problem instantaneously and is the glue that pins everything together.”
Agridigital was dealing with commodity and asset financial information across jurisdictions. In addition, they had a customer promise to keep with their end-user data agreements. Airbyte made it easy to meet these constraints since it is an open source software that is easily auditable. Moreover, Airbyte could operate entirely within the trust boundaries of the platform.
Are you ready to explore the possibilities that Airbyte brings to the table? Share your success story with us or join the community.
Get all your ELT data pipelines running in minutes with Airbyte.
