Data Bytes Recap: A 5-step Checklist on How to Get an AI Project into Production

A 5-step checklist for taking an AI project from prototype to production, recapping practical lessons from the Data Bytes session.

COMMUNITY

November 1, 2024

3 min read

Quinton Wall

Summarize with AI:

Data Bytes Recap: A 5-step Checklist on How to Get an AI Project into Production

Last Wednesday, we hosted our monthly Data Bytes meetup in our San Francisco Office. The topic of the event was lessons learned which building AI solutions. We co-presented this with the team from Fractional.ai. Fractional.ai is the implementation partner whom we partnered with help build the new AI Assistant. If you missed the event, you can check out the recording online.

We kicked off with a presentation from Natik, one of our Engineering managers here at Airbyte. Natik shared an overview of Airbyte and how we are leverage AI to help data engineers create connectors to continue to expand the library available to our community.

Natik shared his experience on how Airbyte builds processes to scale the number of connections, how to leverage open source, and how to think about data pipelines that deliver a flexible approach, whether you want to write code, point and click, or new use AI to generate data streams. And, most importantly, he gave some insights into our philosophy to rapidly build & ship features and work with the community to continuously improve our offerings.

During the rest of the event, Eddie, CTO of Fractional.ai, shared insights on how to bring LLMs to production. One of the best things that Data Bytes, as an event, offers attendees is hearing real-world experiences from developers, companies, and founders who have learned by doing. Fractional.ai has worked on over 100 projects bringing AI solutions to production.

“A lot of the best opportunities in AI, exist in manual processes”.

‍

This simple statement, based on the prompt “Where should I be implementing AI?” was a core tenant on Airbyte’s own approach to creating the AI Assistant. Natik’s team spends a significant amount of time in supporting and scale connectors for the community. These processes, wrapped in modern software engineering practices, have enabled his team is always the need to look for designs and tools to improve their processes. AI was a perfect fit to take it to the next level.

So how does an AI project get into production? Fractional.ai provided a 5 step checklist:

1. Solve a piece at a time.
- For example, the initial problem after crawling an API’s documentation was solve authentication. Once this was complete, solve the next step, such as pagination, and repeat.
2. Develop robust evaluations
- Prove results by clearly defining success metrics and start with clean test data to fully stress the solution. Fully evaluate the results and decide what success looks like.

‍

3. Iteratively improve from POC to production
- Each time we achieve benchmarks with test data, expand the data set to include more endpoints and APIs. And repeat.
4. Decide where and how a human will be in the loop
- For AI Assistant, we wanted a data engineer to be the co-pilot, rather than building a chatbot. LLMs will improve over time, but it was important to leverage the experience of a data engineer to validate and modify the generate connections and streams. In your POC success criteria, choose how much human involvement you want.
5. Launch quickly, with the right expectations
- We fully believe it is better to continuously ship and improve based on usage and community feedback. AI Assistant was no different. We set the expectations that it won’t get things right every time, but will save significant amounts of time already. What we have seen so far is that AI Assistant reduces the time to build a connector by 10x. That’s a win for everyone. Ship fast, and iterate. AI tooling in particular is evolving so rapidly, there may be new tools that were previously unavailable in your iteration. Experiment, learn, and ship.

After a lively Q&A discussion, it was time for pizza, conversations, and a wrap-up to another great event. There is something about sharing food and few drinks with likeminded folks that makes these community events so special. Thanks to everyone who attended, and subscribe to Data Bytes to be notified about upcoming events. And, if you haven’t tried AI Assistant yourself, deploy Airbyte or sign up for Airbyte Cloud to check it out.

‍

About the Author

Quinton Wall

Quinton is the Head of Developer Relations at Airbyte