Article

Not impressed with your AI experience? It’s not the model. It’s the data.

•

October 23, 2024

•

7 min read

At Airbyte, we spend most of our time working with data. We built a platform and connectors that move data from sources to destinations. There are a hundred thousand deployments by data engineers powering this ELT process to and from S3, Postgres, Snowflake, Salesforce, and friends. We provide data infrastructure.

Gen AI use cases are proliferating. We are in the middle of a technical revolution and, like previous technical revolutions (transistors, computers, Internet, mobile) it will be applied to everything. We know data is needed for this and want to make Airbyte great at it. So we have been talking to lots of people about what they are building and how they are building it.

An AI product journey

Some of the people we talked to are using Airbyte, but we cast a wide net in our discovery process, so most are not. The majority are product engineers building something that uses generative AI. They are focused on making magical customer experiences. Though they are using data, they are not thinking about data infrastructure.

For example, we talked to engineers in a retail company powering better in-store experiences. The way these engineers think about it is from the user perspective. What are customers and associates asking? What are good answers? What is the data needed to serve them?

At this point in the journey, the typical product engineer goes looking for data. When someone asks about monitors, your app could search the product catalog API for what they typed in. Then you could use the results as part of the context (pasted into your prompt) for a smart LLM (large language model). The LLM then uses that as it formulates its answer.

This is generally called retrieval augmented generation (RAG). It’s a simple technique that end up supplementing the prompt to an LLM like this:

You are a helpful in-store assisant. Your #1 goal in life is to help
our employees and customers find what they need.

Some helpful context is:
{{ relevant_documents }}

Your task is to answer the question: 
{{ prompt }}

Please don’t make things up.

For example:

You are a helpful in-store assisant. Your #1 goal in life is to help
our employees and customers find what they need.

Some helpful context is:

## Big Green Egg
A simple yet highly-efficient blend of ancient tradition, modern
technology and unmatched quality..

## Benjamin Moore Paint
The highest-quality interior paint and colors provide excellent
coverage for your...


Your task is to answer the question: 
How can I make my home more green?

Please don’t make things up.

This can work and is very powerful. Your tool itself is essentially stateless. It pulls in what it needs in real-time. Everything you searched for worked out super well.

The problem is that you are very familiar with the catalog of products. You subconsciously knew the titles of the items you were searching. Of course your questions worked out. You are a power shopper at this store! But your customers are searching for “green” (eco-friendly) products and you are showing them green paint.

You search around a bit and learn about vector stores. These return results that are semantically near each other. THis will be better than the simple product API used used previously.

Now there is a new job to do. You have to fill up your vector store with data to provide relevant context to your LLM. So you might write some code that runs every night that copies over content from the product catalog into your vector store. The results improve.

But what if it’s still not good enough? People expect magic! You put the data in a tool made for that purpose. You are using that data. The LLM uses it as it sees fit. Maybe AI is not there yet. Maybe the model just isn’t smart enough.

It’s the data

We talked to 100 engineers about how they got over these kinds of hurdles. Across the board, it was not the model. It was the data. The data needed more work to be effectively queried. When the LLM had the right data, it could be magical.

Some example quotes on this pain point:

“It’s almost like you need to pre-process your data twice. Otherwise, it’s garbage in, garbage out.”
“Feeding our vector database clean data is our biggest challenge.”
“Our challenge is fetching appropriate data.”
“Data quality is our biggest pain point with our customers.”
“AI-powered models are built on the data we use. Finding the data is incredibly, incredibly difficult.”

We think about the data’s journey in the context of this pipeline. Here we are mainly talking about optimizing the “Process” step to provide great experiences.

An AI pipeline today.

Let’s walk through it.

What kind of pre-preprocessing might you do?

Enrichment: Add in the industries of the companies in your Salesforce opportunities so you ask about trends.
Summarization: For vector search over earning calls PDF, add a summary of the whole document, so every chunk knows what company it is about.
Use case optimization: This bucket is huge because the whole process is about optimizing for the use case, but in our retail app maybe we’d want to run our product data against a small model to normalize “teal” and “forest” to “green” to facilitate a filtered search.

Next, you would need to do the vector store logistics: chunking and embedding.

Chunking: what’s the best way to break up large documents into useful pieces that provide value independently from each other?
Embedding: what model is best used for this use case for calculating the actual vectors (lots of floating point numbers) to query.

We put both the structured and vector data in a queryable store.

Structured data supports two query patterns:some text
- SQL queries that are best for specific answers like “How many of product X did we sell last week?”
- Filtered vector search: “Only show me products that are in-stock and green.”
Vector search:some text
- Provides documents for the LLM context based on the semantics of the question.

All of this is to get the relevant_documents noted above so the LLM has the context it needs to answer the question well.

Finally, you need an “agent.” Let’s not be fancy. It’s really an “if statement” that determines which type of query to do based on what the user asked for and stuff them into the prompt.

Iteration loop

The most important thing is getting a system in place that allows you to iterate. This is hard stuff and it has an air of unpredictability. The best path through is to continuously modify the inputs and evaluate the outcomes. What is needed to make this happen?

The first thing that is required is an evaluation framework. This is a set of benchmarks you can run with every change to see how they impact the end result. The scorecards I have seen contain a lot of green and red boxes. These are relative values for if the metrics you care about improve or get worse with the most recent change.

Assuming that exists, you have to be able to run it (or a subset) after each relatively minor change to your data pre-processing code. The faster that can run and you still believe the results, the better. It won’t be fast enough if you have to go back to the original data source and, for example, call that API. This requirement of having a fast “playground” is one reason to have the intermediate storage (“Processed Data Destination”) in the architecture above. You iterate with that unchained foundation, testing out each change you think might help move your evaluation benchmarks.

Finally, you need to hold yourself accountable to reality. This takes a few forms. Let’s use a customer support tool that helps agents answer tickets as an example.

Make sure your benchmarks reflect how users are actually using your product. Are they getting the same quality results? For example, are the answers coming back from real tickets consistently calling the person by the right name like your benchmark tried to ensure?
We also need to know if people think your tool is valuable. In the support example, you might track how often they change the suggestion, what percent changes, and even store the diffs to learn from.
Finally, you want to know if it actually moved business outcomes. If it’s a shopping assistant, do people that use it actually buy more? In the support case, is the velocity of the team going up and, because you should always pair your metrics, is customer satisfaction neutral to positive?

Closing

This is what we are seeing in the field today. AI product engineers are trying to create great experiences and coming up against a need for better data. The mental leap is to realize achieving better data is not a new frontier unlike the LLM model itself. You have the opportunity to build on the best practices that have evolved over the last 30 years in the ETL (extract/transform/load) space.

‍

Looking to speak with our AI team about what you're building with AI? Just fill out this form.

Limitless data movement with free Alpha and Beta connectors

Introducing: our Free Connector Program ->

The data movement infrastructure for the modern data teams.

Try a 14-day free trial