Article

AI agents and workflows: a starter kit

•

10 min read

Summarize with ChatGPT

Broadly speaking,”Agents” are the future of getting things done with our new magical AI APIs.The definition varies, but I think the folks at Anthropic did a great job on the blog post Building effective agents.

Here are the key concepts from my experience and that blog post:

Context: Extra information given along with the user prompt to help an LLM respond effectively.
Tool: A code function that takes input (often json) and generates output (also often json). The results of a tool are often used as context.

Workflow: a flowchart of decisions and tool calls to answer a prompt from the user. It commonly will have steps to determine which tools to use and evaluate the results.
Agent: an LLM (like OpenAI models) is given a prompt from the user and a set of tool definitions and decides how to use them to answer the prompt.

Examples of tools agents or workflows might use.

This agent has access to several tools and uses them to answer the user’s question. This allows it to answer a large breadth of questions.

This is a more explicit implementation of what an agent does, but as an explicit workflow. Instead of letting the LLM decide in a black box, you can go step by step. This is useful when there is a well-known use case as you’ll see below.

That ends up with an interaction diagram more like this:

Application

I have created a github repo to implement each of these strategies for a specific use case: chatting with the data I have in Salesforce. I have synced fake Salesforce models (Account, Contact, Opportunity) to a duckdb database. This also includes a “document” of all relevant data and well as the vector embeddings of these documents. From a UI provided by Chainlit, I can chat with either a workflow or an agent implementation that knows how to search this database to provide RAG (Retrieval Augmented Generation) context.

I have also heavily instrumented it to show the steps it took in the UI. In this case, I was using the workflow (Flowchart class) approach and took these steps:

Pick approach and decided database
Talk to an LLM and have it write the query
Execute the query using a tool
Use the database data to create a chat response from the LLM.

Switching the approach

We can switch to using an agent (ToolRouter class) in chat.py by making sure that is the one that is uncommented.

When this is the case, the experience is different.

In the agent case, it took different steps:

The tools and prompt were given to the LLM and it responded with a query
We ran this query on the database and gave the LLM back the response
It used the results to give an answer.

Which is better?

My implementation of the agent (ToolRouter) is basically complete because the pattern is very simple and the LLM does all of the hard work. My workflow (Flowchart) has plenty of room for improvement. For example, a “judge” component might be useful or otherwise have a large circular loop in the flowchart. The chat interface is also not quite as predictable as more concrete use cases.

All that being said: which works better in my app? The short answer is that the agent works better. It simply has more flexibility to respond to complex questions and iterate on the results.

Agent Example

This works out really well because the LLM agent decides to call `Tool.query_database` three times in parallel to get the count of contacts: once where they are in the US, EU, and in other countries. Note how the LLM was even smart enough to use its training data to know that “EU” means the set of countries that include Germany, Italy, etc.

Workflow Example

The workflow falls short because it doesn't have that parallelism or loop behavior built in. I needed to do a better job predicting that use case.

Implementation

There are few interesting parts of the application worth talking about.

Agentic Framework

I decided to use LlamaIndex `Workflow` objects to create both types of agents. I liked how it had a declarative approach. It does this based on the type of event object that `@step` methods accept and return.

For example, here is ToolRouter `Workflow`:

Let’s assume they all have access to the history of messages.

`handle_initial_event` goes first and returns a signal to chat.

`handle_llm_input` responds when there it’s time to chat with the LLM. This might output either a tool call (if the LLM wants to call a tool) or a stop signal (if the LLM is done).
`handle_tool_calls` will run the tools as necessary and signal back to the LLM to chat.

Note that there is a loop until the LLM is satisfied.

The Flowchart `Workflow` was built the same way but is more complicated.

Note the cycles in the tool calls. There is an error. Depending on the error, the code might try again.

Evaluations

I am using the deepeval library (and specifically the GEval metric) to run my two agentic approaches through a few tests.

This compares the actual results to the expected results and uses an LLM to evaluate the differences. In our case, my tests actually fail on the EU question for the `Flowchart` workflow.

In our production code at Airbyte, we generate reports with these kinds of evaluations to understand how the changes we are making to the code affect results.

Embeddings

An important part of the semantic search ability are the embeddings used to search the documents. The database stores vectors (lots of numbers). Getting these numbers is called calculating embeddings.

When is this done? It’s done as we store records in the database. There is a column with the vectors for each Salesforce record. It is also done on the search query when we are looking up documents to use as context.

A model is used to calculate embeddings. This sample app has a command to download a model from HuggingFace. It uses this model with the sentence_transformer library to calculate the embeddings.

Chat Interface

I am using the Chainlit chat library within FastAPI. It is very simple to get a full interface and socket-based API up and running for a single node.

The implementation in chat.py is also relatively straightforward.

Data Syncing and Use

For this sample project, I created a duckdb database of sample Salesforce data. At Airbyte, we have spent most of our time thinking about how to most effectively sync Salesforce and many other sources both completely as well as incrementally over time.

Recently, we have been working on a project to help organizations put their data to use in a way similar to this sample app. We have a platform that can be the full backend for your AI applications. It includes: