Agentic Data Engineering Resources

Resource

What Are Parallel Tool Calls in LLMs?

Parallel tool calls cut AI agent latency by up to 3.7x. How they work, when to use them, OpenAI and LangChain implementation, and production patterns.

Pedro Lopez

March 6, 2026

Last updated September 8, 2026

Summarize with AI:

Parallel tool calling is a pattern where an LLM identifies independent operations, requests them all in a single response, and your infrastructure executes those calls concurrently. Instead of waiting for each tool to finish before starting the next, independent calls run at the same time. Total latency drops from the sum of every tool call to the duration of the slowest one.

This matters as soon as your agent pulls from more than one or two sources. An agent that needs a customer record from Salesforce, recent orders from Snowflake, and open tickets from Zendesk can either query each one in sequence, forcing the user to wait for all three round trips, or request all three at once and get everything back in a single batch.

TL;DR

In many agents, especially those making API and database calls, the dominant latency comes from input/output (I/O) rather than LLM inference. Parallel tool calls let the model request multiple external functions simultaneously, typically reducing total latency to the slowest single tool plus the inference cycles needed for planning and synthesis.
Benchmarks like LLMCompiler show roughly 1.4x to 2.4x latency speedups on many tasks, with some scenarios reaching up to 3.7x. You'll use more tokens per inference step, so plan for a cost-vs-latency tradeoff.
Provider support varies widely. See the provider comparison table in the implementation section for current behavior.
Airbyte Agents handles per-source rate limiting, token renewal, and schema mapping independently per connection through 700+ agent connectors, so concurrent calls don't break each other.

How Do Parallel Tool Calls Work?

In a standard tool-calling loop, the agent follows a strict sequence: prompt, tool request, execute, return results, repeat. Total latency is the sum of all tool execution times plus an LLM inference cycle for each step.

With parallel tool calling, the model analyzes your prompt and identifies which tools can run simultaneously because they don't depend on each other's outputs. It returns multiple tool calls in a single response. Your orchestration layer runs these concurrently, and all results come back in one batch for the model to synthesize.

One important distinction: the model requests parallel execution, but your framework determines whether the calls actually run concurrently. The code example in the implementation section shows exactly where this matters and why it's the most common source of missed speedups.

When Should You Use Sequential vs. Parallel Execution?

Use Sequential When	Use Parallel When
Output of one tool feeds into another as input	Tools have no dependencies on each other
Order matters for correctness	Operations are I/O-bound
Maintaining state across dependent operations is essential	Multiple independent data sources need querying
Debugging and tracing are higher priorities than raw performance	Latency reduction is critical for user experience
Tools modify shared state (prevents concurrent access conflicts)	Information from diverse APIs must be aggregated

If your agent queries a CRM system, a data warehouse, and a ticketing system to build context for a response, those lookups can run simultaneously because they don't depend on each other's outputs. For tools that modify state, enforce sequential execution to prevent concurrent access conflicts.

Two caveats are worth noting. If your bottleneck is model reasoning rather than I/O wait, parallelizing tool calls won't help. You'll pay the same inference cost regardless of how you schedule the tools. And if multiple tools hit the same rate-limited API, concurrent calls can trigger throttling and make total latency worse, not better. Profile your actual tool execution times before committing to a parallel-by-default architecture.

The hybrid approach often works best in production. Fast, independent tools execute in parallel while slow operations run separately or get cached. You might fetch user profile, recent orders, and support history in parallel during the initial phase, then use those results sequentially to determine next steps.

What Performance Improvements Can You Expect?

The LLMCompiler system demonstrated up to 3.7x faster execution on specific benchmarks, with many tasks showing speedups in the 1.4x to 2.4x range.

Suppose three tools each take about 200ms:

Metric	Sequential	Parallel
Tool execution (3 tools × 200ms each)	600ms	200ms (max of three)
LLM overhead	3 × 500ms = 1,500ms	1 × 500ms = 500ms
Total latency	2,100ms	700ms
Speedup	--	3x

In practice, the gains depend on how uniform your tool latencies are. If tools take 200ms, 200ms, and 500ms respectively, the parallel batch latency is 500ms, dominated by the slowest tool. You might get better results grouping the two faster tools together and handling the slow one separately.

You'll use more tokens when the model processes multiple tool results at once. In some architectures like LLMCompiler, better planning and fewer reasoning steps can partially offset this overhead or even reduce overall cost relative to naive sequential baselines. If you're scaling agents in production, that token overhead compounds. Monitor token usage across parallel operations and consider routing tool-result synthesis to a smaller, cheaper model while keeping your primary model for planning and orchestration.

What Are Production Use Cases for Parallel Tool Calls?

Enterprise search is one of the clearest wins. Support teams need information from internal knowledge bases, product-specific documentation, customer configuration history, and compliance guidelines. An agent using parallel tool calls queries all these sources at once and delivers troubleshooting guidance specific to the customer and product configuration, instead of making the support engineer wait for four sequential lookups.

Development tools use the same pattern for context gathering. When you ask for code suggestions, the agent can simultaneously search your codebase for relevant implementation patterns, retrieve related test files, query API documentation, and analyze dependency graphs for compatibility. You get suggestions fast enough to stay in flow instead of waiting and context-switching.

Customer-facing agents see similar benefits. When a user opens a support chat, the agent simultaneously pulls account status from HubSpot, checks inventory levels in the warehouse, looks up shipping tracking from the logistics API, and retrieves relevant product docs. Pulling order, inventory, and shipping context together in one pass is what lets an e-commerce order dispute agent settle a delivery or refund question without bouncing the customer between teams. Without parallel execution, that context-gathering phase adds seconds of visible latency before the agent responds. With it, the user sees a single wait no longer than the slowest backend query.

Try Airbyte to connect your AI agent to production data. Or explore the agent connector repo on GitHub.

How Do You Implement Parallel Tool Calls?

LLM Provider Support

Provider support varies, and capabilities evolve quickly. Confirm the latest behavior in each provider's official API docs for your specific endpoint.

Provider	Parallel Support	Configuration
OpenAI (GPT-4o)	Yes, on supported models/endpoints	Set parallel_tool_calls=True; not all models accept this parameter
Google Gemini	Yes, automatic	Returns multiple tool calls when appropriate
Cohere	Yes, automatic	Returns multiple tool calls when appropriate
Anthropic Claude	Provider-managed loop	Does not expose a parallel flag; design assuming one tool call at a time

Framework Support for Concurrent Execution

Framework	Approach	Best For
LangChain / LangGraph	RunnableParallel within LangChain Expression Language; stateful graph with concurrent processing when no dependencies exist	Explicit developer control over parallel composition; multi-agent workflows
LlamaIndex Workflows	@step decorator with automatic dependency analysis via directed acyclic graphs	Complex workflows where manual parallelization is error-prone
Microsoft AutoGen	Multi-agent conversation patterns with concurrent execution	Teams of AI agents collaborating on complex tasks
CrewAI	Parallel role-based "crew" architectures	Tasks requiring multiple specialist agents

All four frameworks support concurrent execution when dependencies permit, though the specifics vary by framework and provider adapter.

Basic Implementation Example

<pre><code>import asyncio

import openai

# Define independent tools

tools = [

    {&quot;type&quot;: &quot;function&quot;, &quot;function&quot;: {&quot;name&quot;: &quot;get_customer&quot;, ...}},

    {&quot;type&quot;: &quot;function&quot;, &quot;function&quot;: {&quot;name&quot;: &quot;get_orders&quot;, ...}},

    {&quot;type&quot;: &quot;function&quot;, &quot;function&quot;: {&quot;name&quot;: &quot;get_tickets&quot;, ...}},

]

response = client.chat.completions.create(

    model=&quot;gpt-4o&quot;,

    messages=messages,

    tools=tools,

    parallel_tool_calls=True  # Enable parallel requests

)

# Model returns multiple tool calls in one response

tool_calls = response.choices[0].message.tool_calls

# Execute concurrently -- this is where the real speedup happens

async def run_tools(tool_calls):

    tasks = [execute_tool(tc) for tc in tool_calls]

    return await asyncio.gather(*tasks)

results = asyncio.run(run_tools(tool_calls))</code></pre>

The model decides which tools can run simultaneously, but your code must actually run them concurrently. Without asyncio.gather (or equivalent concurrency), you get the same latency as sequential execution regardless of what the model requested. This is the most common mistake in parallel tool call implementations: the model returns parallel requests, the developer processes them in a for-loop, and the expected speedup never shows up.

How Do You Take Parallel Tool Calls to Production?

Session Isolation for Concurrent Tools

Concurrent tool calls introduce a class of bugs you don't see in sequential execution: credential bleed. If two tools share a security context, one tool's authentication token can leak into another's request, especially when connection pooling or shared HTTP clients are involved. Give each invocation its own scoped, short-lived credentials. This way a bug or compromise in one tool path can't access another tool's data.

For broader patterns like least-privilege service accounts and dynamic authorization, see the full agent security guide.

Monitoring Concurrent Execution

OpenTelemetry provides a widely adopted instrumentation layer for distributed tracing. Each concurrent tool invocation receives its own span with attributes identifying tool type, parameters, and execution context. Links connect related spans so you can trace relationships between concurrent tool calls and identify which tool dominated batch latency.

Track these metrics across parallel tool execution:

Metric	Description	Alert Threshold
batch_latency_ms	Total time from dispatch to last tool completion	> 2x median of slowest individual tool
slowest_tool_ms	Execution time of the longest-running tool in a batch	> p95 of that tool's historical latency
tool_success_rate	Percentage of tools completing without error per batch	< 95% over a 5-minute window
concurrent_execution_ratio	Actual parallel execution vs. sequential baseline	< 0.5 (tools are serializing)
token_overhead_ratio	Tokens used in parallel batch vs. equivalent sequential calls	> 1.5x sequential baseline

For observability platforms, tools like Phoenix and LangWatch trace agent workflows from prompt to tool execution to response.

Error Handling and Partial Failures

When one tool in a parallel batch fails, you can either fail the entire batch, return partial results, or retry only the failed tool while caching results from tools that succeeded. The retry approach works best in most cases because you don't waste successful work and the model gets explicit context about what's missing.

Set up per-tool circuit breakers with exponential backoff that track failure rates independently. A circuit breaker monitors consecutive failures for a given tool and temporarily stops calling it once a threshold is reached, preventing a single flaky API from cascading into your entire parallel pipeline. Use a degraded state so your agent continues with reduced functionality. Answering from three out of four data sources is better than returning nothing. Tell the model explicitly when a source was unavailable through the system prompt so it can qualify its answer rather than hallucinate to fill the gap.

What's the Right Way to Build Parallel Tool Architectures for Production?

Most teams get the orchestration layer working quickly. LangGraph or LlamaIndex handle the parallel dispatch and dependency resolution well. The problem shows up one layer down, at the data infrastructure. When three tools fire simultaneously, they're all waiting on the same database connection pool, competing for the same API rate limit, or breaking because a source changed its OAuth flow or response schema last week. Orchestration frameworks don't manage those concerns for you.

Airbyte Agents handles rate limiting against each specific source API through 700+ pre-built agent connectors, so three concurrent tool calls to the same Salesforce instance don't blow past API quotas. OAuth token renewal and refresh happen independently per connection. Each agent connector manages its own schema mapping, so your agent code doesn't break when a source API ships a new response format. For teams that need faster context retrieval before those tool calls even run, Context Store fits naturally into the same architecture. It gives agents a structured way to find and assemble business context across systems before synthesis, which complements parallel execution when latency and context quality both matter. Get a demo to see how Airbyte Agents supports production-ready parallel tool architectures, or try Airbyte Agents today.

Frequently Asked Questions

How do you prevent agents from executing dependent tools in parallel?

Write clear tool descriptions that explicitly note dependencies, like "REQUIRES: Output from fetch_article tool as input." At the orchestration layer, deploy explicit dependency validation through dependency graphs before executing parallel tool calls. Some frameworks like LlamaIndex's LLMCompiler automatically analyze dependencies, while others require manual specification.

How do you handle partial failures in parallel tool batches?

Retry the failed tool while caching successful results. Don't discard work that already completed. The critical step most teams miss is telling the model which source failed. Include the unavailable source name and error type in the tool result so the model can qualify its response instead of guessing.

Can you use parallel tool calls with open-source models?

Yes, but parallel execution depends more on your orchestration framework than the model itself. Models like Llama 3 and Mistral support tool calling, but actual concurrent execution requires framework-level orchestration through LangGraph, LlamaIndex, or AutoGen.

How do you manage rate limits across concurrent tool calls?

Use per-source token buckets that coordinate across concurrent invocations so three parallel calls to the same Salesforce instance share one rate limit pool. If you're managing connections through Airbyte's agent connectors, each connection handles its own API rate limiting without additional coordination code.

How do you estimate the cost impact of parallel tool calls?

Parallel execution uses more tokens per inference step because the model processes multiple tool results at once. Monitor token_overhead_ratio (parallel tokens vs. sequential baseline) and consider routing tool-result synthesis to a smaller model while keeping your primary model for planning.