Agentic Data Engineering Resources

Resource

How to Build Custom AI Agents from Scratch

Custom AI agents fail when data access is weak. Learn how to build reasoning, tools, memory, and permissioned data infrastructure.

June 8, 2026

Summarize with AI:

A custom AI agent that works on real enterprise data needs four layers working together: a reasoning engine, a tool interface, a memory system, and a data access layer. The first three get most of the attention, and data access is often treated as a footnote.

That approach fails once the agent reaches production Salesforce data and nobody built the authentication, normalization, or permission enforcement it needs.

TL;DR

Custom AI agents require four layers working together: a reasoning engine, a tool interface, a memory system, and a data access layer.
Most production failures come from enterprise data access problems, especially authentication, permission enforcement, data freshness, and cross-system entity resolution.
You can build from scratch or use a framework, but neither approach handles enterprise data connectivity, permission-aware retrieval, or data provenance tracking on its own.
The fastest path from demo to production is to build context engineering, access control, and fresh permissioned data into the system from day one.

What Are the Core Components of a Custom AI Agent?

Four layers make up a production agent, and each one carries distinct engineering decisions that determine whether the system holds up outside a demo. Each layer has a specific job, a set of tradeoffs, and a predictable failure mode when those tradeoffs are ignored.

Layer	What It Does	Key Engineering Decisions	Common Failure When Skipped
Reasoning engine (large language model / LLM)	Processes instructions, interprets context, generates responses and action plans	Model selection (cost vs. capability), temperature settings, token budget allocation	Agent produces infinite loops, terminates prematurely, or exhausts the context window mid-task
Tool interface	Connects the agent to external actions: application programming interface (API) calls, database queries, file operations	Tool documentation quality, input/output schemas, error handling and retry logic	Agent hallucinates tool calls or sends malformed requests because tool definitions are ambiguous
Memory and state	Maintains conversation history, task progress, and learned context across interactions	Short-term vs. long-term memory, persistence strategy, staleness management	Agent loses context mid-task, repeats work, or must restart after failures
Data access layer	Provides the agent with fresh, permissioned data from enterprise systems such as customer relationship management (CRM), ticketing, docs, and comms	Authentication flows, permission enforcement, data freshness service-level agreements, cross-system entity resolution	Agent works on demo data but fails on real customer data because it cannot authenticate to or normalize data from production SaaS tools

Anthropic frames a cross-layer principle simply: agents are stateful, and errors compound. When a tool failure slips past the state layer, it reaches the reasoning engine, which then sends bad queries to the data access layer.

Anthropic also recounts an agent with access to a full data warehouse that still failed before answering a production query because it lacked the context to determine which tables mattered and which datasets contained personally identifiable information (PII). Once that happens, the rest of the stack has to work around missing or unsafe context.

Should You Use a Framework or Build From Scratch?

The tradeoff is control versus speed. Building from scratch teaches the mechanics of the agent loop and gives the team direct control over debugging, retries, and observability. Using a framework can speed up prototyping, but it does not remove the need for a separate enterprise data layer.

Factor	Build from Scratch	Use a Framework
Best when	You need full control over the reasoning loop, have a narrow use case, or need to minimize dependencies	You need multi-agent orchestration, pre-built tool integrations, or rapid prototyping
Learning curve	Higher: you must implement the ReAct loop, tool parsing, and state management yourself	Lower: framework handles orchestration patterns, but debugging requires understanding internal abstractions
Data infrastructure support	Neither approach handles enterprise data access; you need a separate data layer regardless	Same constraint applies; frameworks abstract tool-calling but do not manage authentication, permissions, or data freshness
Production readiness	Full control over error handling, retries, and observability from day one	Frameworks provide structure but may introduce abstraction layers that complicate debugging in production
Maintenance burden	You own every component: updates, security patches, and schema changes are your responsibility	Framework updates may introduce breaking changes; you depend on community or vendor maintenance cycles
Recommended for	Teams with strong Python fundamentals building a single-purpose agent with specific performance requirements	Teams building multi-agent systems or needing rapid iteration across different agent patterns

If you build from scratch, you learn fundamentals that transfer across every framework. LLMs output text, and the "agent" is the Python code that interprets that text and decides what to do next. Frameworks that abstract this boundary can make debugging harder when things break in production.

Frameworks still provide value in some cases. Explicit graph models can make dependencies visible, and checkpointing supports long-running workflows. But adding more agents also adds more moving parts, more tool calls, and more chances for compounding failure.

How Do You Build the Agent Reasoning Loop?

The reasoning loop is the control flow that turns LLM output into actions the agent can take against real systems. Two decisions shape it: the pattern the agent uses to decide what to do next, and the way tools are described so the model calls them correctly.

ReAct Defines the Core Reasoning Loop

The ReAct (Reason + Act) pattern is a core reasoning primitive for custom agents. It interleaves reasoning traces with tool calls in a repeating cycle.

Thought → Action → Observation → [repeat until done]

Thought plans and tracks progress, Action calls external tools, and Observation captures results that ground the next reasoning step.

Production agents need guardrails on top of this loop. Explicit loop detection and iteration ceilings prevent infinite loops. Guardrails can exist at the input, output, and tool level, with a tripwire mechanism that halts the agent when triggered. Tool integration protocols such as the Model Context Protocol (MCP) describe how agents can discover and call external tools. Without those controls, the reasoning loop magnifies small mistakes into repeated bad actions.

Tool Documentation Shapes the Agent-Computer Interface

The Agent-Computer Interface (ACI) concept treats the interface between the LLM and its tools as an abstraction layer, analogous to a user interface for the model. A baseline agent without a well-tuned ACI performs worse.

Tool descriptions influence query shape, parameter selection, and what the model assumes each tool is for. Improving the tool description can correct behavior without any model change.

Under-specified tool descriptions cause malformed inputs, which degrades observations and corrupts later reasoning. If a human engineer cannot tell which tool to use, an AI agent will not do better. Tool definitions and conversation history also consume context budget before an agent reads a single request, which turns tool design into a context engineering problem rather than only a documentation problem.

What Does Context Engineering Mean for Agent Builders?

Context engineering governs what subset of available data enters the model's context window at each inference step. Prompt engineering is a one-time authoring act, while context engineering is a per-inference curation decision.

Careful context management matters in agent workflows because an agent running in a loop generates more data at each step that could matter to the next turn of inference. Teams need to refine that information cyclically rather than keep piling it up.

Data freshness is part of context engineering. Stable documentation can be pre-loaded in the system prompt. Semi-dynamic data like user preferences can tolerate higher staleness. Prices, inventory, and live status should be retrieved at inference time rather than served from stale context.

Permission-aware filtering also happens at this layer. Sensitive information should stay out of context unless the user is authorized to see it. This filtering happens before data reaches the model, not as a post-processing step after retrieval. If the wrong context gets into the window, every later safeguard is reacting after the exposure has already happened.

How Do You Connect Your Agent to Real Enterprise Data?

Connecting an agent to enterprise systems usually takes more engineering time than teams expect, especially when authentication, freshness, and entity resolution all have to work together.

Authentication Complexity Becomes a Maintenance Burden Across SaaS Tools

The initial OAuth handshake is only one part of the problem. The sustained engineering cost comes from operating token lifecycles across many providers at once because each provider has different token lifetimes, rotation policies, and scope formats.

No single refresh strategy works uniformly, so when tokens expire or sync pipelines lag, agents may fall back to stale cached or previously replicated data. That produces answers that are structurally correct but factually outdated, and the problem often stays invisible until someone notices the answer is wrong.

As the number of providers and tenant connections grows, OAuth maintenance becomes a standing engineering cost. Teams usually feel this after a handful of integrations, when connection upkeep starts taking more time than agent behavior itself.

Cross-System Entity Resolution Determines Whether Data Matches the Right Account

A single customer might appear under different names across systems because of legal names, support shorthand, billing records, or imported lead data. Without an entity resolution layer, an agent asked about an account's status must map IDs across systems, normalize field names, and handle partial presence.

Without this layer, the agent may return data from only one system, return duplicate records, or require internal IDs to answer a simple question. The reasoning loop is not the main problem in that case. The data model is.

Data Freshness Changes Whether Retrieved Context Is Still Correct

Model hallucination produces content untraceable to any source system, such as a fabricated legal citation or an invented API endpoint. Stale-data hallucination produces an answer traceable to a real source record that no longer reflects the current state.

That creates a harder problem because standard grounding verification checks whether the agent's claim appears in any retrieved document. A stale document can still pass that check if the claim appears in the document.

Change Data Capture (CDC) addresses this by capturing changes as they happen without impacting application performance. Batch extract, transform, load (ETL) on hourly or daily schedules was designed for a more forgiving consumer. Agents often are not.

How Do You Handle Permissions and Governance?

When an agent queries Salesforce, Jira, Slack, and Google Drive in a single session, it must respect per-user permissions across all four at once. An admin-level API token can accidentally expose confidential documents.

Frameworks that focus on orchestration typically do not provide full enterprise governance on their own. Teams using them usually end up building custom middleware for permissions.

Delegated Authorization Applies Row-Level and User-Level Access Controls

Delegated authorization is the core pattern. Each downstream system receives a separately scoped token derived from the user's identity, not a shared admin token. Effective access is constrained by both the agent's permissions and the user's permissions, with enforcement applied across systems.

Audit Trails and Compliance Readiness Determine Whether You Can Deploy Safely

Compliance readiness requires complete, tamper-evident logs of who accessed what data, when, and how. For AI agents specifically, that means which agent made the call, what parameters were passed, and what returned.

Under NIST IR 8587, authorization decisions for AI agents require fine-grained access control and additional contextual signals, aligning with Zero Trust concepts. If you cannot reconstruct who saw what and why, you do not just have a logging gap. You have an agent you cannot safely deploy in regulated environments.

What Are the Most Common Production Failure Modes?

Three failure modes break agents that worked perfectly in demos. Each has a specific mechanism and a specific architectural prevention.

Compounding errors in multi-step reasoning. For a workflow of n steps where each step has reliability r, system reliability is r^n. Reliability drops quickly as workflow length increases, meaning multi-step systems can fail often even when each individual step looks strong on its own. Validate between steps rather than only at the end. Early improvements matter more because errors cascade, and reducing chain length improves the structure of the system. Shortening a workflow can improve system reliability more than a model upgrade at the same chain length.
Stale data causing confident-but-wrong answers. A support agent queries subscription status from a batch-synced warehouse. The batch ran six hours ago. The customer cancelled two hours ago. The agent tells the support rep the customer is active. Faithfulness metrics confirm the response matches retrieved context. But faithfulness and freshness are different evaluation dimensions. Give every retrieved record explicit temporal metadata. Index refresh cadence is an architectural parameter that you set based on how volatile the underlying data is. CDC-based pipelines replace batch sync for data agents query against live business state.
Prompt drift. As a conversation grows, the effective weight of the initial system prompt diminishes because attention spreads across the full context window.

Treat prompts as versioned code artifacts stored in Git. Place key instructions near the beginning or end of prompts where they remain visible in long contexts. Re-run error analysis after every model switch or prompt update. If you ignore these failure modes, the agent looks reliable in a demo and unpredictable the moment it enters a long-lived production workflow.

What Is the Fastest Path From Demo to Production Agent?

Build the data access layer early. Agents that work on synthetic data but fail on real customer data usually do so because the data access layer was deferred, not because the model was wrong or the reasoning loop was broken.

Airbyte Agents provides that data access layer. Agent connectors are open-source Python packages that handle authentication, normalization, and typed access to enterprise systems like Salesforce, HubSpot, and Jira. The Context Store allows cross-source search. Airbyte supports OAuth flows and token refresh within its connector framework, including Managed Auth across connectors. Row-level and user-level access controls apply permissions across sources. It works with common agent frameworks and custom agent architectures, with the Agent MCP server, the CLI, and developer resources as entry points.

Talk to us to see how Airbyte Agents connects your agents to production enterprise data with authentication, permissions, and freshness handled from day one, or try Airbyte Agents today.

Frequently Asked Questions

How long does it take to build a custom AI agent from scratch?

A basic ReAct loop with a few tools can take days. Connecting that agent to production enterprise data with OAuth flows, token refresh, schema normalization, and per-user permissions can add weeks or months without purpose-built infrastructure.

Do I need a framework to build an AI agent?

No. If you build from scratch in Python, you learn the fundamentals: LLM calls, tool parsing, and state management. Frameworks can speed up orchestration and long-running workflow recovery, but they do not address enterprise data access, authentication, or permission enforcement.

What is the hardest part of building a production AI agent?

Authentication complexity across SaaS tools, cross-system entity resolution, data freshness maintenance, and per-user permission enforcement usually consume far more engineering time than the reasoning loop.

How do I prevent my agent from hallucinating on real data?

Distinguish between model hallucination, which is content untraceable to any source, and stale-data hallucination, which is accurate retrieval of outdated records. Use CDC replication to keep data fresh, attach temporal metadata to every retrieved record, and validate that retrieval results reflect current state before the agent reasons over them.

Can I use the same agent architecture for different data sources?

Yes, if the data access layer is abstracted. Agent connectors that handle authentication and normalization per source let you add Zendesk, Slack, or Google Drive without rewriting agent logic.

Try Airbyte Agents

Airbyte connects your agents to all of your data and assembles context before they run. Build agents that actually know your business.

Try it free Talk to sales