I Built an Agent to Cover My PTO

April 1, 2026

Summarize with AI:

In my first post I mentioned that every processed customer call was automatically appending to a running coverage document. The idea was simple. I'm taking two weeks off, and whoever picks up my threads should have context without asking me for it.

That document has been writing itself since February. But somewhere between finishing it and actually packing my bags for London and Scotland, I had a different thought. What if nobody has to pick up my threads at all? What if an agent just handles it?

I'm a tinkerer. I can't help it. If there's a system that works, I want to see if I can make it work better. So instead of handing off a document, I built an agent that can respond to my colleagues on Slack while I'm out.

The inspiration

I'd been reading a post from Prithvi Rajasekaran on the Anthropic Labs team about harness design for long-running applications. He built a multi-agent architecture inspired by GANs where a generator agent produces work and an evaluator agent grades it. They go back and forth until the output meets a quality bar. The use case was frontend design and full-stack coding, not Slack messages. But the core idea stuck with me. Separate the thing doing the work from the thing judging the work.

I wanted that same tension for my PTO bot. Not because Slack replies need ten rounds of refinement, but because I don't want my agent confidently sending bad answers to my teammates while I'm on a golf course in Scotland.

What the agent knows

The biggest advantage I had was that most of my working context already lives in systems Airbyte connects to. Granola for meeting transcripts. Linear for tasks and project status. Pylon for customer threads. Notion for docs. Slack for conversations. I use these connectors daily for my own AI workflows, so wiring them into a PTO agent was straightforward.

On top of that, I have a local folder that serves as my "Jarvis." It's where I keep notes from customer meetings, ideas, rough drafts, things that don't live in any SaaS tool but matter for context. That gets loaded in too.

We also use Kapa.ai internally, which gave me a way to build in deep product knowledge. If someone asks my agent a question about the Agent Engine, it can pull from our actual documentation rather than guessing.

The architecture

The frame is two agents with different jobs.

A draft agent has the full tool suite. It can read from all the connected sources, search across replicated data, and compose a response. A review agent has no tools at all. It receives the draft plus all the tool results as plain text in its prompt and returns one of two verdicts: approve or revise.

If the review agent says revise, it sends back specific feedback. The draft agent incorporates that feedback and tries again. They loop until the review agent approves, and only then does the response go to the user.

async def run_adversarial_loop(client, user_message: str, max_rounds: int = 3):
    # Session-based — client preserves conversation history
    await client.query(build_draft_prompt(user_message))
    draft = await get_assistant_response(client)

    for round in range(max_rounds):
        review = await run_review(user_message, draft, captured_tool_results)
        if review["verdict"] == "APPROVE":
            return draft
        # Only sends feedback — session already has the original context
        await client.query(f"## Reviewer feedback\n{review['feedback']}")
        draft = await get_assistant_response(client)

    return SAFE_FALLBACK  # never returns a rejected draft

A few things worth noting here. The ClaudeSDKClient session preserves conversation history, so revision rounds only send the reviewer's feedback. Not the full prompt and tool results again. That's the non-obvious performance win. Each round is lightweight because the session already has all the context from previous rounds.

The review agent is the opposite. It's stateless and has no tools on purpose. That constraint is what makes this adversarial. The reviewer can't go verify things on its own or pull fresh data to fill gaps. It has to evaluate the draft purely on what's in front of it. If the draft agent didn't pull the right context or cited something vague, the reviewer has no way to cover for it. It either holds up to scrutiny or it doesn't.

I use Opus for both agents because I wanted the reviewer to be genuinely critical, not a rubber stamp. And notice the fallback at the bottom. If the loop exhausts all rounds without an approval, the agent doesn't send the last rejected draft anyway. It returns a safe fallback. I'd rather the agent say "Dallas is out, here's who can help" than confidently send a bad answer.

The key insight from Prithvi's work applies directly here. When you ask an agent to evaluate its own output, it tends to say "looks great!" even when it doesn't. Splitting generation from evaluation gives you a real feedback loop.

Why it's fast

You'd expect two agents going back and forth to be slow. It's not, and the reason is the data layer.

Airbyte replicates data from all my connected sources into a context store. When the draft agent needs to look up a customer thread or check a Linear ticket, it's not hitting each API sequentially and dealing with pagination and rate limits. It's pulling from already-replicated data. That makes tool calls return almost immediately.

Most of the time, the full loop resolves in one or two rounds. The review agent approves the first draft more often than not because the draft agent has good context to work with.

For the rare cases where the agent needs truly real-time data, it falls back to hitting the API directly. But that's the exception, not the rule.

The rest of the stack

The full setup: Claude Agent SDK with ClaudeSDKClient for session management. Airbyte connectors plus context store for the data layer. Railway for hosting. Mem0 for a memory layer so the agent can remember context across conversations while I'm out.

What I'm watching for

I leave in three days. The coverage document is ready as a backup. The agent is deployed and running. But I haven't stress-tested it with two weeks of real questions from real people yet.

I'm curious about a few things. How often does the review agent actually catch something meaningful? Do my teammates trust the responses or just wait for me to get back? Does the memory layer via Mem0 help the agent get better over the trip, or does it plateau?

I'll report back. Probably from a pub in Edinburgh.

This is the third post in Deployed with Dallas, a series on how I've been building AI into my day-to-day as a forward deployed engineer at Airbyte.

Limitless data movement with free Alpha and Beta connectors

Introducing: our Free Connector Program ->