Blog

We ran our agent platform for six months. Then we rewrote it.

Building at Turnkey
·
·
Conner Swann
Staff Engineer, Applied AI
Abstract teal and copper fluid texture

About: What six months of running Valet v0 taught us, why we rewrote its runtime, and what building v1 taught us.

Audience: Engineers and platform teams building internal agents, and anyone else deciding whether to build on a hosted agent runtime or own the agent loop.

What you’ll learn:
  • Which product decisions held up after six months of daily use
  • Why splitting one agent across five components made the runtime fragile
  • Why owning the agent loop matters more than it first appears
  • Why named assistants stopped working once more people used them
  • Why agents need to sleep and wake on events
  • What the v1 architecture looks like and where its failures moved

Reading time: ~12 minutes

Definition

What is Valet?

Valet is Turnkey's internal agent platform for tasks that span code and company systems. This article is the second in a series of articles that explore how we built Valet. The first, Why serious companies build their own agent platforms, covers identity, permissions, approvals, and audit trails for company agents.

Recently, I was walking my dog when I sent Valet a Telegram message about an open pull request. I asked it to check CI, merge the request if the checks passed, and file a follow-up ticket. Valet started an isolated work session and returned the results before I got home.

This request crossed five separate systems: Telegram, my Valet, a remote sandbox, GitHub, and Linear. There was no need for me to keep a laptop open or watch the work. Valet did the work while I did other things.

Most days, this is how I use our internal agent. Six months of using Valet v0 has confirmed the product decisions that make this kind of work possible.

Still, the runtime underneath those decisions was much harder to operate than it needed to be. Valet v0 split one agent across several processes, stores, and protocols, and each boundary added synchronization and recovery code.

We needed to keep the behavior but replace the runtime. The reasons why are detailed below.

In doing so, we learned a second set of lessons that were just as valuable and will help us in future iterations. This post covers what v0 got right, why its runtime failed, and what building Valet v1 showed us that v0 could not.

What six months of Valet v0 confirmed

Four product decisions that we made with v0 held up under daily use. Valet v1 keeps the behavior behind all four with a few modifications.

1. Work reaches the agent from any channel

Slack, Telegram, the web app, and scheduled jobs all deliver requests to the same conversation. The channel controls how messages are presented and delivered. The channel does not create a separate agent.

I can start a task in the web app, add context from Slack, and approve an action from my phone without copying context between tools.

2. The agent sandbox works like a full workstation

Valet v0 gave each session a full remote development environment: VS Code in the browser, a terminal, a headless browser, a remote desktop, and public preview URLs for development servers. The person and the agent worked against the same files.

About half of the useful work involved more than editing files. Browser tasks, visual checks, preview links, and interactive commands all depended on the workstation model.

3. A personal assistant coordinates the work

My Valet keeps memory, runs scheduled prompts, and starts isolated work sessions for separate tasks. My Valet prepares a morning digest, checks recurring work, and stays available while those sessions run in parallel. After six months, the coding agent had become one worker that the assistant used.

4. Approvals survive after the agent stops waiting

We built a human-in-the-loop mechanism to pause a risky action until a person approves it. The same mechanism worked for clarifying questions and missing credentials, because all three pause a tool call until a person answers.

Valet stores the pending request, sends it to the active channel, and resumes the tool call after the answer arrives. The stored request survives a process restart, so I can approve a destructive action from my phone without anything running while I decide.

Together, these four decisions made Valet useful in daily work, and we wanted to preserve them. The problem wasn't what Valet could do, but the complexity of the architecture required to make it work.

Why the Valet v0 agent runtime architecture failed

Valet v0 treated the agent runtime as a distributed system. Each failure below comes from that decision.

Valet v0 runtime
One prompt, five components
Every handoff needed a receipt, a retry policy, and a recovery rule.
Client
Web, Slack, or Telegram
Cloudflare Worker
Admits the request
SessionAgent Durable Object
Keeps the official record
Runner
Carries instructions into the sandbox
OpenCode
Does the agent work
Three components each held part of the session state.
State writers
Durable Object
Runner
D1 database
About 10,000 lines of orchestration code kept these components behaving as one runtime.

One prompt crossed five components

A request passed through the client, a Cloudflare Worker, a SessionAgent Durable Object, a Runner process inside the sandbox, and an OpenCode subprocess. These components communicated over HTTP, WebSocket, and server-sent events, and we maintained two custom message protocols between them.

The design worked like a paper file passed down a row of desks. The Worker admitted the request. The Durable Object kept the official record. The Runner carried instructions into the sandbox. OpenCode did the agent work.

Each handoff needed a receipt, a retry policy, and a rule for what to do when the next desk stopped answering. We wrote about 10,000 lines of orchestration code to make the desks behave like one runtime. Every component could report healthy while the session as a whole was broken.

Ramp improved its background agent, Inspect, along a narrower path. In “Inspect at Scale,” Ramp describes removing the HTTP server inside each sandbox and connecting its Durable Object to the sandbox directly over WebSocket.

Ramp's change cut the average time to interactive from about 6.5 seconds to 2 seconds. We moved the boundary further. In Valet v1 the agent runtime is a library: a prompt goes in, events come out, and after a restart the host rebuilds the runtime from a durable store.

We could not change OpenCode's agent loop

OpenCode let us ship quickly. We could start OpenCode and observe it, but we could not change how it worked inside. Valet had to infer OpenCode's internal state through its external interfaces.

Context compaction was particularly annoying to deal with. When a conversation grows too long for the model, the agent loop summarizes older messages to make room.

We wanted the assistant to save important information before that happened, so we had to build the memory step outside the runtime and time it against a process we could not inspect. When OpenCode crashed, the Runner counted failures, applied backoff, and decided when the session could not recover. We built machinery around OpenCode's agent loop because we did not own it.

Valet v1 owns the loop, and compaction is where that paid off first. After we moved teammates onto v1, they reported that v1 forgot context after about 15 messages and forgot which tools it had been using.

We traced the reports to three defects in our own compaction code. One defect kept only about 4 percent of the context window word for word. Another defect used two different token counts to decide when to compact and where to cut. A third defect discarded tool output before the summarizer could read it.

We shipped fixes within about two hours. A few weeks later, I had threads more than a week old that still kept their important context. In v0, the same investigation would have ended at OpenCode's process boundary.

Agent session state had three writers

Three components wrote session state: the Durable Object, the Runner, and the D1 database. Together they worked like three people editing offline copies of the same document. Each component held part of a session's state.

Most of the time the copies agreed. After a failure, the system had to decide which copy described what had actually happened.

The implementation used delayed writes and a resynchronization protocol that ran when the Runner reconnected. The resynchronization protocol had to decide which copy was newer and whether an interrupted write had finished.

Valet v1 keeps one copy of the conversation in one durable store. Connected clients still receive live events immediately, but they read from that record and never write competing versions of it.

Sandbox snapshots preserved stale state

The sandbox's persistent disk kept the repository and other files across hibernation. The persistent disk also collected temporary runtime state, acting as a system of record. A snapshot faithfully photographed the disk, including expired authentication tokens, lock files left by dead processes, and databases from older runtime versions.

We made a version of this mistake again in v1. Prebuilt sandbox images were tagged by repository commit alone. When we updated the base image underneath, nodes kept serving the old image because its tag had not changed.

The fix was to name each image by everything that went into it. The lesson now applies to every layer: a stored artifact needs a name that changes whenever its inputs change, and conversation state belongs in the database, not on the disk.

Agent credentials and tool policy lived in the wrong places

Valet v0 passed credentials through environment variables. Child processes inherited the credentials, diagnostics could expose them, and rotating them required a restart that interrupted long-running sessions.

Valet v1 resolves credentials at the point where a tool or command needs them.

Tool policy in v0 grew as wrappers and conditional checks around individual tools. Adding a rule often meant changing several call sites, and approval behavior could differ by channel or tool.

Valet v1 asks one policy subsystem whether a specific person can invoke a specific tool with specific arguments in the current context. The fourth post in this series covers that design.

What building Valet v1 taught us that v0 could not

Valet v0 had essentially one daily user: me. That was enough to test features. It was not enough to test the concepts people use to understand the product.

Valet v1 went to the whole company, and four lessons came from that wider use.

Lessons from Valet v1
What one user could not show us
Valet v1 went to the whole company, and four lessons came from that wider use.
1
Named assistants did not scale to more users
v0: one named assistant per person
New users could not tell which assistant was talking
Everything is a thread; a persona is a setting on it
2
Agents need to sleep and wake on events
v0: channels, schedules, and webhooks all delivered a prompt
To learn whether CI passed, the agent polled or was reminded
Threads subscribe to events and wake on them
3
A stored approval still has to reach a person
v1: durable approvals worked as designed
A child thread waited two and a half hours, unseen
Route the request to wherever the person is
4
A workstation earns trust when the real tests run
v0: judged by its editor, terminal, browser, and desktop
Without a local cluster, engineers fell back to draft PRs and CI
Kubernetes in the sandbox: 643 of 643 end-to-end tests pass
One user tested the features. The whole company tested the concepts.

Named assistants did not scale to more users

We built v1 around the same object model as v0. Each person had a named assistant, the assistant owned a set of sessions, and each isolated work session was a new session with its own sandbox.

The model was clear to me because I designed it. New users kept asking which assistant was talking, where it was talking, and which one they should address. Slack made the problem concrete: one Slack app cannot present several addressable bots, so named assistants could not be addressed by name where most of our work happens.

We replaced the object model in October. Valet v1 now treats everything as a thread. A thread holds a conversation, can have child threads, and can attach a sandbox.

A persona is a setting on a thread. It no longer owns other threads. Where an agent listens and replies is a separate setting, which we call Presence.

The change removed named assistants and the session layer from the interface. It also separated choices that v0 had tied together.

A new isolated work session in v0 always meant a new conversation, a new sandbox, and a new lifecycle. In v1, each of those is its own decision.

Agents need to sleep and wake on events

Valet v0 treated channels, schedules, and webhooks as different ways to deliver a prompt. Nothing gave an agent a way to wait. To learn whether CI passed, the agent either polled or a person reminded it.

Valet v1 routes asynchronous work through one event pipeline. Webhooks from GitHub, Linear, and Slack are converted into one shared event format, and a thread can subscribe to the events it cares about.

An agent can open a pull request, arm a wait for CI to finish, and stop. When the checks complete, the event wakes the thread with the result. The thread does not need to understand GitHub's webhook format, and nothing runs while it waits.

The same pipeline delivers results from child threads, timers, and channel messages.

A stored approval still has to reach a person

Durable approvals worked as designed in v1, and that exposed the next problem. A teammate left a task running and came back two and a half hours later. A child thread had been waiting the whole time for approval to push unsigned commits.

The approval was stored correctly and appeared when he opened the child thread. Nothing told the parent thread or the Slack conversation that the approval existed.

The wait itself cost almost nothing. The sandbox had hibernated, and the agent's state was in the database. The cost was the teammate's time.

A durable approval solves half the problem. The other half is routing the request to wherever the person is, including approvals raised by child threads.

A workstation earns trust when the real tests run

In v0 we judged the sandbox by its tools: an editor, a terminal, a browser, a desktop. Engineers judged the sandbox by whether their own project ran.

Our main backend repository runs its end-to-end tests against a local Kubernetes cluster. If the sandbox cannot run that cluster, an engineer falls back to opening a draft pull request and waiting for CI.

Valet v1 now runs Kubernetes inside the sandbox without giving the sandbox host privileges. The main backend repository's full end-to-end suite, 643 of 643 tests, now passes in Valet sandboxes, split across the same six parallel shards that CI uses.

Startup time is the other half of earning that trust. When we asked engineers what kept them from trusting Valet with larger tasks, slow sandbox startup and repeated toolchain setup ranked near the top.

We now bake project setup into reusable images. A workstation that is complete but slow to start still loses to a laptop.

How Valet v1 runs the agent loop as a library

We did not replace the multi-channel conversations, workstation sandboxes, scheduled work, isolated work sessions, or durable approvals. We replaced the runtime that implements them, and later the object model that organizes them.

The agent loop is a library. The Valet API server embeds the agent loop library directly. The library owns the loop, the message queue, the tool catalog, and approvals, and it reaches storage, event delivery, sandboxes, credentials, and file storage through interfaces.

The sandbox no longer hosts the agent. It provides a filesystem and a shell for tools that need them. The API process holds the conversation, applies policy, resolves credentials, and writes events to one store.

Valet v1 runtime
The agent loop is a library
One process owns the conversation. The sandbox runs tools.
Channels and webhooks
Slack, Telegram, web, schedules, GitHub, Linear
Event pipeline
One shared event format wakes the thread that subscribed
API process
Agent library
The API server embeds it directly.
Loop
Message queue
Tool catalog
Approvals
The process also holds the conversation, applies policy, and resolves credentials.
One durable store
One copy of the conversation and its events
Sandbox
A filesystem and a shell for tools that need them
Credential provider
Credentials resolve when a tool needs them
Restart the process and the library rebuilds from the store.

Centralizing state also moved the failure mode. In September, Valet slowed to a crawl for every user. Database queries were finishing in under a millisecond, yet authenticated pages took 15 to 40 seconds.

Background polling, cleanup sweeps, and lease renewals had used up the database connection pool, and ordinary requests waited behind them. We identified pool exhaustion the same evening, and a fix that limited the background work restored normal response times that night.

In v0, the same symptom could have come from any of five components. In v1, there was one process and one database to inspect.

Valet v1 lets an operator choose where each part runs

The Valet v1 boundaries let an operator choose where the API, storage, models, and sandboxes run. Turnkey runs Valet on Kubernetes because Kubernetes fits our infrastructure. The agent library talks to storage and sandboxes through interfaces, so Kubernetes is one implementation and not a requirement.

An organization that values privacy or needs an air-gapped environment could self-host the API, storage, sandboxes, and local inference on one machine. The architecture could also support Cloudflare with a remote sandbox provider. We have not built that path.

Adding a host or a sandbox provider does not require another rewrite of the agent runtime.

Final thoughts: Ship the first version to learn which lessons matter

Shipping v0 was the right decision. Running Valet v0 on real work showed us that multi-channel conversations, workstation sandboxes, a personal assistant, and durable approvals were worth keeping. It also gave us the operational evidence to identify which runtime boundaries to remove.

With one heavy user, v0 tested features well. Concepts such as named assistants, sessions as containers, and an agent that could not wait only failed when more people relied on them. Valet v1 has now been used across the company, and its design reflects what those users ran into.

In the next post, I’ll cover Valet v1 itself: one self-contained binary, the deployment options, and how to run Valet yourself. Then, my fourth post will cover how Valet decides whether an agent may take a specific action.

Conner Swann
Staff Engineer, Applied AI

Related articles

Introducing TVC Verifiable State: Persistent storage for TEE applications (Alpha feature preview)

How Verifiable State gives applications running in Trusted Execution Environments (TEEs) persistent storage.

Why serious companies build their own agent platforms

What it takes to build internal AI agents that act in platforms like GitHub, Slack, and AWS: agent identity, scoped permissions, durable approvals, and audit trails