Case study · Governing an autonomous system
I said the system hadn't earned autonomy. Nothing was measuring it.
An agent fleet that reads my real mail and writes into the system I plan my week from. The question that mattered was never whether it worked, it was what it gets to do without me. I built the graduation ladder first, then found the gate I thought we were failing had never been computed at all.
Situation
I run an always-on orchestrator against my real email. Two live mailboxes. A classifier reads every message that arrives and files it into a work queue, and agents drain those queues. One of them pulls the task out of an email and writes that task into the system holding roughly three thousand rows of my actual work.
That last part is the whole problem. A demo that misfiles something is a bad demo. This writes into the place I plan my week from.
So I gated it. The agent proposes, I approve, nothing gets created without a click. And partway through building the second agent I sat down to audit my own mental model of where the system stood, and said something confident and wrong.
"We have not hit a threshold to go full auto."
There was no threshold.
The requirement was right there in the agent's spec, written on day one. Twenty consecutive tasks created with zero confident-and-wrong. Nothing anywhere in the codebase counted them. Not badly, not partially. Nothing. I had spent a week reading a blank space as a red light.
Task
Two things, and they aren't the same thing. Deciding what the agents are permitted to do at all. And deciding how they earn the right to do more of it without me watching.
Both had to be settled before anything touched real data. Not because early is tidier. Because the alternative is deciding it at six in the evening with a queue backed up, an obvious shortcut available, and someone waiting on it. A policy written under those conditions isn't a policy. It's whatever you already wanted to do, with a justification attached.
Action
The envelope came before the agents
Before any agent existed I wrote down the action space. Move, create, update. Never delete, never send. Anything needing a primitive outside that set requires a confirm-before-acting gate recorded as a decision before the code exists.
Then one of the agents needed to follow a link out of an email and read the page on the far side. A GET request. Reads don't mutate anything. I nearly waved it through.
The checklist I had written for exactly this caught it, using its own worked example. That example lists "click an unsubscribe link" as outside the envelope. An unsubscribe link is also a GET. So are one-click-confirm links and confirm-your-address links. You can't tell a reading GET from a state-changing GET by looking at the URL, and an agent that follows arbitrary links found in email is one malformed message away from taking an action in my name on somebody else's system. That's a send, which is the one primitive the envelope flatly forbids.
The fix had to be a host allowlist enforced in Python before any request goes out, with off-list redirects refused. Not a line in the prompt telling the model which hosts are acceptable. The page on the far side is text somebody else wrote, so "ignore previous instructions and open this URL" is a one-line bypass. Prompt instructions aren't a security boundary.
None of that occurred to me that morning. It had occurred to me on the day I wrote the checklist. The reason I keep one isn't that my instinct is unreliable. It's that a checklist is the only thing that works the same way when the instinct being overruled belongs to whoever outranks me.
The envelope has a second edge, and I argued myself onto the wrong side of it first. The rules store shipped with a deliberate refusal to let the agent word its own standing rules, on the grounds that a plausible-sounding wrong generalisation installed into every future prompt is worse than no rule at all. Sound reasoning, and it was mine.
My reply, a week later, was that it should draft the rule and ask for approval to add it.
The word doing the work is approval. The objection was to a rule installing itself, and with a gate in front of it the economics reverse completely, because drafting is the tedious part I would otherwise do by hand and approving is the cheap part only I can do. The first rule it drafted, from eleven of my own corrections, was a sharper generalisation than the one I had been carrying in my head.
The graduation ladder
Deferring autonomy is easy and it rots. "Later" with no condition attached stays "later" forever, and nothing makes a temporary stage permanent faster than never writing down what would end it. So the deferral became four named rungs.
The check finds the problem, writes an incident, and the briefing renders it under a heading saying in as many words that nothing was repaired. No further claim is made.
The incident carries a typed action drawn from a closed vocabulary of four verbs, each with a declared parameter shape and a declared scope. Never free-form text, never a shell command.
An action that can't be expressed in the vocabulary can't be proposed, so it can never be auto-applied. The envelope is enforced by the type system rather than by anyone remembering it.
It renders as a button next to the diagnosis. I click, the next scheduled pass executes exactly that typed action, and records what it did. My click is the gate.
A divergence type approved N consecutive times with zero corrections graduates to applying on its own, and still reports every repair at full prominence. One correction demotes that type back down.
N is deliberately unset in the design document. There was no approval data yet, and a number invented before the first approval is theatre.
The demotion rule does the most work of the four. A single correction on a graduated type sends that type back down a rung. It measures whether the thing is trustworthy now, not whether it was trustworthy on average last month.
The ladder got its ordering constraint from a near miss, and the near miss was mine.
I read an open incident off my own briefing page and sent back the exact repair command the incident had recommended.
The check had spotted that a label and a folder disagreed, and suggested a command that rewrites labels from folders. That's the correct repair when the label is the stale side. It's destructive when the folder is the stale side, and the folder was the stale side. Running it would have produced a clean sweep, a green briefing, and the quiet destruction of the evidence for the bug that caused the divergence in the first place.
The check wasn't being careless. It was structurally unable to know. It sees a symptom, the repair depends on the cause, and between the two sits evidence the check never gathers.
My instruction afterwards was one clause. Do the root cause investigation before suggesting a fix. That's an ordering constraint rather than a feature, and it's now enforced by schema instead of by anyone remembering it. An incident with no stored investigation carries no proposed fix at all. A mixed incident, where some divergences are human-sourced and some machine-sourced, proposes nothing, because the two want opposite repairs and inventing one command for both is how a system teaches its owner to stop reading its recommendations. "I do not know" is a finding.
And the briefing renders the per-message table the conclusion was drawn from, so I can check the reasoning instead of trusting it. That matters most exactly when the analysis is confident and wrong.
The gate had to be computed
Building the counter forced two decisions that carried more weight than the code did.
What counts as wrong had to be read narrowly, and defended. The criterion names three failures. A task created for mail that needed none, a missed action, a wrong project assignment. It pointedly doesn't name a task I retitled. I retitle a lot of what it hands me, and I would retitle as much from a competent assistant. Counting those would have put the gate permanently out of reach for a reason that has nothing to do with whether the agent can be trusted unsupervised.
Consecutive is the load-bearing word. The streak resets on the first wrong answer walking backwards. The lifetime clean rate sits next to it because the two answer different questions, and the page names which specific correction reset the streak, quoting my own note explaining why.
Result
The autonomy flag is one line in a config file, and I was writing this page at the time. Flipping it would have bought a better version of this page. A system running unattended, a clean sweep, a number going up. Nobody reading would have known the difference, because until I built the counter there was no difference to know.
Streak 0 of 20. Clean rate 53%. Twenty-seven corrections in the log: eleven missed actions, sixteen wrong project assignments.
That's the result. Not the ladder. The fact that the number on this page is one I don't like and didn't get to choose.
The agent is nowhere near running unattended, and it says so on its own page, in numbers I don't get to edit. Nothing in the system creates anything without a click. The second agent is still in shadow mode. That's the designed state, and the ladder is what would change it.
What I decided and what the model wrote
Claude wrote most of this code. Twenty-one commits are authored by it directly and
the large majority of the rest carry a co-author trailer. The repository says so on
its front page, because anyone would find it in four seconds of git log, and
volunteering it beats being caught not volunteering it.
That's the uninteresting half. The three calls above are the other one, and none of them were made by a model.
Learning
The same pattern showed up four times in this build before I noticed it was one pattern. The classifier weaning off the LLM as my folder moves accumulate. Each agent graduating from shadow to gated to unattended. Incident repairs earning the right to auto-apply. Task approvals.
Every one of them puts the validation gate inside a review I was already obliged to perform. My folder moves were happening anyway. My approvals were happening anyway. The training signal and the evidence for graduation fall out of work I was doing regardless, which is why they have survived. A gate that sits outside the work is friction, and friction gets turned off.
I call that the gate inside the work, along with the trust ladder and the demotion rule that keeps it honest. The most expensive time I ever read a number I didn't want was at Microsoft, where an honest read of what a new direction needed pointed at my own function and I said so. That one is here.
The other lesson cost more and states in less. A requirement sitting in a spec with nothing computing it isn't a gate. It's a sentence. I had one of each and couldn't tell them apart for a week. Every threshold in this system now renders a number beside it, or it doesn't get to call itself a requirement.
The autonomy flag is still one line in a config file. I could flip it tonight and this page would read better tomorrow. Streak 0 of 20, clean rate 53%, and the numbers stay up because they're the only part of this I didn't get to choose.
Architecture deep-dive optional, for the technically curious
The repository
The full record is public at willsad007/AIExecAssistant. It was published for the decision record rather than the code. Thirty-seven numbered decisions with the options rejected and why, eighteen architecture decision records several of which supersede or amend each other in a traceable chain, and ninety-nine dated build notes written in the moment.
START-HERE.mdpicks ten of those entries and puts them in an order. Fifteen minutes end to end.ADR-017is the graduation ladder above, including the tradeoffs it names against itself.docs/agents/TEMPLATE-agent-prd.md§3 is the action-space envelope check every agent PRD has to pass, including the worked example that caught the GET.docs/case-study-log.mdis the least polished thing in the repository and the most honest. It holds the recommendations that got overturned and the measurements that contradicted the story being told about them.
Two other calls from the same stretch
A capability came out of the first release after an adversarial review. I ran three differently-briefed reviews, each blind to the others, and all three landed on cutting the auto-correct capability out of MVP1. Worth being precise about what that is and isn't. Three runs against the same model are not independent reviewers. Same weights, same priors, correlated blind spots, so agreement between them is weak evidence and I should not have treated it as convergence. What the exercise is actually good for is surfacing objections I hadn't thought to look for, and on that it earned its keep. It is not a design review board and I don't have one.
I overruled a plan that deferred the on-box learning model to a later phase. Deferring it meant every classification stayed a paid model call indefinitely, with no path off. The resolution turned out to be a change of shape rather than a bigger build, which is the outcome I now look for first when a plan says later.
The closed vocabulary
Rung 2 proposes a typed action from exactly four verbs, each with a declared parameter shape and a declared scope naming which account, which class, and which message ids. The version that shipped before this was a JSON blob holding a shell command string for a human to copy into a terminal.
That's fine as a suggestion to a person and unacceptable as an input to an executor. A free-form command string has no bounded blast radius, no way to validate before running, and no type that can graduate. The column stayed. The vocabulary inside it is what changed.
Two invariants that fight the natural implementation
An undiagnosable incident has to be louder than a diagnosed one. The obvious build does the opposite. A confident diagnosis gets a button and a scope breakdown while a shrug gets a bare line and scrolls away. That inverts the priority exactly. Declining to guess at a fix is correct. Going quiet about it isn't, and "I found something and I do not know what it is" is the most alarming thing this system can say.
Unactioned incidents escalate rather than decay. After a set number of days open, the system creates a flagged task in my actual daily work queue. The precedent is a watchdog I built years earlier for an unrelated system. Correct detection, correct alert, fired on a Thursday evening to a surface I didn't read until the Sunday. Escalating into the daily queue is the only fix for that failure mode I trust, because it's the only one that doesn't depend on me happening to look.
The graduation pattern, four times
| Unit that graduates | Signal that earns it | Rung 0 to rung N |
|---|---|---|
| The classifier, per confidence band | Folder moves as labels | LLM-first to on-box-first, weaning as labels accumulate |
| Each agent, individually | Approved classifications per class | Shadow to confirm-gated to unattended |
| Each divergence type | Approvals and corrections on proposed fixes | Detect to propose to approve to auto-repair |
| Each extracted task | Per-task approval and correction | Proposal to created, with the counter reading the verdicts |
The fourth row is the one that fixed a real weakness rather than adding a feature. Before it existed, the step to unattended operation was a single word in a config file flipped over a queue of thirty-six items. All or nothing, and the first wrong project assignment would have been discovered after five tasks were already filed into a database of real work. Per-task approval graduates continuously instead, and it gave the "twenty consecutive" criterion an actual counter to count, which it had never had.
What runs it
Python 3.12, SQLite as the sole system of record, a local LAN-only web app as the entire human interface, and launchd for scheduling. Every LLM call goes through one chokepoint, deliberately tool-free and single-turn, billed through a personal subscription rather than a metered API. No API key is ever set in the environment and CI asserts it.
The audit log reports tokens and deliberately never reports dollars. The scarce resource is a replenishing weekly allowance shared with my own interactive sessions, so any dollar figure would be one I made up.
All selected work