A new AI agent should reach your real work in three stages: shadow mode, then supervised, then autonomous. Have a named person on your side decide each move. Don't skip a stage because the demo went well.
That order is how we roll a build out, and it is written into our published process. This note walks the three stages, the step before them, and who should hold the decision at each gate.
You can use it on any agent, including one somebody else built.
How to roll out an AI agent safely
Roll it out in stages, and let each stage answer one question before the next one starts. Shadow mode asks whether the agent's output is right, and supervised asks whether its actions are safe. Autonomous asks whether anyone still needs to look first.
| Stage | What the agent does | What your person does | What should move it on |
|---|---|---|---|
| Shadow mode | Drafts only, and none of it goes out | Does the work as usual, compares | Your owner agrees the drafts meet the test sentence |
| Supervised | Acts, after a person approves | Approves, edits or rejects each action | Your owner agrees the approvals have gone routine |
| Autonomous | Acts inside its scoped list | Watches the dashboard, pauses if needed | Nothing. It stays until a person changes it |
The test sentence is the line in the signed scope that says what "working" means. Write it before the build, because every gate in the table is checked against it.
The same four surfaces are in place from day one: the workspace, the scoped action list, the management dashboard and the audit trail. Only the gate changes. That is what human oversight looks like in practice: a named person, standing at a named gate.
Stage 1: what shadow mode is
Shadow mode means the agent runs on live inputs while its output goes nowhere. Your team keeps doing the job by hand. The agent's version sits beside theirs, and somebody compares the two.
This is the cheapest stage to be wrong in. A bad draft costs a minute of reading, while a bad action can cost a client.
In Moffatt v. Air Canada, a British Columbia tribunal held the airline responsible for what its website chatbot told a customer, per McCarthy Tétrault's summary of the decision. The case is told in full in the support-agent escalation note.
Your agent speaks for you from its first live action. So the first stage lets it speak to nobody.
What you watch here is the gap between the draft and what your person did. Keep the drafts and the comparisons somewhere you can search later. The disagreements are the useful part. Each one is either a wrong draft or an unwritten rule your team follows without noticing.
Stage 2: how long an agent should run supervised
An agent should run supervised until the approvals stop teaching anyone anything. There is no honest number of days. A loop that runs forty times a day settles the question fast; one that runs twice a week takes months.
Supervised means the agent acts only after a person approves each action, and that reviewer can change it first. The pause is one click, and the dashboard lets you override anything flagged. What that screen should show you on day one has its own field note.
This is where most teams already are. LangChain, which sells agent tooling, surveyed more than 1,300 professionals for its 2024 State of AI Agents report.
LangChain found that most teams gave their agents read-only tools, or required human approval for more significant actions, such as writing or deleting. LangChain's respondents rated performance quality as their top concern, more than twice as significant as cost or safety.
Watch for one signal in this stage: approvals turning into rubber stamps. When your reviewer approves fifty in a row without reading, the gate has stopped doing work. That is the moment to decide something.
Leaving it in place builds a second inbox, and a second inbox gets ignored. How to gate on consequence so that does not happen is a separate question.
Stage 3: what autonomous means in practice
Autonomous means the agent acts on its own, inside the action list it was scoped to, with nobody approving each step. It does not mean unwatched. The dashboard stays on. So do the audit trail and the pause.
Once the agent acts, each action and the reasoning behind it go into the audit trail.
It also does not mean everything at once. Let the routine actions run on their own while anything that sends, pays or deletes stays behind an approval. Our process states it plainly: "Anything sensitive gates on a human approval until you say otherwise." That split is a normal place for a careful rollout to sit.
The action list itself is fixed at design. When the job changes, have your owner change the list in writing. How to draw that list is its own piece of work.
Decide in advance who can put the approval step back, and write that down too. After a surprising mistake, moving back is the right call. What to do in the hour after that mistake is a separate procedure.
The private beta before shadow mode
Before shadow mode, the agent should pass a private beta you can drive end to end. Demo data answers whether the idea works. Only live inputs answer whether this agent works on your loop.
Our build step ships that private beta first. Run it on real cases and try to break it on purpose. Fix whatever feels wrong before anything touches live work. Then we turn it on, in shadow mode.
The two steps test different things. In the beta, your owner hunts for failures on cases they chose. In shadow mode, the agent meets live inputs nobody chose, like forwarded threads or half-filled forms.
This is also where you find out whether the test sentence was right. If the drafts look good and the test still fails, the sentence was measuring the wrong thing. Fix the sentence before you promote anything.
After it goes live, we stay on to run it with you.
Who decides when an AI agent can act on its own
A named person on your side should decide each move, and the decision should be written down. Don't let the agent's own numbers make the call. A vendor can recommend, but your person should decide.
Our process asks you for one internal owner who can answer "is this how it should sound, look, decide?" That person is the natural gate-holder. Have them sign off on the move from shadow to supervised, the move to autonomous, and any move back.
Write that person's name into the scope before the build starts.
Stanford researchers asked 1,500 workers across 104 occupations how much human involvement they want in AI-assisted tasks. In Stanford's data, the most-wanted level in 45.2% of those occupations was an equal partnership between worker and agent.
Anthropic's study of how people use its Claude Code agent shows how that involvement changes in practice. In Anthropic's data, new users switched on full auto-approve in about 20% of sessions; by around 750 sessions, it was over 40%.
Anthropic also found that experienced users interrupted more often, in about 9% of turns against 5% for new users. They stopped approving each action and started watching.
Anthropic, a model vendor, concludes that rules requiring a person to approve every action add friction without necessarily adding safety.
That is the shape of a good promotion: the person moves from approving to monitoring, with the pause still within reach.
What that owner does the rest of the week is its own role, with its own field note.
When a build stays supervised
Some builds stay supervised for good, and that is a legitimate outcome. A loop where every action is expensive, and each one is different, may never get your owner's sign-off for autonomy. That is the owner doing the job.
A permanently supervised agent can still pay back. It gathers and proposes; your person approves work they used to do by hand. The only question is whether that trade clears the bar you set before the build. If it doesn't, check the next loop against the five questions that decide whether it should be an agent.
Common questions
Does shadow deployment work for LLM agents?
Partly: shadow mode works for an agent's single-turn replies, and poorly for long conversations. Arize, which sells AI observability tools, says a shadow conversation drifts from the real one after the first reply that differs. Arize adds that any tool that writes data or sends a message must be switched off or sandboxed during the run. So shadow single steps, and let the supervised stage test whole threads.
What's the difference between human-in-the-loop and human on the loop?
Human-in-the-loop means a person approves each action before it happens, while human on the loop means the agent acts and a person monitors, ready to step in. In this rollout, supervised sits in the loop; autonomous sits on the loop. Shadow mode is neither, because the agent takes no action for anyone to approve or watch.
How long should a shadow deployment run?
Shadow mode should run until it has seen every kind of input your loop gets, with enough real runs to judge the test sentence. Arize, which sells AI observability tools, says a shadow deployment should usually cover at least one full weekly cycle. A loop with a monthly close needs longer. The overlap costs double work; see what an AI agent costs to build and run.
What the rollout call covers
Bring the loop you want to hand over, and the person who would own it. The diagnostic call is thirty minutes, free. You leave with a blunt yes, no, or not-yet on that loop. A yes comes with a one-page write-up of what we'd scope. Book it here. Decide who owns the gate before you decide anything else.