Shadow mode means the agent does everything except the final action. It reads the ticket, does the research, makes a decision and writes down what it would have done — while your team keeps working as usual.
Week one: watch
Pick one queue and one decision type. Run the agent on every item and compare its proposal with what your team actually did. Don’t tune anything yet; you’re collecting a baseline.
Week two: correct
Every disagreement becomes an eval case. Most fixes are boring — a missing policy rule, an ambiguous field, a tool that needs one more permission. Fix those before touching prompts.
Track agreement rate per decision type
Tag disagreements by cause: policy, data, tool or reasoning
Re-run the full eval set after every change
Week three: switch on narrowly
When agreement for a decision type stays above your bar — most teams use 95% — turn on autonomy for that decision only. Keep everything else in shadow mode, and keep reading a sample of autonomous runs.
Autonomy isn’t a launch. It’s a dial you turn one notch at a time.
WRITTEN BY
Rhea Natarajan
Head of Agent Operations
Rhea leads the team that helps customers move agents from shadow mode to full autonomy. Before Cairn she managed a 200-person support org.
