Agent Ops

Agent Ops

What we learned running 11 million agent tasks a month

What we learned running 11 million agent tasks a month

The numbers behind autonomous agents in production: where they fail, how teams earn trust and why the trace matters more than the model.

The numbers behind autonomous agents in production: where they fail, how teams earn trust and why the trace matters more than the model.

Portrait of Ines Calloway

Ines Calloway

9 min read

Abstract data illustration for “What we learned running 11 million agent tasks a month”

In September, agents built on Cairn completed 11.4 million tasks for 2,400 companies. Not drafts, not suggestions — tasks: refunds issued, tickets closed, invoices matched, access granted. This post is about what that volume taught us.

Most failures aren’t model failures

When we sampled failed runs, fewer than one in five traced back to the model reasoning badly. The rest were ordinary software problems: an expired token, a rate limit, a tool that returned an empty list, a policy document that contradicted itself.

That changed how we build. Retries, idempotency keys and clear tool errors did more for success rates than any prompt change we shipped this year.

Autonomy is earned per decision, not per agent

Teams that succeed don’t switch an agent “on”. They switch on specific decisions — refunds under $150, address changes before fulfilment — once the agent has agreed with humans often enough in shadow mode.

  • Start in shadow mode and log every proposed action

  • Grant autonomy one decision type at a time

  • Keep an approval gate on anything irreversible or expensive

  • Review a random sample of autonomous runs every week

The trace is the product

Customers rarely ask which model we use. They ask what the agent looked at, why it decided and whether they can prove it to an auditor. A readable trace answers all three — and it’s the thing that turns a skeptical team into a confident one.

Nobody trusts a black box with a refund button. Everybody trusts a log they can read.

What’s next

We’re investing in multi-agent handoffs, cheaper evals and better replay. If you run agents in production and want to compare notes, our Agent Operations team would love to hear from you.

Portrait of Ines Calloway

WRITTEN BY

Ines Calloway

Co-founder & CEO

Ines ran support operations at a public payments company before co-founding Cairn. She writes about running agents in production and what changes when software finishes the job.

Field notes

One email a month. Only the useful parts.

Lessons from running agents in production, new templates and the releases worth knowing about. Unsubscribe in one click.

Create a free website with Framer, the website builder loved by startups, designers and agencies.