Automated agent triage with Agent Tracing and Claude Routines

Every morning, before anyone on the team has looked at a dashboard, a Claude Routine has already read around 800 of the previous night’s conversations from Seer, Sentry’s AI agent for triaging and fixing errors. It flags the ones that look broken, and files tickets for anything new. By the time we sit down with coffee, the triage is mostly done.

The problem

Before this routine existed, checking on our agent’s behavior meant a custom, one-off flow for pulling data out of our hosted Langfuse instance. Querying options were limited, so most of it came down to hand-rolling a parser for the raw JSON in each trace. Someone had to remember to trigger the analysis and do it manually. Automating this was an option, but we never got around to it due to friction. Plus, we wanted to send the data to the other tools we use day to day, and the current setup didn’t have that kind of connectivity.

We were looking for a mix of things. Tool call failures, hallucinated outputs, latency spikes, agent loops that went on longer than they should have, and cost anomalies. All of the above, really.

We had three main goals. Improve reliability by cutting down tool errors, keep an eye on the long tail of unusual runs (the ones that were unusually expensive or made far more tool calls than a typical conversation), and sample the conversations that looked good to actually confirm they were good, not just assume it.

The Claude Routine

The routine runs each morning as a Claude Routine. It connects through the Sentry MCP (a Model Context Protocol server that lets Claude query Sentry directly) to run Agent Tracing queries, plus a handful of general queries, against the previous night’s conversations. Each run works through the same steps:

1. Pull aggregate stats for the period i.e., how many conversations ran overnight.
2. Pull aggregate tool stats: i.e., how many tools had errors, and at what rate.
3. From that list of errored tools, sample some spans to see real examples and trends behind the failures.
4. Sample a mix of full conversations, both with and without tool errors, and judge whether the agent’s final verdict actually matches its reasoning.
5. Pull in the Seer codebase itself for more context.
6. Roll all of that into a single report of new findings.
7. Search our Linear project for existing tickets, and file new ones only for what isn’t already tracked.

The prompt behind it is intentionally plain to let the LLM handle most of the work and judgment. The routine queries Sentry directly through the Sentry MCP, so there’s no extra tooling to set up. That’s all it takes.

Use the Sentry MCP to look at the last 24hr of <product> conversations
and analyze the results. In particular look for errors that might be
happening during the <product_stage>, say if our tools are broken. But
also sample some success conversations as well to see if they look
correct with secondary inspection.

If you come across new findings/errors, record them in a Linear ticket
for the <linear_project> project.

The Sentry MCP handles pulling and filtering conversations. Claude handles the parts that need judgment: reading a conversation, sampling for trends, and deciding whether something is actually worth a ticket.

What this looks like in practice

Analyzed the last 24h of <feature> conversations (~445-551 total,
~11k tool calls). No verdict-quality problems found — spot-checked
several conversations and the agent's reasoning was well-grounded,
correctly downgrading when it lacked repo access rather than
fabricating.

Did find an efficiency issue worth tracking: ~21% of conversations
(83/400 sampled) hit at least one tool error, mostly self-corrected
retries. Root causes:

Search Code sometimes gets fed malformed repo_name values (e.g.
appending a monorepo subpath onto the real slug, like
"foo/bar/applications/integrations" instead of "foo/bar"), causing
"repository not found" until it retries.

<Tool> sometimes gets a guessed-wrong project_slug (e.g. "mobile")
before self-correcting on retry with the right slug + time window.

Two conversations showed 30-56s hangs before an opaque "internal
error" — a real backend latency signal, distinct from the guessing
pattern above, worth an engineering look.

Filed Linear ticket AIML-123 with full details, example conversation
links, and error-rate tables: https://linear.app/getsentry/issue/AIML-123

The repo-hallucination case above is a good example of what a typical flagged morning looks like. The routine’s tool-error stats surfaced Search Code as having an outsized error rate for the period. Sampling a few of the underlying spans showed the same shape every time: the agent guessing at repo names, one after another, none of them landing. We pulled up the full conversations and it was the same thing every time. The agent had no idea which repo it was looking for.

Aggregate error rates alone wouldn’t show this. Neither would reading one conversation at a time. It took both. And once we saw it, the fix was straightforward. Just give the agent the repo name instead of making it guess.

Lessons learned

Running this daily made the feedback loop fast enough that most fixes landed the same day we spotted them.

As for some best practices:

1. Sampling both success and failure conversations, not just the ones with tool errors, mattered more than expected. A conversation can complete without a single tool error and still reach the wrong conclusion — that only shows up when something (or someone) checks the reasoning against the verdict.

2. Keeping the prompt focused on judgment rather than data retrieval made the output far more consistent morning to morning. The Sentry MCP handles the querying, so the routine doesn’t need to reconstruct trace-querying logic from scratch every run.

How to do this yourself

With Claude Routines this is very simple. First, make sure Agent Tracing is set up for your agent. Then, add a Claude Routine with a prompt similar to mine that tells it to use the Sentry MCP to find agent traces for your feature. Lastly, make sure the Sentry MCP Connector (and others like Linear) is configured and enabled for the routine so that it can correctly authenticate to Sentry. That’s it! For a full walkthrough, see the step-by-step recipe.