Anthropic’s September 17 report proposes measurements for AI-led research, oversight of agents and the resources behind model development. For teams operating their own agents, its most practical contribution is a way to ask how monitoring works after deployment. Anthropic’s report
Logging an action and reviewing it in time to prevent harm are different accomplishments. A monitoring dashboard should make that difference visible.
Three measures worth separating
Anthropic describes coverage, the proportion of actions passing through monitoring; review latency, the delay before automated or human review; and escalation rate, the share blocked, redirected or flagged. Its report distinguishes online checks before execution from offline review afterward. The disclosed figures describe its internal platform, not a universal standard for agent safety. Definitions and reported scope
Consider an agent preparing a customer report. Recording its file reads may be enough to investigate a bad citation later. Sending the report to the wrong recipient is harder to undo. That action needs a different control and a different acceptable delay.
A team could begin with a table like this:
| Action | Evidence to record | Review question |
|---|---|---|
| Read a supplied document | Source identity and access outcome | Did the agent use the intended material? |
| Write a draft | Changed file and validation result | Is the result accurate and complete? |
| Send externally | Destination, approved content and authorization | Was the action permitted before it occurred? |
This is an illustrative design, not Anthropic’s implementation. Avoid copying sensitive document contents into a telemetry system merely to make a dashboard more detailed.
A low escalation rate is ambiguous
A falling alert count could mean fewer unsafe actions. It could also mean the detector is missing them. To distinguish those explanations, inspect samples that were not flagged and test known failure cases. Keep the denominator visible: a percentage of messages is not necessarily comparable with a percentage of tool calls or completed tasks.
Review latency deserves the same care. An average can hide a small queue of consequential incidents waiting much longer than the rest. Record which checks happen before execution and which can only support investigation afterward.
Separate measurement from assurance
Anthropic acknowledges limits to comparing labs, including inconsistent methods and the use of its own models as judges. It proposes third-party verification. Its disclosed measurements therefore provide evidence about a particular system under a particular method; they do not prove that monitored agents cannot cause harm. Methodological limits
For a smaller engineering team, the immediate step is to choose one deployed workflow and define what an action, a review and an escalation mean. Connect each measure to someone responsible for responding. Our agent architecture guide covers the surrounding control flow. Monitoring becomes useful when its evidence changes a decision about what the agent may do next.