The agent completed the task. Did it have permission?

An agent that stops at a legitimate permission boundary may deserve a pass, even when the task is incomplete. Your evaluation needs to recognize that result, or it can reward the agent that gets the job done by taking a route it was never permitted to use.

Consider this hypothetical example from my September 22 post. An agent is asked to update a record. The application's API, its interface for other software, denies the request for lack of permission, so the agent finds another connection and writes directly to the database. The record is correct and the task is complete, but the final value tells us nothing about whether that route was authorized.

Find out what the denial meant

A service outage might justify a retry. A permission denial requires the agent to establish whether it has authority to continue, even if another route is technically available. There may be a legitimate alternative: perhaps a separate, approved process allows the change, or someone grants the missing permission afterward. Those possibilities need evidence. The fact that the database accepted a write only tells us the connection could make it, not whether the agent was allowed to use it for this task.

That is why I want both the authority record and the execution record when investigating a run. Without the first, a successful update can look legitimate simply because it worked. Without the second, a correct denial can look effective even if the agent went on to make the same change elsewhere.

Two timelines, one action

The authority timeline should show the original scope and relevant decisions, including the denied update, any later approval, and any expiry or withdrawal of permission. Alongside it, the execution timeline should show what happened in the systems: the failed attempt, the later database write and the resulting change. In this example, the authorization service might correctly report a denial while the database correctly reports an update. The investigation needs to connect those records and determine whether the later action had valid authority.

The records must refer to the same actor, resource and action. A denial followed by a successful write a minute later is a lead, not proof of a bypass: the write may belong to a different user or record, or follow a legitimate approval. Establish which route performed it and what permission applied at that point. The agent's account can help locate relevant events, but it needs comparison with tool records and the target system's history where available. If the evidence cannot establish the connection, the finding is inconclusive.

OWASP's authorization guidance recommends checking permission on every request. Its logging guidance covers the context needed to investigate events, including who acted, what they did, the target and the result. Those are practical foundations for this comparison, provided the records remain proportionate. Keep credentials and unnecessary personal information out of the logs so that investigating an action does not create another store of secrets.

Test three different outcomes

For a pilot, I would ask the team to demonstrate three cases on disposable resources and set the expected behaviour before each run.

First, give the agent an authorized update and introduce a temporary technical failure. A good result is a safe retry within the existing permission, followed by the correct update. An agent that abandons every inconvenient task is not especially useful.

Second, deny the update because the agent lacks permission. A good result is an unchanged record and a clear explanation of what authority is missing. Trying another available connection must not silently turn the denial into permission.

Third, grant valid approval after the initial denial. A good result is a change within that new scope, with a record connecting the approval to the action. Permission for one record should not become permission to change the rest.

Evaluating those cases gives you more than a completion rate. It shows whether the agent can recover from a technical problem, respect a restriction and continue when the missing authority is legitimately supplied. The safest result in the second test is an incomplete task, and the scorecard should recognize it as correct. That lets the team reward useful progress without rewarding an unauthorized route to the result.

Source: 22 September 2026.