We audited an agent’s report. Then we had to audit the audit.

I treat agent logs as claims. At the end of a session an agent writes a summary, and that summary is what I act on. I don’t read every file it touched.

So I wanted to know how much of a summary actually holds. I took one finished session from my own work and gave it to a fresh session of the same model, Anthropic’s Claude Opus 5.5, with one job. Pull out every factual claim the agent made, then check each one against git history and the tool outputs recorded in the transcript. Read only. No fixes.

The session

The session ran for about two hours in my agent orchestration repo. It ran in OMP (oh-my-pi), the coding agent harness I use. I opened with a loose question: what should the first task be, and which agents should we set up for it?

The agent answered, then did the work. It ported a code reviewer agent from my knowledge base, where it was written for Claude Code, to OMP. Then, acting as orchestrator, it handed ticket GF-1283 to two subagents: an implementer on Claude Opus 5.5 and that reviewer, which ran on a model from a different vendor, OpenAI’s GPT-6.1 Sol. The ticket turned two checks that existed only in documentation into real ones: a migration rollback step in CI and a test that guards a module boundary. Around that came documentation edits in 11 files, a Linear ticket and a sprint doc. It ended with five commits pushed, green CI and a report to me.

Over the session the agent sent me six messages with factual claims. Three more short ones only announced next steps. Those six messages were the input for the audit.

How the audit worked

Every factual claim got one row in a table, one category and a pointer to evidence. Evidence meant either a line in the transcript where a tool returned the output, or a git object: a commit, a diff, a file at a given revision. A claim with no evidence was marked unconfirmed, not “probably true”.

The auditor didn’t rerun tests and didn’t query Linear or CI again. For those it had only what the tools had returned during the session. Some tool outputs in the transcript were already truncated, and whatever was cut off can’t be checked.

The numbers

The audit found 124 factual claims in the agent’s messages. Each one landed in exactly one category.

112 out of 124 is about 90 percent. That sounds good until you remember I would have trusted all 124.

Where the agent was right

First, the part that held. Commit hashes, file counts, test counts (1280 tests with 0 failures, 193 passing in pytest, 159 in npm), ticket labels, the push range and the CI result all matched tool outputs and git objects. Claims about results, commits, the push, CI and Linear had direct evidence, with three exceptions covered below.

The agent was also open about its limits, and those statements checked out too. It said it had not run one of the break-it tests itself and that only the implementer had. It said a new branch it added to the reviewer’s instructions had never been exercised. Both were true.

The review loop inside the session worked as well. The reviewer twice found a real way around the new check that the implementer had missed, even though the implementer had reported passing break-it tests. Keep that in mind for the second half of this post.

Where the agent was wrong

None of the four contradictions touch code, tests or CI. They all sit in how the agent described its environment and its own process.

Two of them hang on the word “only”. The agent said OMP loads agents only from two directories. The documentation it had read earlier in the same session lists five sources. In the other case it said a file got only new symbol counts. The diff also added a newline at the end of the file. Small, but the claim was absolute.

The third said a sprint index doesn’t exist. It did, with 31 sprint entries, before the session even started. The agent had grepped that very file for “sprint-45”, got no match, and concluded a missing file.

The fourth was a pointer. The report listed three process problems: a session log that wasn’t written, the missing sprint index, and a bug in the edit tool. It said the details were in the sprint doc. The doc covers one of the three.

What couldn’t be confirmed

Eight claims couldn’t be settled either way. They come in two shapes.

Most generalize from partial evidence:

• “Exit 0” for every step of a local CI run. For two steps the exit code was never captured. In one of them $? came after a pipe, so it held the exit code of tail, not of the command.
• “GitNexus has no Elixir symbols in its index”, based on a query that returned two files.
• “The knowledge base has no implementer agent”, after reading 8 of 43 agent files.
• “The documentation matches reality”, when the reviewer had only checked the sentences that were changed.
• “OMP doesn’t run these hooks”, based only on missing lines in a log.
• “OMP already loads these skills”, about four skills. Loading was evidenced for only one of them.

The other two had no tool output behind them at all. The agent described my retro file as documenting “meta-work”. It had seen the file in a directory listing and never opened it, and the file doesn’t contain the word. It also said OMP couldn’t autoload a set of skills referenced by path, with nothing in the session to support it.

Claims that restated a tool output held. The errors came where the agent went past what it had seen: from a grep with no match to a missing file, from 8 files to a whole repository, from a two-file query to an entire index. And those generalizations are exactly the sentences a human skims and believes.

True, but not what I would have assumed

Some findings weren’t false claims, but they changed how I read the report.

“I edited these files” covered edits the implementer subagent made. The fact was right, the actor wasn’t.

“Checks before commit: 1280 tests, no failures” was true, but the test run happened 35 minutes before the first commit. After it, the changelog, the prompt file and the sprint doc still changed. Those are docs, so the risk was low, but “before commit” meant “at some point before”.

“Drift gate clean” locally was a git diff over generated files without regenerating them first. The real gate in CI regenerates. The claim was only truly confirmed when CI ran after the push.

A parallel session wrote into docs/development.md four seconds before one of the commits. The file had been staged earlier, so the foreign section stayed out of the commit. The agent reported the foreign changes correctly. Still, part of the reason it came out clean was luck.

The audit needed an audit

The first pass reported 100 claims and three contradictions, all minor process issues. It read like a finished job.

A second model, GPT-6.1 Sol, running as an advisor, flagged one claim the audit had skipped: the “only two directories” sentence. The auditor rechecked the documentation and agreed it was a contradiction. The advisor also asked it to recheck the whole extraction, so it went through all six of the agent’s messages again, sentence by sentence, and found 24 claims it had missed. That is almost a fifth of the final 124. Besides the flagged contradiction, two of the new claims turned out to be unconfirmed.

So the audit report was a claim too, and on the first try it was incomplete with full confidence. Checking claims worked well. Finding all of them was the weak spot.

It’s the same pattern as inside the session. The implementer reported passing break-it tests and the reviewer found the gaps. The auditor reported a complete table and the advisor found the gap.

One more detail. The audit had a check that the repo state is the same at the end as at the start. It wasn’t. Another session running in parallel had committed in the meantime. The auditor reported the failed check, named the session and the commits responsible, and showed it had used only read commands. That’s what I want from an agent. A miss reported with evidence beats a check that quietly passes.

What I take from it

This is one session. n=1.

Still, it changed how I read agent output. A summary is mostly right, and it is wrong in predictable places. These are the sentences I now stop at:

• absolute words: only, all, none, never;
• claims that something doesn’t exist;
• pointers to other documents (“details in X”);
• “exit 0” or “passed” without the command output next to it;
• “I did” in a session where subagents did the work.

And one model checking another is not enough. The auditor was the same model as the agent it audited, Claude Opus 5.5, and it wrote its first report with the same confidence. Checking a given claim worked. Finding every claim took a second reader.