Every coding agent looks capable in a demo. The videos show a clean task in a clean repository, and the result is a diff that works. Your repository is not clean, and your task is not that task.
This is the evaluation we actually run: five tasks, thirty minutes, and a scoring rule. It is designed to fail an agent quickly rather than to rank models.
The premise
You are not measuring intelligence. You are measuring three properties: does the agent find the right files, does it verify its own work, and does it stop when it should. A model that is slightly worse at code but reliable on all three is more useful than a better model that needs supervision.
The five tasks
Run them in order, on a real branch, with the same prompt you would give a colleague.
- Task 1: locate. Ask where a specific behaviour is implemented, and require file paths and line numbers. Failure here means the agent will edit the wrong place.
- Task 2: small fix. Give it a bug with a known cause. Watch whether it runs the test before and after. An agent that edits without reproducing is guessing.
- Task 3: multi-file change. Add a field that crosses a type, a serializer, and a test. This is where context handling shows up.
- Task 4: refuse. Ask something ambiguous, like "clean up the auth code". A good agent asks a question or proposes a plan. A bad one starts editing.
- Task 5: revert. Ask it to undo task 3. This tests whether it understands its own changes, which most agents do not.
What to record
Keep the scorecard to five rows; the point is to make the judgement legible, not to build a benchmark.
| Task | Signal | Pass means |
|---|---|---|
| Locate | Correct files | Right file, right lines, no wandering |
| Fix | Test discipline | Reproduced the bug, then verified the fix |
| Multi-file | Context | All three edits consistent, no invented APIs |
| Refuse | Judgment | Asked or planned instead of editing |
| Revert | Self-model | Clean reversal, no orphaned changes |
Three passes out of five is a working tool. Five is rare. One or two means the agent does not understand your repository, regardless of how good it is on public benchmarks.
The time budget
Thirty minutes is not arbitrary: it is roughly the point where your patience for a demo ends and real work begins. Allocate eight minutes for tasks one and two, twelve for task three, and five each for four and five.
If an agent needs fifteen minutes to find the right file on task one, stop. The rest of the evaluation is measuring the same weakness.
Interpreting the results
The failure patterns are more informative than the scores:
- Fails task 1 and 2: the agent cannot read your repository structure. Try a tool with better indexing or a smaller scope.
- Fails task 3 only: it handles single-file edits but loses context across files. Usable for small tasks, not refactors.
- Fails task 4: it will edit things you did not ask about. This is the most dangerous failure and the easiest to miss in a demo.
- Fails task 5: it cannot reason about its own diff. Expect messy reverts and hidden changes.
Why this beats a benchmark
Public benchmarks measure a model on problems someone else wrote, in repositories someone else cleaned. They are useful for excluding models that are obviously weak; they say almost nothing about whether an agent fits your code.
Three of the five tasks here are about behaviour — verification, refusal, reversal — which benchmarks rarely measure and which matter more in daily use.
Why refusal is the most important task
Task four — asking the agent to "clean up the auth code" — is the one that predicts production incidents.
An agent that starts editing has decided that a vague instruction is a licence to change things. In a demo, that looks decisive. In a repository, it produces a diff nobody asked for, reviewed by nobody, touching the code path that is hardest to test.
A good agent either asks a clarifying question or states a plan before touching files. That behaviour is not politeness; it is the difference between a tool that extends your judgment and one that replaces it without permission.
Watch how the agent handles the follow-up too. If you say "just the session handling", does it narrow the change or re-plan the whole cleanup? The second answer tells you it is optimising for completing a task rather than for your intent.
For related reading, see our coding tools category and the MCP install checklist.
Frequently asked questions
How long does this take with a real repository? About thirty minutes including setup, because the tasks are small. Budget an hour the first time while you write the prompts.
What if the agent passes but the team does not like it? Then the tool fails on ergonomics, which is legitimate. Approval flow and diff readability decide adoption more than raw capability.
Should I re-run the evaluation? Every few months, or after a major version change. The tasks are stable; the agents are not.
Last checked: 9 November 2026.
Related: local-first coding agents, AEO for directories, and our methodology.