Choosing an assistant without reading a benchmark

The three properties that decide which one you open daily, and why every leaderboard is the wrong input.

Level 17 minUsing agents

Every assistant on the market is good at summarising an article and answering a factual question. That is not a useful differentiator, because it is not what you will use it for on a Tuesday afternoon.

The three properties that matter

**What it can reach.** A model that cannot see your files, your repository, or your calendar is a smarter search box. The one that can reach your work is the one you will actually open.

**How it behaves over length.** Short prompts make every model look equal. The differences appear at the 20th turn of a long task, where one keeps the thread and the other starts inventing requirements.

**How it fails.** All of them fail. The question is whether the failure is visible — a wrong citation, a diff that does not apply — or invisible: confident prose that reads correct.

Why benchmarks do not help

Benchmarks measure a model on tasks someone else wrote, in a format the model has seen. Your work is not that. A one-point difference on a leaderboard has never once predicted whether an agent will handle your monorepo.

Worse, benchmark-chasing produces a specific failure: models tuned to be good at exams become fluent at the shape of an answer without doing the work. The tell is a response that sounds right and cannot be checked.

A ten-minute test instead

Take one real task you did last week — one with a checkable outcome. Run it on two assistants. Compare the result, not the experience: did the file parse, did the query return the right number, did the summary miss a section.

Then run a task where the correct answer is "I do not know". Only one thing separates a useful assistant from a liability here: whether it says so or invents something plausible.

Where the differences actually live

Our comparisons track two axes: what the tool is built around (editor ownership, browsing, hosting) and what it costs at the entry tier. Those are the properties that decide a subscription. The assistant category lists the current entrants with a line about who each one suits.

Next: prompts that survive contact with reality.

Prompts to practice with