This is the first finding from firstrun, a harness that gives a coding agent its first run at a library and grades the result with a hidden test. The repo has every transcript and a one-command reproduce.

What I did

I gave a coding agent (Claude Code, headless, in a clean Docker container with nothing but Python, Node, and network access) one realistic integration task at a time, then graded the result with a hidden, deterministic test the agent never sees. Every command, tool call, web fetch, and error the agent hit is recorded. I ran two models (claude-haiku-4-5-20251001 and claude-sonnet-5, the ids as they appear in the transcripts) against five libraries in nine task variants: 63 graded runs, about $12 in API spend.

The tasks were the kind of thing a developer actually asks for: “add JWT auth to this DRF project with simplejwt”, “move this FastAPI service from dicts to SQLAlchemy asyncio”, “add Pydantic v2 settings and event validation”, “expose this catalog module as an MCP server”, “build a small Starlette service with a startup hook, routes, a middleware, and a 404 handler”.

Finding 1: the agents never opened the documentation

Across all 63 runs there were zero WebFetch or WebSearch calls. Not one. Both tools were available: every transcript’s init event lists them among the agent’s tools, so this is a choice the agent made, not a capability it lacked.

That includes:

• a library released after the models’ training data (MCP Python SDK 2.0, 2026-07-28), where the agents’ memory was guaranteed wrong;
• runs where the library’s own error message contained the migration guide URL;
• a variant where the task prompt itself linked the documentation site (six runs, still zero fetches).

What the agents read instead, when memory failed: the error message, then the installed package. dir(), inspect.signature, inspect.getsource, pip show, cat and grep in site-packages. One Haiku run, unable to find the high-level API, built the whole server on the low-level class it could introspect, in 43 turns.

For a maintainer this means the documentation an agent consumes is the wheel: the install name, the exceptions, the signatures, the docstrings, the exported names, and how readable the source is. The website is off the path.

Finding 2: a breaking release has a measurable cost even when every run passes

Same task, same prompt, same grader, three conditions for the MCP SDK. The only difference is which version pip resolves, controlled by a hidden constraint file.

Sonnet 5 (claude-sonnet-5)	pass	turns (mean ± sd)	cost/run	wall time
mcp 1.x (the version the model remembers), n=5	5/5	14.0 ± 3.2	$0.13	60 s
mcp 2.x, as shipped, n=5	5/5	20.0 ± 1.8	$0.23	92 s
mcp 2.x, prompt links the docs, n=3	3/3	22.0 ± 1.6	$0.28	140 s
mcp 2.x, helpful import error replaced by a plain one, n=5	5/5	26.0 ± 4.0	$0.51	124 s

The pass rate never moves. The release costs each integration about six extra turns and nearly twice the money.

Twelve cents is not the point. Turns are. Every extra turn is another guess at an API that doesn’t exist, another command run against a half-working server, another chance to ship the wrong thing while still reporting success. The dollar column is just the turn column priced. Haiku shows the same shape at higher turn counts (24.7, 34.3, 42.0) with a noisy 1-in-3 pass rate that has nothing to do with the release: it fails the same way on 1.x, by never actually speaking the protocol to its own server.

Finding 3: one error message is worth about six turns

mcp 2.x renamed FastMCP to MCPServer and moved the module. Importing the old path raises a ModuleNotFoundError whose text names the new import and links the guide. Every Sonnet run wrote the old import first, hit that message, and used the new import from the message. None followed the link.

The last row of the table ablates that message: a hidden import hook makes the old path fail with Python’s plain “No module named” and neutralizes the stub on disk so there is nothing to find. Sonnet goes from 20.0 turns to 26.0 and from $0.23 to $0.51 per run (five runs each), because its fallback is reading source, which is slow and token-heavy. Haiku goes from 34.3 to 42.0 (three runs each).

That message is the single most valuable thing in the package for agent users, and this is the first measurement of what it’s worth.

The same thing shows up on Starlette 1.0 (released 2026-03-22), which removed @app.on_event, @app.route, on_startup=, and friends. On the pinned 0.x version Sonnet passes in 8.3 turns; on 1.x it passes in 12.7, and two of three runs hit TypeError: unexpected keyword argument 'on_startup' first and recovered. That message is Python’s generic unknown-keyword error, so it names the keyword but not the replacement; Starlette would have to catch the removed arguments explicitly to point at lifespan=. Whether that’s worth doing is the maintainer’s call, and this harness can measure what it buys.

Finding 4: the agent copies the project’s pin style, and that can quietly keep users on your old major

This one started as a mistake. The Starlette workspace’s starting requirements.txt contained uvicorn>=0.30,<1.0. On the 1.x task, Haiku wrote starlette>=0.40,<1.0 in five of five attempts, installed 0.52.1, used the removed decorator API, and passed without ever seeing 1.0. My first reading was “the weaker model avoids the new major from memory.” A reviewer pointed out the line is the uvicorn pin with the name swapped.

The control settles it: with the starting file changed to a bare uvicorn, Haiku installed 1.6.0 in three of three runs, hit 'Starlette' object has no attribute 'route' (or the on_startup TypeError) in every one, recovered, and passed at 32 to 44 turns. No avoidance. The agent was imitating the pin convention it saw.

That’s a smaller claim, and it isn’t a lever the maintainer holds, since the pin lives in the user’s project. It is still countable. A <1.0 upper bound copied from a neighboring line is how a project ends up quietly pinned to your previous major, with the old idioms working and no error ever raised, and it’s one way a release can look unadopted through no fault of its own. The harness records it as a flag (pinned-old-major) so it’s countable per run.

Finding 5: the failures that do happen are usually the model’s verification habit

Every Haiku failure on the MCP task, on either version, came from the same thing: it “tested” its server with timeout 2 python server.py and declared success when the process didn’t crash in two seconds. It never spoke the protocol. Sonnet always did. A benchmark that blames the library for that would be wrong; the report separates the two.

“Model memory will catch up, so this isn’t my problem”

Per release, that’s true: the next model trained after your major ships will know it, and the tax goes to zero for that model. For a project that ships majors it is false, because the tax comes back with every one, and it lands during exactly the months when you most want people on the new version. The docs can’t reach the agents in that window. The error message can, and it’s the one lever in this whole story that the maintainer controls unilaterally.

What this is for

For a coding agent, the traceback is the quickstart.

If you maintain a Python library, the number you can’t currently see is what your last release cost the agents integrating you. This harness gives you that number per release, along with the classes of error the agents hit and recovered from, the stale idioms they still use, and whether they ended up pinned to your previous major without ever seeing the new one. All of it comes from execution, not from linting your docs.

Caveats

• Two models, one agent harness. Other harnesses may read docs; this one didn’t, and it’s the one a large share of developers use.
• Three runs per cell except where stated. Turn counts vary about 30% within a cell; the five-run MCP deltas are the ones to quote.
• Rows aren’t comparable to each other (task difficulty differs). Only compare within a row across versions or changes.
• The first ablation attempt leaked: agents found the helpful text on disk by reading the stub file. Those runs are excluded (kept under runs/_contaminated/) and the rerun neutralized the file. Agents will read anything unusual in site-packages.
• Four Haiku runs were excluded as harness errors: the agent ran pkill -f "uvicorn app:app" and killed its own process, because the harness passed the prompt on the command line. Fixed (prompt on stdin); replacements were run.
• One grader check I wrote asked for behavior the prompt never specified (4xx on an invalid URL). It was removed and the affected runs re-graded from their recorded checks. The grader is the spec; a check the prompt doesn’t state is a bug in the benchmark, not the library.

The ask

If you maintain a Python library that shipped a major in the last year, open an issue with the library and the one integration you’d most want to work first-try. I’ll write the task and the hidden grader, run it against the previous major and the current one, and post the before-and-after in the issue. Private libraries and paid SDKs are especially interesting, because for those the agents’ memory never catches up.

Reproduce

git clone https://github.com/caseydm/firstrun && cd firstrun && uv sync
echo "ANTHROPIC_API_KEY=..." > .env
uv run firstrun build
uv run firstrun check mcp-server          # proves the grader without spending tokens
uv run firstrun run mcp-server --model sonnet --runs 3
uv run firstrun run mcp-server-v1 --model sonnet --runs 3
uv run firstrun report