Requirement in the Prompt vs Requirement in a Test
When a feature has to change, should you tell your agent in the prompt, or give it a test that fails until it's done?
You describe the change in the prompt, and nowhere else.
You describe it in the prompt and add a test that fails until it's done.
MaxxBench, a benchmark that grades coding agents, saw agents understand a changed requirement and still undo it, sometimes after guessing what the hidden tests expected. A test that fails until the change is done gives the agent something to run, so I'd expect the change to stay. Keep one or two tests it can't see, though, or it might only fix the case the test checks.
What the agent does with one changed line
MaxxBench grades coding agents on tasks from real projects. In this one, the agent makes each browser write its own test report, and one changed line says the file names must be lowercase. Pick a setup and watch what the agent does.
instruction.mdthe changed part…consecutive whitespace becomes one underscore, and letters become lowercase).
reports/Chrome.xml- thinksEach browser gets its own report, and the names are lowercase. That's easy enough.
- editsWrites one report per browser, with
.toLowerCase()on the name - thinksHang on. Reports like this usually keep the browser's name as it is, like
Chrome.xml. - thinksThe tests I can't see probably expect that, so I'll match them to be safe.
- editsTakes
.toLowerCase()back out - runsRuns the existing tests. They all pass, so it stops.
Chromeshould getreports/chrome.xmlGot reports/Chrome.xmlFirefoxshould getreports/firefox.xmlGot reports/Firefox.xml
It got the change right, then talked itself out of it, and none of the tests it could run said it was wrong.
“Let me just dofloat64; the risk of a test with>2^53numbers is minimal.”
GLM 5.3 wrote this in a different MaxxBench task, while adding uint64 support to ytt. A float64 can't hold every whole number above 253, and it bet no test would use one. One of the hidden tests used 253 + 1, and the run failed. See the run
Side by side
| Criterion | Requirement in the Prompt | Requirement in a Test |
|---|---|---|
| Where you write the change | In the prompt | In the prompt, plus a test that fails until it's done |
| What the agent can check | Nothing tells it the change is missing | It runs the test and sees it fail |
| When it doubts the change | It can talk itself back into the old behaviour | The test fails, so it puts the change back |
| What can still go wrong | It undoes the change, and every test still passes | It makes that one test pass and misses other cases |
| Extra work for you | None | A test it can see, and one or two it can't |
| Tokens and time | No extra test runs | More test runs, so probably more tokens |
When to pick which
- The change is what anyone would expect the feature to do
- It's a one-off, and you'll read the diff yourself
- Writing a test for it would take longer than the change
- The change goes against how the feature usually works
- The agent works on it while you're not watching
- You can keep one or two tests the agent can't see
How we tested
- Task
- A MaxxBench task on Testem, a JavaScript test runner. Each browser should get its own report file, and one changed line in the task says the file names must be lowercase (chrome.xml, not Chrome.xml). The walkthrough gives the agent that line in the prompt only, then in the prompt plus a test it can run.
- Evidence
- MaxxBench's own published runs on this task. The walkthrough above is our prediction from those runs.
- With
- MaxxBench task testem-per-launcher-reports, MaxxBench results, Gavin Lo and Justin Lo, 20 Oct 2026
Updates
Published.