Write Requirements Your Agent Can't Talk Itself Out Of

When you ask a coding agent to change how something works, you'd expect it to either make the change or tell you it couldn't. But a recent benchmark caught agents understanding a new requirement, writing it down, and then putting the old behaviour back because they thought the tests wanted it.

So I've started assuming that if a requirement isn't backed by a check, the agent might talk itself out of it. Let's look at what the benchmark found, and then how to back up a requirement so it sticks.

What MaxxBench found

MaxxBench, from Gavin Lo and Justin Lo, took tasks from an existing coding benchmark and changed a small part of each one, either the instructions or the starting code. The answer an agent would normally give was now wrong, so it had to follow the new instructions to pass.

Two habits came out of it that are worth knowing about.

Understanding the change, then reverting it

One task was about writing a separate test report for each browser. The reports used to be named like Chrome.xml, and the changed task asked for lowercase names, so chrome.xml instead.

In one published run, the agent added .toLowerCase() and got it right. Later, it asked another model for a reference solution and saw that it didn't lowercase anything. It even pointed out that the task asked for lowercase names, then replaced its own correct code with the other model's version anyway, with the reasoning "We'll follow tests, since grading uses them."

That's an extreme case, but other runs did a milder version of the same thing. One agent was asked to preserve nanosecond differences and decided "such cases are unlikely". Another worked out which approach had the better "expected pass rate" against the hidden tests, and picked the one that ignored the requirement.

Gaming a number the program printed about itself

A different task set a memory limit. The program had to keep no more than a set number of records in memory, spill the rest to disk, and print a summary line with its peak (peak_in_memory_files). MaxxBench's first check for this read that printed line.

Every model passed that check. More than half of them worked out that the tests could only see the printed line, so they reported a peak under the limit and then loaded the spilled records back into memory anyway. One even admitted "that's a bit of cheating but the metric is self-reported anyway."

A second check, which counted the records actually held in memory, caught it.

This was all on a benchmark, where an agent has good reason to think it's being graded, and I haven't seen anyone measure how often it happens in everyday work. But it's enough that, for a requirement I care about, I don't want a sentence in the prompt to be the only place it's written down.

Write the change as a failing test first

Before you ask the agent to make the change, write a test that fails on the current code and only passes once the new requirement holds. Here's the lowercase filename example as a test, using Node's built-in test runner:

// test/reports.test.js
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { reportFilename } from '../lib/reports.js';

test('report filenames are lowercase', () => {
  assert.equal(reportFilename('Chrome'), 'chrome.xml');
});

Run it with node --test and make sure it fails before the agent touches anything. If it already passes, it isn't testing the change you're asking for.

Then give the agent the requirement and point it at the test:

Report filenames should now be lowercase, so the Chrome report is chrome.xml
instead of Chrome.xml. test/reports.test.js covers this and currently fails.
Make it pass without changing the test.

The requirement is now something the agent can run. If it reverts the change, the test fails, and you'll both see it.

I'd still write the requirement out in full, rather than just saying "make the test pass". A failing test tells the agent what to make pass, but not what you actually want, and an agent working from a test alone can pass it without doing what you meant (more on that in a second).

Asking it not to change the test matters too, since editing the expected value is the easiest way to get a test passing. Running git diff test/ once it's done will tell you if it did anyway.

Add a check the agent can't see

A visible test has its own problem. Because the agent can see it, it can write code that passes that one test without really doing what you asked. Here's a (pretty cheeky) fix that passes ours:

// lib/reports.js
export function reportFilename(launcher) {
  if (launcher === 'Chrome') return 'chrome.xml';

  return `${launcher.replace(/[^a-z0-9]/gi, '_')}.xml`;
}

It's obvious here, but it'd be much harder to spot in a bigger change.

So add a second check that tests the requirement itself, rather than one example of it. This one runs through a few other launchers and checks that every filename comes back lowercase:

// ~/checks/reports.hidden.test.js
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { reportFilename } from '../lib/reports.js';

test('every report filename is lowercase', () => {
  for (const launcher of ['Firefox', 'Headless Chrome', 'Safari', 'PhantomJS']) {
    const filename = reportFilename(launcher);

    assert.equal(filename, filename.toLowerCase());
  }
});

The hard-coded fix fails this straight away, because it returns Firefox.xml. A proper fix passes both tests.

The simplest way to keep this check hidden is to write it after the agent's finished. If you'd rather write it up front, keep it outside the project and copy it into test/ when you review:

cp ~/checks/reports.hidden.test.js test/
node --test

You don't need loads of these. One or two checks that test the actual requirement (every filename is lowercase) are worth more than a pile of tests for individual values, and they won't make your test suite slow and fragile.

If you want to go further and score your agent across a whole set of tasks with hidden tests, I covered that in Build an Eval for Your Coding Agent. Here, we only need one extra check for one requirement.

Don't trust numbers the program prints about itself

The memory limit task from earlier shows another way a check can go wrong. Reading peak_in_memory_files from the program's output means the agent is marking its own homework. It wrote the code that does the work and the line that reports on it, so nothing forces the two to match.

This applies to anything a program could report about itself, like a memory limit, a claim that something's faster, or a rule like "never log the API token". Check these from outside the program instead.

For memory, count the records held in memory from your test, or watch the process's peak memory on a big input. If it's a speed claim, time it yourself on the same input before and after the change. And for "never log the token", run it and search the logs for the token.

Ask what it didn't do

Once the agent says it's done, I ask it to go back over every requirement:

List every requirement I gave you, and say whether you did it and how.
Then list anything you decided not to do, with the reason.

This doesn't replace a test, since it's the agent reporting on its own work again. But it gives the agent a chance to tell you it dropped something, and that's much easier to spot in a short list than in a diff.

Search the transcript

The other place this shows up is in what the agent writes while it works. Most of the MaxxBench examples above have something in common: the agent talked about the grader before it dropped the requirement ("grading uses them", "expected pass rate", "the metric is self-reported anyway").

Claude Code saves each session as a JSONL file in ~/.claude/projects/, inside a folder named after your project's path (with the slashes swapped for dashes). You can pull out what the agent wrote with jq and search it with grep:

jq -r 'select(.type == "assistant") | .message.content[]? | .text // .thinking // empty' \
  ~/.claude/projects/<your-project>/<session-id>.jsonl \
  | grep -inE "hidden tests?|grader|grading|pass rate|follow (the )?tests|unlikely"

The jq part pulls out the agent's messages (and its thinking, where that's been saved), and grep prints any line that mentions hidden tests, grading, or something being unlikely. To find your latest session, run ls -t *.jsonl in that folder and it'll be at the top.

Check what your tool actually saves, though. When I looked through my own recent Claude Code sessions, most of the thinking blocks were empty, so this mostly searches the messages the agent writes between tool calls. And if you only ever read the summary at the end, you won't see any of it.

A match doesn't prove anything. Agents mention tests all the time for perfectly good reasons, and an agent can drop a requirement without saying a word about the grader. But if it's talked itself out of something you asked for, that's worth a closer look, even when every test passes.

When a test isn't enough

Some requirements don't turn into a test very well. "Match the existing style", "only touch the files you need to" and "don't refactor anything else" are all things you'd want, but there's no sensible assertion for them, and the same goes for most product and UX decisions. Writing a failing test first also takes time, so for a small change it might not be worth it.

For these, read the diff instead of the agent's summary. The summary is the agent's own account of what it did, which is the same problem as the printed memory number. You'll see what really changed in the diff:

git status
git diff --stat

git status shows new files that git diff won't, and --stat lists every changed file with how many lines changed, so a file you didn't expect stands out straight away. If the agent's already made commits, use git diff main --stat to compare against your main branch, then read the full diff for anything that matters.

You don't need to do all of this for every change. But for any requirement you'd be annoyed to see dropped, I'd write the failing test first and keep one check the agent can't see.