Measure Whether AGENTS.md Actually Helps

Most projects have an AGENTS.md (or CLAUDE.md) these days, and it tends to grow every time the agent gets something wrong. But it's rare that anyone checks whether all those instructions are making the agent better, or just making it slower and more expensive.

Let's look at what the research says so far, then set up a simple way to run the same tasks with and without the file. That way you can find out for your own project, and work out which instructions are worth keeping.

What the research says so far

There's been a bunch of research on this over the last year, and the results don't all agree. Here are the studies worth knowing about.

Context files cost more and don't reliably help

The most cited study is from ETH Zurich (Gloaguen et al.). They ran Claude Code, Codex and Qwen Code against SWE-bench Lite, plus 138 tasks from repos that already had a developer-written AGENTS.md or CLAUDE.md.

Neither LLM-generated nor developer-written files made a statistically significant difference to success rate. Cost went up by around 20%, though, and the agents took more steps, because they followed the instructions. They ran more tests, read more files and used whatever tools the file told them about.

They also found that repository overviews didn't help agents find the relevant files any faster. Their advice is to skip LLM-generated files (the kind /init writes for you) for now, and keep hand-written ones to minimal requirements, like the specific tooling your repo uses.

But they can make runs faster

A smaller study (Lulla et al.) ran Codex on 124 real pull requests from 10 repos, with the root AGENTS.md either kept or removed. With the file, median run time dropped by about 29% and median output tokens by about 17%.

Saying that, they didn't check whether the code was correct, only that the agent made a real change. So it tells us an AGENTS.md can make an agent quicker, but not whether it makes it better.

And they help a lot when the model is missing knowledge

Vercel tested agents on Next.js 16 APIs that aren't in the models' training data. With no docs, the pass rate was 53%. With a compressed docs index in AGENTS.md (8KB of pointers to the docs files), it hit 100%.

This is a vendor eval, and they don't say which models they used. But it's the clearest case of a context file paying off, because it gave the agent knowledge it had no other way of getting.

Or they make no difference at all

A study from July 2026 ran Claude Code and Codex across 17 tasks and 288 runs, and found the context file didn't measurably change correctness for either agent. The real AGENTS.md never once turned a near miss into a pass. When the agents failed, it was down to how they implemented things, not missing knowledge about the repo.

So who's right? Probably all of them. It depends on your repo, your model and what's actually in the file, which is why it's worth testing your own.

How the test works

The idea is simple. Take some tasks you've already solved in your project, roll the code back to just before each one, and let the agent have a go with the file and without it.

Then compare four things: whether the task passed, how many mistakes the agent made along the way, how long it took and how much it cost.

Your git history is the best place to find tasks. Every commit that came with tests is a task with a known answer, so you don't have to judge the results by eye.

Pick your tasks

Pick somewhere between 5 and 15 commits. Each one should be a self-contained feature or bug fix, it should have added or changed some tests, and it should be the kind of work you'd normally hand to your agent.

For each one, create a folder with the commit hash and a prompt:

evals/
  tasks/
    coupon-expiry/
      commit
      prompt.md
    invoice-rounding/
      commit
      prompt.md

The commit file holds the hash of the real commit. The prompt describes the task the way you'd write an issue, without giving away the solution:

Coupons should stop working after their `expires_at` date. Right now, an expired
coupon can still be applied at checkout. Fix this so an expired coupon is rejected
with a validation error.

The tests from the real commit become hidden tests. The agent never sees them. Once it's finished, we copy them in and run them, which is how SWE-bench scores agents too.

Set up a clean copy for each run

Your first instinct might be to git checkout the commit before the fix and point the agent at your repo. That works, but every run ends up sharing the same folder, and the agent can see your full git history. That history includes the real fix, which the agent could find with git log --all.

Instead, let's export a fresh copy of the code for every run:

prepare() {
  local sha=$1 dir=$2

  mkdir -p "$dir"
  git archive "$sha^" | tar -x -C "$dir"

  (
    cd "$dir"
    git init -q && git add -A && git commit -qm "Start"
    composer install -q
    cp .env.example .env && php artisan key:generate -q
  )
}

git archive "$sha^" exports the files as they were one commit before the fix, with no history attached. We then create a brand new repo from them, so the agent can still use git diff on its own work, and so can we.

The last two lines are setup for a Laravel app. Swap them for whatever your project needs (npm install, copying a test database, and so on).

Remove the file for the "without" runs

For runs without the file, delete it from the copy:

rm -f "$dir/AGENTS.md" "$dir/CLAUDE.md"

It's worth knowing which file your agent actually reads before you start. Claude Code reads AGENTS.md natively from v2.1.277, but only when there's no CLAUDE.md in the project.

If you've got both and your CLAUDE.md doesn't import AGENTS.md with @AGENTS.md, Claude has been ignoring your AGENTS.md all along. Run /memory in a session to see which files it picked up.

If you've got nested AGENTS.md files in subdirectories or a .claude/rules/ folder, decide whether they're part of what you're testing and remove them too.

Keep everything else the same

Your personal ~/.claude/CLAUDE.md loads in both conditions, which is fine since it's the same both times. Auto memory is different, because Claude can save notes in one run and read them back in another. We'll switch it off with CLAUDE_CODE_DISABLE_AUTO_MEMORY=1.

You might spot the --bare flag, which skips CLAUDE.md entirely. I'd avoid it here, because it also skips your hooks, skills and MCP servers, so you'd be changing a lot more than one file.

Run the agent

Here's the function that runs Claude Code headlessly:

run_agent() {
  local dir=$1 prompt=$2 out=$3

  (
    cd "$dir"
    CLAUDE_CODE_DISABLE_AUTO_MEMORY=1 claude -p "$prompt" \
      --model claude-sonnet-5-5 \
      --output-format stream-json --verbose \
      --permission-mode acceptEdits \
      --allowedTools "Bash" \
      --max-turns 80 \
      --max-budget-usd 5
  ) > "$out"
}

There's a fair bit going on here, so let's go through it.

• p runs Claude Code without the interactive UI. -output-format stream-json writes out every message as a line of JSON, so we've got a full record of what the agent did. The last line is the result, which includes total_cost_usd, duration_ms, num_turns and token usage.
• -permission-mode acceptEdits plus -allowedTools "Bash" lets the agent edit files and run commands without stopping to ask. Since it can run anything, I'd run all of this inside a container or a throwaway VM. The model is pinned to a full model ID so both conditions definitely use the same one. --max-turns and --max-budget-usd stop a run that's gone off the rails from eating your budget.

The cost figure is a client-side estimate rather than your actual bill, but it's the same estimate for both conditions, so it's fine for comparing them.

If you're using Codex instead, codex exec --json gives you a similar stream with token usage on each turn.completed event. It doesn't include cost or duration, though, so you'll need to time the run yourself and work out cost from the token counts.

Score each run

Now for the bit that tells us how the agent did. Let's start with mistakes, since we want to look at the agent's code before we add the hidden tests.

Mistakes

A mistake here is anything that might pass the tests but you'd still flag in review. Some of these come from tools you already run, and some come from your own rules. Here's an example for a Laravel project:

mistakes() {
  local dir=$1

  (
    cd "$dir"
    git add -A

    # Static analysis errors
    vendor/bin/phpstan analyse --error-format=json --no-progress 2>/dev/null \
      | jq '.totals.errors + .totals.file_errors'

    # One of our own rules: no raw queries in app code
    git diff --cached -- app/ | grep -c '^+.*DB::raw'
  ) | paste -sd+ - | bc
}

Each check prints a number, and paste -sd+ - | bc adds them up into a single count. PHPStan checks the whole codebase, so any errors that were already there count in both conditions equally.

The second check is the interesting one. If your AGENTS.md says "never do X", write a quick grep against the agent's diff for X. That's the most direct way to see whether a rule is changing anything.

Success

Next, copy in the tests from the real commit and run the suite:

score() {
  local dir=$1 sha=$2

  for file in $(git diff --name-only --diff-filter=AM "$sha^" "$sha" -- tests/); do
    mkdir -p "$dir/$(dirname "$file")"
    git show "$sha:$file" > "$dir/$file"
  done

  (cd "$dir" && php artisan test > /dev/null 2>&1) && echo true || echo false
}

git diff --name-only lists every test file the real commit added or modified, and git show writes each one into the agent's copy. We run the full suite here, not only the new tests, so a change that breaks something elsewhere counts as a fail.

Put it all together

With those functions in place, here's the loop that runs every task, in both conditions, a few times each:

#!/usr/bin/env bash

RUNS=3
mkdir -p results

for task in evals/tasks/*/; do
  name=$(basename "$task")
  sha=$(cat "$task/commit")
  prompt=$(cat "$task/prompt.md")

  for condition in with without; do
    for run in $(seq 1 "$RUNS"); do
      dir="/tmp/agents-eval/$name-$condition-$run"
      out="results/$name-$condition-$run.jsonl"

      prepare "$sha" "$dir"

      if [ "$condition" = "without" ]; then
        rm -f "$dir/AGENTS.md" "$dir/CLAUDE.md"
      fi

      run_agent "$dir" "$prompt" "$out"
      found=$(mistakes "$dir")
      passed=$(score "$dir" "$sha")

      tail -n 1 "$out" | jq -c \
        --arg task "$name" --arg condition "$condition" \
        --argjson passed "$passed" --argjson mistakes "$found" \
        '{task: $task, condition: $condition, passed: $passed, mistakes: $mistakes,
          cost: .total_cost_usd, duration_ms: .duration_ms, turns: .num_turns,
          output_tokens: .usage.output_tokens}' >> results/scores.jsonl
    done
  done
done

Why run each task more than once? Agents aren't deterministic, and the same task can pass on one run and fail on the next. Three runs per condition is the minimum I'd go for.

Keep an eye on the numbers, though. 10 tasks, two conditions and three runs is 60 agent runs, so it's worth trying a couple of tasks first to see what a single run costs you.

Read the results

Once everything's finished, this sums up each condition:

jq -s 'group_by(.condition) | map({
  condition: .[0].condition,
  pass_rate: ((map(select(.passed)) | length) / length),
  mistakes: (map(.mistakes) | add / length),
  minutes: (map(.duration_ms) | add / length / 60000),
  cost: (map(.cost) | add / length),
  output_tokens: (map(.output_tokens) | add / length)
})' results/scores.jsonl

Be careful with the pass rate. With 30 runs per condition, a difference of one or two passes is noise. Even the 288 run study above could only rule out effects bigger than about 10 to 15 percentage points, so you're looking for big differences here, not small ones.

Rather than staring at the totals, look at which tasks flipped between conditions and open up the transcripts for those. Cost, turns and mistakes tend to be steadier than pass rate (the ETH study saw costs go up consistently), so lean on those too.

You'll usually end up in one of three places.

The file helps

You get more passes or fewer mistakes, and the extra cost is worth it to you. Keep the file, and move on to trimming it.

Same results, higher cost

This is the most common outcome in the research. The agent is reading and following your instructions, but it isn't getting anything useful out of them, so you're paying for extra turns for nothing.

Worse results

Something in the file is misleading the agent. It's usually outdated information (a command that's changed, a folder that's moved) or so many rules that the important ones get lost. Look at the transcripts for the tasks that failed, and you'll often spot the agent doing something because the file told it to.

Find which instructions are worth keeping

Anthropic's own advice for CLAUDE.md is to ask, for each line, "Would removing this cause Claude to make mistakes?" Now you've got a way to answer that properly.

The thorough approach is to split the file into sections, remove one section at a time, and re-run everything. That gets expensive fast, though, so let's start with a much cheaper check.

Did the agent use the instruction at all?

You already have every transcript, so you can check whether an instruction changed what the agent did. Here's how many times the agent ran composer test in each condition:

for condition in with without; do
  count=$(cat results/*-"$condition"-*.jsonl \
    | jq -r 'select(.type == "assistant") | .message.content[]?
      | select(.type == "tool_use" and .name == "Bash") | .input.command' \
    | grep -c 'composer test')
  echo "$condition:$count"
done

This pulls out every shell command the agent ran and counts the ones that match. If the number is about the same in both conditions, the agent is working it out from your repo anyway (it's probably sitting in composer.json), and that line is doing nothing.

Non-obvious instructions are a different story. In the ETH study, agents used uv about 1.6 times per task when the file mentioned it and almost never when it didn't. So it's the instructions about things the agent wouldn't find on its own that make a difference.

What's usually worth keeping

Based on the research and your own results, these are usually worth having in the file:

1. Commands the agent can't guess, with the exact flags. How to run the tests, how to run a single test, how to lint.
2. Tooling and environment quirks. "Use pnpm, not npm", or "the feature tests need Redis running".
3. Conventions that differ from your framework's defaults. If you do things the way the Laravel docs do, the model already knows.
4. Knowledge the model doesn't have, like a framework version newer than its training data or your own internal packages. Vercel's result suggests a short index pointing at the docs works well here.

What's usually worth cutting

And these are the ones I'd test first, expecting to remove them:

1. Project overviews and directory tours. The ETH study found they didn't help agents find files, and Augment Code reported an architecture overview made their agent read a load of irrelevant docs and do worse.
2. Anything that's already in your README or docs. When the ETH researchers removed a repo's other docs, LLM-generated files started to help, which suggests they were mostly repeating what was already there.
3. Generic advice like "write clean code" or "follow best practices".
4. Long lists of things not to do. Augment found that files with 30+ warnings made their agent take twice as long. Both of Augment's results are from an internal eval, so treat them as a hint, not a rule.
5. Rules that must always be followed. CLAUDE.md is context, not configuration, so a hook or a linter will do this job much more reliably than a line in a file. Once you've cut the obvious ones, re-run the eval against the trimmed file. If the results hold up and the cost drops, keep the trimmed version.

Wrapping up

I'd start small, with five or so tasks and three runs each, just to see whether the file changes cost and turns at all. It won't tell you everything, but it's usually enough to spot a file that's costing you money for no benefit.

Once the script exists, you can also re-run it every time someone adds a line to AGENTS.md, so you'll know whether it's doing anything.