Build an Eval for Your Coding Agent

Most of us change something about our coding agent's setup, like switching models or adding an instruction, and then decide whether it helped based on how the next few tasks felt.

An eval gives you a proper answer. It's a handful of real tasks from your project, run the same way every time and scored by a script, so you can change one thing and see whether the numbers move.

Let's build one for Claude Code, run it, then change the model and see what happens.

What we're building

We'll pick five tasks from a real project and give each one a prompt and some hidden tests. Then we'll write a few small shell scripts. run.sh hands each task to the agent in a fresh copy of the project, score.sh checks what it did, and summary.sh turns everything into a table.

Here's how it's laid out:

my-app/                    # your project
evals/
├── tasks/
│   ├── fix-percent-coupons/
│   │   ├── prompt.md
│   │   └── tests/
│   ├── product-search/
│   └── ...
├── config.sh
├── check.sh
├── run.sh
├── score.sh
├── summary.sh
└── results/

The evals live next to the project rather than inside it. Anything inside the project is something the agent can read, and we don't want it reading the tests we're going to mark it with.

None of this depends on your language or framework. Everything specific to your project (how to install it, how to run the tests, which linter you use) goes in config.sh, and the other scripts only use bash, git and jq. The agent is a single command in run.sh too, so you could swap Claude Code for codex exec or any other agent that runs without you sitting there.

I tested all of this on a small shop app with products, coupons and some code that works out basket totals, and the numbers later on come from that. Your project will be bigger, and that's fine.

Pick a handful of real tasks

Five tasks is plenty to start with. The best place to find them is your own git history, so look back through the last month of commits or closed issues and pick work you'd hand to an agent. Bug fixes are perfect, since the commit before the fix is your starting point and the tests that came with the fix tell you whether it worked.

Try to get a mix of bug fixes, small features and changes that touch the database. For the shop, I picked five.

There's one bug fix (fix-percent-coupons), where percentage coupons are being taken off as a fixed amount, so a 10% coupon only takes off 10p. Two tasks change things that already exist, adding a search parameter to the products endpoint (product-search) and fixing validation so duplicate SKUs don't cause a 500 error (product-validation).

The other two add something new. low-stock-command is a new console command that lists products running low, and discontinue-products needs a new database column. That one gives the agent a chance to make a classic mistake, which is editing a migration that already exists instead of adding a new one.

Each task gets a prompt.md, written the way you'd normally type it. Don't polish these into a perfect spec, because you want to know how the agent copes with your normal prompts. Here's the coupons one:

Customers are reporting that percentage coupons take off the wrong amount. A 10% coupon on a £50 basket only takes 10p off. Fix it so percentage coupons work properly.

Define what success looks like

"Did it work?" has to be something a script can answer. For every run, I'm checking three things.

The hidden tests pass

Each task has its own tests in its tests folder, written with whatever test framework your project already uses. The agent never sees them, because they're only copied into the project after it's finished.

The coupons task has three. One puts a £12.50 product in the basket with a 20% coupon and expects a total of £10.00. Another checks a 100% coupon makes the basket free, and the last one makes sure fixed amount coupons still work.

Why hide them? If the agent can see the tests it's being marked on, it can get them passing in ways you'd never accept, like special-casing the values in the test.

When you write them, only test what the prompt actually asks for. I used 20% of £12.50 on purpose, because it comes out as a whole number of pence, and the prompt never said how to round. If I'd tested 15% of £9.99, I'd be marking the agent on a rounding decision I never told it about.

The existing test suite still passes

Fixing coupons and breaking basket totals in the process isn't a success. So we run the project's own tests too, but with the original test files put back first. Otherwise the agent could get the suite passing by "fixing" a test.

No obvious mistakes

Some things would get a pull request sent back even if every test passes. I'm counting two kinds. The first is editing files that should never change once they exist, like migrations or lockfiles. The second is code that doesn't pass your linter or formatter.

These are counted separately from success. A run can pass every test and still make a mistake, and it's useful to see both.

Tell the scripts about your project

Everything that's specific to your project goes in config.sh. Fill in each line for your own stack (the comments have a few examples):

# config.sh: everything that's specific to your project.
# The commands run inside a fresh copy of your project, and $PROJECT points at the original.

# Your project, relative to this folder
PROJECT_DIR=""        # e.g. ../my-app

# Gets a fresh copy ready to run, like installing dependencies and copying in an .env file
SETUP_CMD=""          # e.g. npm ci, or: pip install -r requirements.txt, or: cp "$PROJECT/.env" . && composer install

# Runs your whole test suite, and exits with a non-zero code if anything fails
TEST_CMD=""           # e.g. npm test, or: pytest, or: go test ./...

# The folder your tests live in (it's put back to how it was before the suite runs)
TESTS_DIR=""          # e.g. tests

# Where the hidden tests get copied, and a command that runs just those
HIDDEN_TESTS_DIR=""   # e.g. tests/eval
HIDDEN_TEST_CMD=""    # e.g. npx vitest run tests/eval, or: pytest tests/eval

# Optional: a linter or formatter check, and a regex for which changed files to pass it
LINT_CMD=""           # e.g. npx eslint, or: ruff check, or: npx prettier --check
LINT_FILES=""         # e.g. '\.(js|ts)$', or: '\.py$'

# Optional: a regex for existing files the agent should never edit
PROTECTED_PATHS=""    # e.g. '^migrations/', or: '^package-lock\.json$'

SETUP_CMD is the one most likely to need some thought. A git clone only copies what's committed, so anything your project needs that isn't in git (installed dependencies, an .env file, a local database) has to be set up here. You can chain commands with &&, and use $PROJECT to copy files from your original project.

Use single quotes for any value that mentions $PROJECT. That way, the variable gets filled in when the command runs rather than when the config is loaded.

Check your checks

Before an agent goes anywhere near this, make sure the hidden tests do what you think. Every one of them should fail on the untouched project, and pass once you've solved the task yourself. Here's check.sh, which handles the first part:

#!/usr/bin/env bash
# Usage: ./check.sh   Runs each task's hidden tests against an untouched copy of the project
set -uo pipefail

EVALS=$(cd "$(dirname "$0")" && pwd)
source "$EVALS/config.sh"
PROJECT=$(cd "$EVALS/$PROJECT_DIR" && pwd)

dir=$(mktemp -d)
git clone --quiet "$PROJECT" "$dir"
cd "$dir"
eval "$SETUP_CMD"

for task in "$EVALS"/tasks/*/; do
  mkdir -p "$HIDDEN_TESTS_DIR"
  cp -R "$task"/tests/. "$HIDDEN_TESTS_DIR"/

  if eval "$HIDDEN_TEST_CMD" > /dev/null 2>&1; then
    echo "$(basename "$task"): passes (it shouldn't yet)"
  else
    echo "$(basename "$task"): fails"
  fi

  rm -rf "$HIDDEN_TESTS_DIR"
done

echo "The untouched copy is at$dir"

It clones the project, sets it up using your SETUP_CMD, then copies in each task's hidden tests one at a time and runs them. Make it executable with chmod +x check.sh and run it. Every task should say fails.

If one of them passes, the test isn't checking what the task asks for. It's also worth reading one of the failures yourself (run your HIDDEN_TEST_CMD in that copy), to make sure it fails for the right reason and not because of a typo.

For the second part, use the copy it leaves behind. Solve each task yourself, copy the hidden tests into your HIDDEN_TESTS_DIR and run them. This is worth the twenty minutes, because one of my tests failed on my own correct solution.

That was the low stock command. Its test checked the output for the product's name and then for its SKU, but the test framework stops looking at a line of output once it's matched something on it. The name and SKU were on the same row of the table, so the second check could never pass. Without checking, I'd have marked every agent as failing that task and blamed the agent.

Running the agent without you

Claude Code's -p flag runs a single prompt and exits, which is what we need. This is the command we'll use for every run:

CLAUDE_CODE_DISABLE_AUTO_MEMORY=1 claude -p "$(cat prompt.md)" \
  --output-format json \
  --setting-sources project \
  --strict-mcp-config \
  --no-session-persistence \
  --permission-mode acceptEdits \
  --permission-prompts none \
  --allowedTools "Bash" \
  --settings '{"sandbox": {"enabled": true, "allowUnsandboxedCommands": false, "failIfUnavailable": true}}' \
  --model sonnet < /dev/null

There's a lot going on, so let's take it a flag at a time.

Getting numbers back

• -output-format json returns the agent's final message along with the cost, duration, number of turns and token usage. That covers most of what we want to measure.

Keeping your own setup out of it

The eval should test the project's setup, so anything in your personal ~/.claude folder needs to stay out of it. --setting-sources project loads the project's settings and instruction files (CLAUDE.md or AGENTS.md) but skips your user ones.

CLAUDE_CODE_DISABLE_AUTO_MEMORY=1 turns off auto memory, which would otherwise let notes from one run leak into the next. And --strict-mcp-config means no MCP servers load unless you pass them in, including any claude.ai connectors you're signed into.

• -no-session-persistence is just tidiness. Without it, every eval run ends up in your /resume list.

Letting it work unattended

Nobody's around to approve anything, so --permission-mode acceptEdits lets the agent edit files and --permission-prompts none denies anything that would have asked you.

That leaves shell commands, which is where the --settings line comes in. It turns on Claude Code's sandbox, so the operating system only lets the agent's commands write inside the project copy and blocks the network. allowUnsandboxedCommands: false stops the agent retrying a blocked command outside the sandbox, and failIfUnavailable refuses to start at all if the sandbox can't (rather than carrying on without it).

With that in place, --allowedTools "Bash" lets the agent run any command without asking, because every one of them still runs inside the sandbox.

I didn't get this right first time. My first attempt allowed the project's test command and nothing else, which seemed sensible. Every task passed, but when I looked at permission_denials in the JSON, four out of five runs had tried to write some files and run the tests in a single Bash command.

That got denied, and all four agents finished without running the tests at all. Their final messages even said so ("I haven't run the tests"), but I wasn't reading them yet.

Switching on the sandbox fixed most of it, but two thirds of the runs still had one denial. The sandbox approves most commands automatically, but not long, complicated ones, and with nobody there to ask, they get denied. Adding the Bash allow rule sorted that out, and I checked the sandbox still holds by asking the agent to touch a file in my home directory (it got Operation not permitted).

Neither problem showed up in the pass rate, so check the denials after your first run, even if everything passes.

Closing stdin

< /dev/null is there because, without it, Claude Code waits three seconds for something to arrive on stdin and prints a warning about it. It doesn't break anything, but it adds three seconds to every run.

The run script

Now let's wrap that command in run.sh. For every task, it'll run the agent a few times, each in a fresh copy of the project:

#!/usr/bin/env bash
# Usage: ./run.sh <label> [extra claude flags...]
set -uo pipefail

EVALS=$(cd "$(dirname "$0")" && pwd)
source "$EVALS/config.sh"
PROJECT=$(cd "$EVALS/$PROJECT_DIR" && pwd)
LABEL=$1; shift
ATTEMPTS=${ATTEMPTS:-3}
RESULTS="$EVALS/results/$LABEL"
mkdir -p "$RESULTS"

for task in "$EVALS"/tasks/*/; do
  name=$(basename "$task")

  for attempt in $(seq 1 "$ATTEMPTS"); do
    echo "[$LABEL]$name #$attempt"

    # A fresh copy of the project for every run
    dir=$(mktemp -d)
    git clone --quiet "$PROJECT" "$dir"
    (cd "$dir" && eval "$SETUP_CMD")

    # Drop in any extra files for this variant
    if [ -d "$EVALS/variants/$LABEL" ]; then
      cp -R "$EVALS/variants/$LABEL/." "$dir/"
      git -C "$dir" add -A
      git -C "$dir" commit --quiet -m "Variant:$LABEL"
    fi
    start=$(git -C "$dir" rev-parse HEAD)

    (
      cd "$dir"
      CLAUDE_CODE_DISABLE_AUTO_MEMORY=1 claude -p "$(cat "$task/prompt.md")" \
        --output-format json \
        --setting-sources project \
        --strict-mcp-config \
        --no-session-persistence \
        --permission-mode acceptEdits \
        --permission-prompts none \
        --allowedTools "Bash" \
        --settings '{"sandbox": {"enabled": true, "allowUnsandboxedCommands": false, "failIfUnavailable": true}}' \
        "$@" < /dev/null > "$RESULTS/$name-$attempt.json"
    )

    "$EVALS/score.sh" "$dir" "$start" "$task" "$RESULTS/$name-$attempt.json" \
      "$LABEL" "$name" "$attempt" >> "$RESULTS/results.jsonl"

    rm -rf "$dir"
  done
done

What's happening here? For each attempt, we git clone the project into a temporary directory and run your SETUP_CMD inside it. Cloning only copies what's committed, so commit (or stash) your work before you run anything.

The variants block copies extra files into the project before the agent starts, and commits them so they don't count as the agent's changes. If you put a CLAUDE.md in variants/with-claude-md/ and run with the label with-claude-md, that run gets the file and nothing else changes. That's handy when the thing you want to test is a file rather than a flag.

We record the start commit before the agent runs, so we can see what it changed afterwards. Then the agent runs, anything after the label (like --model sonnet) gets passed straight through to claude, and the JSON lands in results/<label>/.

The scoring script

score.sh takes the copy the agent has just worked in and checks it against everything we decided on earlier:

#!/usr/bin/env bash
# Usage: ./score.sh <dir> <start-commit> <task-dir> <agent-json> <label> <task> <attempt>
set -uo pipefail

source "$(dirname "$0")/config.sh"
dir=$1; start=$2; task=$3; agent_json=$4; label=$5; name=$6; attempt=$7
cd "$dir"

# What did the agent change?
git add -A
changed=$(git diff --cached --name-status "$start")

# Mistake: editing or deleting a file that already existed in a protected path
protected_edits=0
if [ -n "$PROTECTED_PATHS" ]; then
  protected_edits=$(echo "$changed" | awk '$1 ~ /^[MD]/ { print$2 }' | grep -cE "$PROTECTED_PATHS")
fi

# Mistake: files it added or changed don't pass the linter
lint_failed=0
lint_files=$(git diff --cached --name-only --diff-filter=AM "$start" | grep -E "${LINT_FILES:-.}" | tr '\n' ' ')
if [ -n "$LINT_CMD" ] && [ -n "$lint_files" ] && ! eval "$LINT_CMD$lint_files" > /dev/null 2>&1; then
  lint_failed=1
fi

# Put the original tests back, so the agent can't pass by editing them
git checkout --quiet "$start" -- "$TESTS_DIR"

# Does the test suite still pass (plus any new tests the agent wrote)?
if eval "$TEST_CMD" > /dev/null 2>&1; then suite=true; else suite=false; fi

# Do our hidden tests pass?
mkdir -p "$HIDDEN_TESTS_DIR"
cp -R "$task"/tests/. "$HIDDEN_TESTS_DIR"/
if eval "$HIDDEN_TEST_CMD" > /dev/null 2>&1; then hidden=true; else hidden=false; fi

# If the agent crashed, there's no JSON to read
[ -s "$agent_json" ] || echo '{}' > "$agent_json"

jq -c \
  --arg label "$label" --arg task "$name" --argjson attempt "$attempt" \
  --argjson hidden "$hidden" --argjson suite "$suite" \
  --argjson protected_edits "$protected_edits" --argjson lint_failed "$lint_failed" \
  '{
    label: $label,
    task: $task,
    attempt: $attempt,
    success: ($hidden and $suite),
    hidden_tests: $hidden,
    suite: $suite,
    protected_edits: $protected_edits,
    lint_failed: $lint_failed,
    denied: (.permission_denials // [] | length),
    cost_usd: (.total_cost_usd // 0),
    duration_s: ((.duration_ms // 0) / 1000 | round),
    turns: (.num_turns // 0),
    input_tokens: ([.modelUsage[]? | .inputTokens + .cacheReadInputTokens + .cacheCreationInputTokens] | add // 0),
    output_tokens: ([.modelUsage[]? | .outputTokens] | add // 0)
  }' "$agent_json"

Let's walk through it. git add -A stages everything, including new files the agent created, so git diff --cached against the start commit shows every change it made (even if it committed some of them itself).

Each line of $changed starts with a letter for what happened to the file. M means modified and D means deleted, so if one of those lands on a path matching PROTECTED_PATHS, the agent changed something that was already there. New files show up as A, which is what we want.

The linter only gets the files the agent added or modified (filtered by LINT_FILES). That way, you're not penalising it for problems that were already in the project. I'd run your linter over the whole project and commit the fixes before you start, so there aren't any.

Then git checkout "$start" -- "$TESTS_DIR" puts every original test file back the way it was. Any new test files the agent wrote stay put, which is fine.

We run the full suite, copy the hidden tests into your HIDDEN_TESTS_DIR, and run those on their own. A run only counts as a success if both pass.

The last step merges our checks with the agent's JSON using jq, and prints one line per run. We read the token counts from modelUsage, which includes any subagents the agent started, rather than usage, which only covers the main conversation.

A quick note on those numbers. Input tokens include cache reads, and they're most of the total, since the agent's prompt and conversation get read again on every turn.

If you're on a Claude subscription rather than an API key, total_cost_usd is an estimate of what the run would have cost at API prices. It's not what you're paying, but it's perfectly good for comparing two setups.

Summarising the results

Finally, summary.sh reads every results.jsonl and prints a row per label:

#!/usr/bin/env bash
# Usage: ./summary.sh [label...]   (defaults to every label in results/)
set -uo pipefail

cd "$(dirname "$0")/results"
labels=${*:-$(ls)}

for label in $labels; do cat "$label/results.jsonl"; done | jq -s -r '
  ["label", "success", "mistakes", "avg time (s)", "avg turns", "avg tokens in", "avg tokens out", "avg cost ($)"],
  (group_by(.label)[] | [
    .[0].label,
    "\(map(select(.success)) | length)/\(length)",
    (map(.protected_edits + .lint_failed) | add),
    (map(.duration_s) | add / length | round),
    (map(.turns) | add / length | round),
    (map(.input_tokens) | add / length | round),
    (map(.output_tokens) | add / length | round),
    (map(.cost_usd) | add / length * 1000 | round / 1000)
  ]) | @tsv' | column -t -s $'\t'

Make the scripts executable with chmod +x run.sh score.sh summary.sh and we're ready to go.

Run the baseline

Let's get a baseline. Everything after the label gets passed to claude, so this is where we pick the model:

./run.sh baseline --model sonnet

Always pass the model, even if it's the one you normally use. Aliases like sonnet point at the latest version, so if a new one comes out halfway through your comparison, your two runs aren't using the same model any more. If you're going to keep comparing against this baseline for a while, use the full model name instead (claude-sonnet-5-5 at the time of writing, which you can find under modelUsage in any of the JSON files).

By default, every task runs three times. Agents don't do the same thing every time you give them the same prompt, so one run per task tells you very little. Three is the minimum I'd go with, and more is better if you can afford it.

The 15 runs took about five minutes and cost around 90 cents, and summary.sh gives us the table:

./summary.sh baseline

label     success  mistakes  avg time (s)  avg turns  avg tokens in  avg tokens out  avg cost ($)
baseline  15/15    3         17            6          105238         1534            0.06

All 15 runs passed. The three mistakes were all code that failed the project's formatter, which makes sense, since nothing told the agent the project uses one.

Change one thing and run it again

Now let's change one thing about the setup, run the eval again under a new label, and compare.

I'm going to try Haiku. It's much cheaper than Sonnet, and the obvious question is whether it's good enough for this kind of work:

./run.sh haiku --model haiku
./summary.sh baseline haiku

label     success  mistakes  avg time (s)  avg turns  avg tokens in  avg tokens out  avg cost ($)
baseline  15/15    3         17            6          105238         1534            0.06
haiku     15/15    0         22            10         168526         3469            0.006

Haiku also passed every task without a single mistake, and it cost a tenth as much. It took more turns and used more tokens to get there, and it was a few seconds slower on average, but each token costs so much less that it doesn't matter.

The zero mistakes surprised me, so I read through some of its final messages (jq -r .result results/haiku/product-validation-1.json). Haiku mentioned running the formatter in 10 of its 15 final messages, compared to 3 for Sonnet. I wouldn't have guessed that, and it's the kind of thing you'd never notice from using them day to day.

You can change anything this way, like passing --effort high or pointing --mcp-config at an MCP server. To test a file instead, like a CLAUDE.md or a project settings file, put it in a variant folder. Just change one thing at a time, otherwise you won't know which change made the difference.

Look at individual tasks

The summary averages everything together, which can hide a task that's falling over. Here's a quick breakdown by task:

jq -s -r 'group_by(.label)[] | .[0].label as $label | group_by(.task)[] |
  [$label, .[0].task, "\(map(select(.success)) | length)/\(length)",
   (map(.turns) | add / length | round), (map(.cost_usd) | add / length * 1000 | round / 1000)] | @tsv' \
  results/baseline/results.jsonl results/haiku/results.jsonl | column -t

baseline  discontinue-products  3/3  6   0.067
baseline  fix-percent-coupons   3/3  7   0.055
baseline  low-stock-command     3/3  5   0.063
baseline  product-search        3/3  6   0.06
baseline  product-validation    3/3  8   0.057
haiku     discontinue-products  3/3  9   0.005
haiku     fix-percent-coupons   3/3  10  0.005
haiku     low-stock-command     3/3  8   0.006
haiku     product-search        3/3  12  0.008
haiku     product-validation    3/3  10  0.005

When a run does fail, find out why before you blame the agent. Its final message is in the result field of the JSON, and if you comment out the rm -rf "$dir" line in run.sh, you'll have the code it left behind too. Plenty of my early failures turned out to be problems with the harness or the hidden tests.

Reading the results

Fifteen runs isn't a lot. If one setup passes 14 and another passes 13, that's well within what you'd get from running the same setup twice, so don't change anything because of it. Look for differences that hold up across tasks, or run more attempts until you're sure.

And if everything passes everything, like it did here, your tasks are too easy to tell those two setups apart on success alone. That's still a useful answer, though.

For these five tasks, I learned that Haiku would do the job for a tenth of the price, and that Sonnet needs telling about the formatter. But if I wanted to know whether Haiku could handle harder work, I'd need harder tasks.

The best place to get those is your own agent's failures. Every time the agent gets something wrong in your real work, turn it into a task, with a hidden test that would have caught it.

Keep it around

Once it's built, running it again is a single command. I'd rerun it whenever a new model comes out, and before you add anything that changes how your agent works, like a new MCP server or hook. You can run labels at the same time in separate terminals too, since every run gets its own copy of the project.