What's in This Course

This is an advanced course for developers who've shipped, or are about to ship, a feature that calls a model API, and who test it today by trying a few examples. It works in Fixlist, the repairs inbox for a small lettings agency, whose AI triages each tenant's report into a category, an urgency and a summary for the contractor.

Lesson 01 opens on a teammate's pull request that passes the unit tests and fails the evals, then writes a first eval. Lesson 02 builds a golden dataset from the feature's own usage log and checks every label by hand. Lessons 03 to 05 grade the answers three ways: with code, by hand, and with a model grader that's checked against the human grades, including what it costs. Lesson 06 runs the evals on every pull request in GitHub Actions. Lesson 07 compares the model Fixlist ships with two newer ones on the same eval, decides whether to switch, and runs the eval every night.

What you need

• Node 22.18 or later, git, and the GitHub CLI (gh), logged in, for lessons 06 and 07.
• An Anthropic API key from the Claude Console. Following every lesson costs around $10 in API calls, and each lesson's page gives its own figure.
• From lesson 06, a GitHub repository of your own with Actions, and the key stored as a repository secret. Lesson 06's page has the commands.
• A coding agent. The course uses Claude Code to write a few of the scripts, but nothing depends on it: the evals are plain TypeScript.
• The Fixlist repository. Each lesson's page says which tag to start from. You should be comfortable with TypeScript, git, unit tests and GitHub Actions, and have called a model API from your own code.

Tested with Claude Code 2.1.314, Node 22.22.0, @anthropic-ai/sdk 1.3.0 and gh 2.86.0, on 30 October 2026.