Evals in CI: a starter kit
We have written before about why the evaluation harness is the real product of any AI build. This is the practical companion: how to get one running in continuous integration with what you already have, so that "does the AI still work" is answered on every pull request rather than in the occasional panic.
No platform is required. A folder, a script, and a CI job will do.
The folder
Start with a directory in the repository, next to the tests, named for what it evaluates. Inside, one file per case. Each case is a small structured record: the input exactly as the system would receive it, the expected outcome, and a tag or two for grouping.
The expected outcome takes one of three forms. An exact value, for structured outputs like a category or an extracted field. A set of assertions, for outputs that must contain or avoid certain things. Or a rubric, a short list of criteria in plain language, for prose outputs that a grader will score.
Twenty cases is enough to start. Real inputs, not invented ones. Include the easy cases, the hard ones, and at least two where the correct behaviour is to refuse or escalate.
The runner
A script that loads every case, runs the system against the input, compares the result to the expectation, and produces a score. For exact values and assertions, comparison is code. For rubrics, comparison is a second model call with the rubric and the output, asked to score each criterion and explain. Keep the grader prompt in the repository too; it is part of the harness.
The runner prints a summary and writes a report: overall score, score per tag, and the full list of failures with input, expected, and actual side by side. The report is the thing an engineer reads when the job goes red.
The CI job
Run the harness on every pull request that touches the AI code, the prompts, or the evaluation cases themselves. Fail the job if the overall score drops below a threshold you set, or if any case tagged as critical fails.
Two practical notes. Cache aggressively, so unchanged cases do not cost money on every run. And run the harness on a schedule as well as on pull requests, weekly is fine, because the model vendor can change behaviour with no commit on your side. A scheduled run going red is how you find out before a customer does.
The threshold
Set the threshold from the first honest run, not from ambition. If the system scores eighty on day one, the threshold is seventy-eight, and the goal is to raise both. A threshold set at ninety-five for a system that scores eighty is a job that is always red, which is a job everyone learns to ignore.
Review the threshold monthly. As the harness grows and the system improves, ratchet it up.
Growing the harness
Every production incident adds a case. Every "what about when" from a stakeholder adds a case. Every override captured at a human gate is a candidate case, with the person's correction as the expectation. Within a few months the harness reflects the real failure modes of your system rather than anyone's guess at them.
Prune too. A case that has passed for six months and covers the same ground as ten others can go. The harness should be as large as it needs to be to catch regressions and no larger, because slow harnesses get skipped.
What this gets you
A pull request that changes a prompt and drops the score is caught before merge. A model update that changes behaviour is caught on the weekly run. A stakeholder asking whether the system is good enough for the next team gets a number and a list of the cases it still gets wrong. And the engineer who inherits the system next year has a way to change it without fear.
None of this needs a vendor. It needs a folder, a runner, a job, and the discipline to add a case every time something surprises you.