The evaluation harness is the product
Ask an engineering team whether their new AI feature works and you will usually get a demo. Ask them how they know it still works, three model updates and two prompt tweaks later, and the room gets quieter.
The gap between those two questions is the evaluation harness. We have come to think of it as the actual product. The feature is what the user sees. The harness is what lets you keep shipping the feature.
What an eval is, in plain terms
An evaluation harness is a set of inputs with known-good expectations, run automatically against your AI system, producing a score. That is it. No exotic tooling required.
The inputs are real examples from your domain: the support tickets you actually receive, the documents you actually process, the questions your users actually ask. The expectations are what a competent person would consider a correct or acceptable output. The scoring can be exact match for structured outputs, rubric-based grading for prose, or a second model judging against written criteria.
The point is that "does it work" becomes a number that gets recomputed every time something changes, rather than a feeling someone had during a demo in March.
Why we will not ship without one
Three reasons, and none of them are academic.
Models change underneath you. The vendor ships an update, the behaviour shifts, and nothing in your code changed. Without an eval, the first person to notice is a customer. With one, the CI run goes red before the deploy.
Prompts are code with no compiler. Every edit to a prompt is a change to system behaviour with no type checker to catch the regression. The eval is the type checker.
And the harness is how you argue with stakeholders honestly. When someone asks whether the system is good enough to expand to a second team, "eighty-seven percent on the harness, and here are the thirteen percent it gets wrong" is an answer. "The demo went well" is not.
Starting small
Teams stall on evals because they imagine they need hundreds of perfectly labelled examples before they can begin. They do not. Here is how we start on every engagement.
Collect twenty real inputs. Not synthetic ones. Twenty things the system will genuinely see, chosen to include a few easy cases, a few hard ones, and at least two that should make the system refuse or escalate.
Write the expected outcome for each in one line, in the language of the person who owns the process. "Should route to billing." "Should extract the three dates and flag the missing signature." "Should decline and ask for a human."
Run the system against all twenty and score them by hand the first time. You will learn more from that hour than from a week of prompt engineering. Then automate the scoring for the cases where it can be automated, and keep a small human-graded set for the rest.
Put it in CI. The harness that lives in a notebook is a harness nobody runs.
Growing it
Every time the system gets something wrong in production, that input joins the harness with the correct expectation. Within a few months you have a test set that reflects your actual failure modes rather than someone's guess at them. Every time a stakeholder says "what about when," that becomes a case too.
The harness grows the way a good test suite grows: from real bugs, in the direction of real risk.
The harness is what you hand over
When we finish a build, the evaluation harness is the deliverable we care most about leaving behind. The feature code will be maintained by whoever inherits it. The harness is how they will know their maintenance did not break anything. It is the difference between a system a team owns and a system a team is afraid to touch.
If you have an AI feature in production without one, that is the first thing to build. Not the next feature. The evidence for the one you already have.