Model evaluation

Model evaluation is the practice of measuring a workflow's output against a defined standard using a fixed eval set. It produces two things: the baseline that makes a pilot provable, and the ceiling that makes a self-improving workflow (L3) auditable.

Reviewed by Transformation.ai editorial · Markdown version · Stage: Autonomate

What model evaluation is

Evaluation is how you answer the question "is this workflow working?" without asking the team that built it. That sounds adversarial. It isn't — it's the only way to build an eval regime that survives the pressure to ship. The eval set is fixed. The standard is agreed. The number is checked. If a change makes the workflow better by the eval's measure, it ships. If it doesn't, it goes back.

Evaluation sits between two things teams routinely conflate: observability (what happened in production — logs, traces, error rates) and testing (whether a specific change broke something). Evaluation is a fixed set of cases, run periodically, that measures whether the workflow is meeting its quality standard — regardless of what happened recently, and regardless of whether a specific change shipped.

Why it matters

Two failure modes are impossible to see without evaluation. First, silent drift — the workflow was accurate in February, is accurate enough in March, and quietly produces 20% more errors by October because the input distribution shifted, the model quietly changed, or the workflow accumulated small tune-ups nobody reviewed. Second, self-improvement without a ceiling — the L3 failure mode where the agent tunes itself against a metric that has drifted, and the metric moves but the quality doesn't.

Evaluation is also the mechanism that lets an organization move a workflow up the autonomy tiers. Without a stable eval set, there's no evidence the workflow is ready for L2, L3, or L4 — only opinion, and opinion doesn't hold in an audit.

The four eval types

All four are needed. Unit and regression evals catch the routine "did we break it." Held-out evals catch the systemic "did we overfit it." Adversarial evals catch the exotic "did it handle the case we didn't expect?" Skipping any of the four leaves a class of failure untested.

Building an eval set that survives

The eval set is the artifact everything else depends on. Four rules:

1. Edge cases outweigh volume. Fifty to two hundred cases is enough for most workflows — if the set is built well. Twenty percent of the cases should be edge cases, ten percent adversarial probes, the rest representative routine cases.
2. Answers are written by humans, not derived from the model. An eval set where the "right answer" came from the model is not an eval set — it's a confirmation bias machine. Every expected answer is agreed by a named human who defends it.
3. Held-out means held out. Cases used for prompt tuning cannot be in the held-out set. Ever. If they are, overfitting tests won't be meaningful.
4. Reviewed quarterly, refreshed on drift. Eval sets go stale. Track the real-world distribution against the eval set's distribution and refresh the eval when the distribution shifts. Also refresh after any material change to the workflow itself.

The evaluation ceiling (L3 requirement)

An evaluation ceiling is a named metric, an acceptable band, and an owner — the standard a self-improving workflow (L3) is not allowed to breach. The agent can tune itself within the ceiling. It cannot change the ceiling.

The three parts, each of which is commonly missing:

The ceiling is what makes L3 self-improvement safe. Without it, L3 is not self-improvement — it's drift with better marketing.

Three failure modes

Overfitting to the eval set

The workflow passes the eval but fails in production. Cause: the eval set — or the held-out set — shares the same distribution as the training/tuning data. Held-out cases must be genuinely distinct. The fix is adversarial evaluation and periodic refresh of held-out cases.

Stale eval set

The eval set tests February's distribution. Production runs October's. The eval passes; the workflow fails. The fix is quarterly review of the eval set against the actual distribution, not the assumed one.

Narrow coverage

The eval set tests easy cases and misses the edge cases that produce most failures in production. Cause: the eval set was built by sampling production logs, which over-represent routine cases. The fix is deliberate edge-case authorship — a named person writes the ten hardest cases they can think of, once a quarter.

What model evaluation is not

• Not observability. Logs, traces, error rates, and dashboards measure what happened. Evaluation measures whether what happened was good. You need both.
• Not A/B testing. A/B testing compares two variants in production. Evaluation measures against a fixed standard. Complementary, not the same.
• Not a substitute for HITL. HITL catches failures the eval set didn't anticipate. Evaluation catches failures the reviewer missed. Neither replaces the other.
• Not a one-time setup. An eval regime that doesn't refresh will drift as surely as a workflow. The evals are as much a living artifact as the workflow map.