There’s a category of work in AI products that nobody owns by default. It isn’t quite engineering, because the hard part isn’t making the system run. It isn’t quite product management, because the artifact is a test suite, not a spec. It’s evals, and after building an eval framework for the production agent system at Mattoboard (an AI-powered interior design platform), I’ve come to think it’s the single best bridge between the two disciplines, and the best proof-of-work for anyone claiming to do both.
When you ship a deterministic feature, “does it work” is a question with an answer. When you ship an LLM-powered feature, “does it work” is a question with a distribution. The output on Tuesday is not the output on Wednesday. A model upgrade that improves your benchmark scores can quietly degrade the one behavior your users actually depend on.
Most teams respond to this with vibes. Someone notices generations “feel worse,” a Slack thread erupts, an engineer reruns a few prompts, everyone squints at the results, and the team either panics or shrugs. I’ve lived this exact sequence: a suspected model regression where the only tools available on day one were screenshots and adjectives.
The fix wasn’t a better model. It was a rubric (what does “good” mean, in writing, with examples) and a harness that could run that rubric repeatedly. That’s an eval. And notice what it took to build: someone had to make a product decision (what quality means, which failures are disqualifying, which are tolerable) and encode it as an engineering artifact (fixtures, assertions, CI). Neither discipline can do it alone.
The framework I built for Mattoboard’s LangGraph agents is deliberately boring. It runs on Vitest, the same runner as our unit tests, because an eval you have to remember to run is an eval that doesn’t run. The core pattern is three steps: construct an input state, execute a single graph node, assert on the output.
A few decisions that mattered more than I expected:
Test at the node level, not the pipeline level. End-to-end evals are seductive and mostly useless for debugging. When a five-node pipeline produces a bad output, an end-to-end failure tells you nothing about where. Node-level evals localize the failure the same way unit tests do. You still want a thin layer of end-to-end checks, but the bulk of your coverage should sit at the seams.
Tier your assertions. Not everything needs an LLM judge. Exact-match assertions for structured outputs, schema validation for shape, contains/excludes checks for known failure strings, and only then semantic judging for the genuinely fuzzy stuff. Each tier is cheaper and more deterministic than the one above it. Spend your judge budget where nothing else works.
Mock the LLM as an optimization, not a default. Mocked runs are fast and free, and they’re right for testing the deterministic scaffolding around your model calls. But the whole point of evals is catching behavior changes in the model itself, so live calls have to be in the loop somewhere. A nightly CI run works fine.
Shared fixtures are your regression corpus. Every real-world failure that reaches you is a gift. Snapshot the input state that produced it, add it to the fixture set, write the assertion that would have caught it. Your eval suite becomes an accumulating record of everything the product has ever gotten wrong, which is a more honest product document than most PRDs.
Here’s the part that took me longest to articulate. Writing the assertion is easy. Deciding what to assert is the entire job.
“The layout node should return a valid composition” is an engineering statement. “A valid composition never places the hero image below the fold, always leaves breathing room around materials, and prefers asymmetry over grid when the palette is warm” is a product statement, and it’s the one the assertion has to encode. To write it, you have to interview whoever holds the taste (a designer, a founder, your users via their complaints), extract the implicit rubric they’re carrying in their head, and make it explicit enough to fail a build.
That extraction process is product management. It surfaces disagreements that would otherwise live in vibes (“wait, you think that output is fine?”), it forces prioritization (which failures block release, which get logged), and it creates a shared, versioned definition of quality that outlives any individual’s judgment. The eval suite becomes the place where engineering and product stop talking past each other, because it’s the one artifact both sides can read, run, and amend.
If your team has no evals, don’t design a framework. Take your last three quality complaints, turn each into a fixture and one assertion, and wire them into whatever test runner you already have. You’ll learn more from arguing about those three assertions than from a month of architecture. The framework can grow from there; the judgment it encodes is the part that compounds.