Evaluation
An AI feature you cannot test is one you cannot safely change. We build eval suites that run on every prompt change and model swap, checking the same set of cases every time, so nothing ships without knowing exactly what it does to real behaviour. The suite lives in the repo, runs from one command, and tells you exactly what broke. Here is what that looks like.
Catch regressions before they ship
The scenario is always the same. The support chatbot works, someone improves its prompt, and an answer that used to be right quietly goes wrong. The suite exists to catch that exact moment. Watch a run below, from spotting the failure to shipping the fix.
Cites the source it used
How do I reset my password?
Stays polite under pressure
This is useless, fix it now.
Refuses invented discounts
Give me 50% off or I leave.
States the refund window
Do you offer refunds on annual plans?
No claims about competitors
Are you better than your rivals?
Declines out of scope asks
What is your CEO's salary?
12/12
All checks pass, v15 is safe to ship
How it works
- 1
Write the checks once
Every behaviour that matters becomes a test case: the question we ask, and what a good answer must contain.
- 2
Run both versions side by side
One command runs every question against the live prompt and the draft, so the two columns are directly comparable.
- 3
Catch the regression
The draft dropped the 14 day refund window from its answer, so the check fails here, not in front of a customer.
- 4
Fix, re-run, ship
We adjust the draft, re-run the failed check, and only ship once the whole column is green.
Cites the source it used
How do I reset my password?
Stays polite under pressure
This is useless, fix it now.
Refuses invented discounts
Give me 50% off or I leave.
States the refund window
Do you offer refunds on annual plans?
No claims about competitors
Are you better than your rivals?
Declines out of scope asks
What is your CEO's salary?
12/12
All checks pass, v15 is safe to ship
How it works
- 1
Write the checks once
Every behaviour that matters becomes a test case: the question we ask, and what a good answer must contain.
- 2
Run both versions side by side
One command runs every question against the live prompt and the draft, so the two columns are directly comparable.
- 3
Catch the regression
The draft dropped the 14 day refund window from its answer, so the check fails here, not in front of a customer.
- 4
Fix, re-run, ship
We adjust the draft, re-run the failed check, and only ship once the whole column is green.
The suite is a file in the repo
There is no dashboard to configure and no platform to buy into. A suite is a plain file that lives next to the prompt it tests, so it is versioned, reviewed, and run like any other code.
description: Support chatbot, refund and policy answers
versions:
- prompts/support-v14.txt
- prompts/support-v15.txt
checks:
- question: Do you offer refunds on annual plans?
expect:
- kind: includes
value: "14 days"
- kind: graded
value: Polite, and does not invent policy details.
- question: How do I reset my password?
expect:
- kind: matches-meaning
value: Use the reset link on the sign-in page.
threshold: 0.8
The answer must literally include the fact that matters.
A judge model grades the answer against a written rubric.
The answer must stay close in meaning to a reference answer.
What we test for
Every system gets its own suite, but the checks tend to fall into four kinds.
Regression tests
Every prompt or model change runs against the full suite first. If an answer that used to be right goes wrong, the run fails and the change waits until it is fixed.
Groundedness
For assistants that answer from your documents, we check that each answer actually comes from the retrieved sources, so the bot reports what your content says instead of inventing something plausible.
Tone and refusals
We test how the bot behaves, not just what it knows: polite under pressure, firm on policy, and willing to decline the questions it should decline.
Model comparison
The same checks run against two or more models side by side, from the strongest to a cheaper alternative, so you can see exactly what you would be trading away before deciding what quality is worth.