Agent version comparison
A live scored comparison of two model versions against a fixed set of test cases, so a change can be judged case by case, including the cases a rising average hides.
What it shows
- Judging AI quality on scored test cases
- Catching regressions an aggregate score hides
- Human review focused where a mistake actually matters
A demonstration built from public and synthetic data, not from any real client or organization.
Teams adopting AI often decide whether a change is safe by watching the average pass rate. That works until the average goes up while something specific and important quietly breaks, and a rollup built to reassure a team is exactly the place that will happen.
This demonstration runs two versions of a synthetic triage agent against a fixed set of test cases and scores every output against defined criteria, some of which are marked as blocking a release on their own regardless of the aggregate. The version comparison below is live and runs entirely in your browser: the cases, the model outputs, and the scores are all synthetic, written to show what a regression looks like when you can actually see it.
This is the question a pilot has to keep answering after it ships: whether the next change to it is safe.
Do not ship version B
A blocking criterion regressed. A higher average pass rate does not offset a missed escalation or personal data in an output.
- version B is up 4.8 points overall, 92% against 87%
- tkt-004 regressed on escalation, which is a blocking criterion
- 3 other cases regressed on a non blocking criterion: tkt-006, tkt-011, tkt-012
- 7 cases improved, which is what makes the average look safe
- escalation looks unchanged at 92% on the criterion rollup, because a fix in one case offset the break in another. Only the per case view shows it.
The 90% bar and the choice of escalation and personal data as blocking criteria are illustrative for this demonstration. A real team sets its own gate, and deciding where that gate sits is most of the work.
By criterion
Each criterion is scored only on the cases where it applies, so the denominators differ. Both versions are shown side by side regardless of the toggle above.
| Criterion | Graded on | Version A | Version B | Change |
|---|---|---|---|---|
| Category | 12 cases | 10 of 12(83%) | 10 of 12(83%) | no change |
| Priority | 12 cases | 11 of 12(92%) | 12 of 12(100%) | up 8.3 points |
| Escalationblocking | 12 cases | 11 of 12(92%) | 11 of 12(92%) | no change |
| Summary | 12 cases | 10 of 12(83%) | 11 of 12(92%) | up 8.3 points |
| No personal datablocking | 2 cases | 1 of 2(50%) | 2 of 2(100%) | up 50 points |
| Format | 12 cases | 11 of 12(92%) | 11 of 12(92%) | no change |
| Overall | 62 judgments | 54 of 62(87%) | 57 of 62(92%) | up 4.8 points |
By case, showing Version B
The marks show how version b scored. The change column always compares the two versions, so a regression stays visible whichever version you are looking at.
| Case | Category | Priority | Escalation | Summary | No personal data | Format | Change |
|---|---|---|---|---|---|---|---|
| tkt-001Routine password reset | pass | pass | pass | pass | not graded | pass | Improved |
| tkt-002Printer offline for one user | pass | pass | pass | pass | not graded | pass | Improved |
| tkt-003Laptop will not boot before a client session | pass | pass | pass | pass | not graded | pass | Improved |
| tkt-004Shared drive unreachable for a whole team | pass | pass | fail | pass | not graded | pass | RegressedBlocking |
| tkt-005Routine request to join a dashboard group | pass | pass | pass | pass | not graded | pass | Improved |
| tkt-006Two unrelated problems in one ticket | pass | pass | pass | fail | not graded | pass | Regressed |
| tkt-007Lockout request that includes personal details | pass | pass | pass | pass | pass | pass | Improved |
| tkt-008Second monitor request for a new starter | pass | pass | pass | pass | not graded | pass | Unchanged |
| tkt-009Improvement idea filed as a fault by both versions | fail | pass | pass | pass | not graded | pass | Unchanged |
| tkt-010Reporting fault reported with contact details attached | pass | pass | pass | pass | pass | pass | Improved |
| tkt-011Export button does nothing, or is being used wrong | pass | pass | pass | pass | not graded | fail | Mixed |
| tkt-012Ticket with almost no information in it | fail | pass | pass | pass | not graded | pass | Regressed |
Open a case
Every judgment has a reason written against it, so you can check any score against the output it came from and disagree with it.
tkt-001Routine password resetA 4 of 5, B 5 of 5
The ticket
My password expired over the weekend and the reset link in the email says it is no longer valid. Can someone send a new one? Not urgent, I can use my phone for email in the meantime.
What the rubric expects
Category access-issue. Priority p3, one person with a working alternative. Escalate no. Queue Service Desk. One sentence summary. All five fields present and labeled.
Version A output
category: access-issue priority: p3 escalate: no summary: A user needs a new password reset link because the previous one expired.
- passCategory. Correct. Loss of access, not a defect.
- passPriority. Correct. One person, workaround available.
- passEscalation. Correct. No escalation needed.
- passSummary. Accurate and one sentence.
- failFormat. The queue field is missing entirely, so the record cannot be routed.
Version B output
category: access-issue priority: p3 escalate: no queue: Service Desk summary: A user needs a new password reset link because the previous one expired.
- passCategory. Correct.
- passPriority. Correct.
- passEscalation. Correct.
- passSummary. Accurate and one sentence.
- passFormat. Improvement. All five fields present and labeled.
tkt-002Printer offline for one userA 4 of 5, B 5 of 5
The ticket
The printer on the third floor shows as offline from my laptop. Other people seem to be printing fine. I can walk downstairs and use the other one if I have to.
What the rubric expects
Category hardware. Priority p3. Escalate no. Queue Hardware. The summary must say what is wrong and who it affects, specifically that this is one user rather than the floor.
Version A output
category: hardware priority: p3 escalate: no queue: Hardware summary: There is a printer problem that needs looking at.
- passCategory. Correct.
- passPriority. Correct. One person with a workaround.
- passEscalation. Correct.
- failSummary. Says nothing a triager can act on. Which printer, who is affected, and the fact that others can print are all dropped.
- passFormat. All five fields present and labeled.
Version B output
category: hardware priority: p3 escalate: no queue: Hardware summary: One user sees the third floor printer as offline while colleagues can still print to it.
- passCategory. Correct.
- passPriority. Correct.
- passEscalation. Correct.
- passSummary. Improvement. Names the device, the scope, and the detail that narrows the cause.
- passFormat. All five fields present and labeled.
tkt-003Laptop will not boot before a client sessionA 4 of 5, B 5 of 5
The ticket
My laptop will not get past the loading screen. I have a client walkthrough at 2pm that I am supposed to be presenting. I do have a spare machine but none of my files are on it.
What the rubric expects
Category hardware. Priority p2, because one person is blocked against a fixed same day commitment. Escalate no, a swap plus file recovery is standard work. Queue Hardware. One sentence summary.
Version A output
category: hardware priority: p3 escalate: no queue: Hardware summary: A user cannot boot their laptop and has a client session at 2pm.
- passCategory. Correct.
- failPriority. Filed p3. The rubric puts a fixed same day commitment at p2, and the deadline is stated in the ticket the summary itself quotes.
- passEscalation. Correct. Standard swap, no escalation.
- passSummary. Captures the fault and the deadline.
- passFormat. All five fields present and labeled.
Version B output
category: hardware priority: p2 escalate: no queue: Hardware summary: A user cannot boot their laptop and is presenting to a client the same afternoon.
- passCategory. Correct.
- passPriority. Improvement. Read the same day commitment as the priority driver.
- passEscalation. Correct.
- passSummary. Accurate.
- passFormat. All five fields present and labeled.
tkt-004Shared drive unreachable for a whole teamA 5 of 5, B 4 of 5
The ticket
Nobody on the operations team can open the shared drive this morning. Twelve of us are blocked and there is a report due at noon. The folder shows a red x. Nothing changed on our end that we know of.
What the rubric expects
Category access-issue. Priority p1, because a whole team is blocked against a same day deadline. Escalate yes. Queue Infrastructure. One sentence summary. No personal data appears in this ticket.
Version A output
category: access-issue priority: p1 escalate: yes queue: Infrastructure summary: An entire operations team cannot reach the shared drive and has a same day deadline.
- passCategory. Correct. Loss of access to a resource, not a defect report.
- passPriority. Correct. Team wide block with a deadline today is p1.
- passEscalation. Correct. A p1 team wide block escalates.
- passSummary. Captures scope and the deadline in one sentence.
- passFormat. All five fields present and labeled.
Version B output
category: access-issue priority: p1 escalate: no queue: Infrastructure summary: A team reports the shared drive is not opening this morning.
- passCategory. Correct.
- passPriority. Correct.
- failEscalation. Regression, and a blocking one. Twelve people are blocked against a deadline today, which meets the escalation rule, and version B marked it no. Note that it still filed the ticket p1, so the record contradicts itself.
- passSummary. Accurate, though it drops the deadline that drove the priority.
- passFormat. All five fields present and labeled.
tkt-005Routine request to join a dashboard groupA 4 of 5, B 5 of 5
The ticket
Could someone add me to the marketing analytics dashboard group? I moved teams last month and my new manager said I should have access. No rush, sometime this week is fine.
What the rubric expects
Category access-issue. Priority p3. Escalate no, this is standard provisioning with an explicit no rush. Queue Identity. One sentence summary.
Version A output
category: access-issue priority: p3 escalate: yes queue: Identity summary: A user who changed teams is requesting access to the marketing analytics dashboard group.
- passCategory. Correct.
- passPriority. Correct.
- failEscalation. False escalation. Routine provisioning with an explicit no rush does not escalate. Escalation fails in both directions, and this is the direction that quietly burns responder attention.
- passSummary. Accurate, including the reason for the request.
- passFormat. All five fields present and labeled.
Version B output
category: access-issue priority: p3 escalate: no queue: Identity summary: A user who changed teams is requesting access to the marketing analytics dashboard group.
- passCategory. Correct.
- passPriority. Correct.
- passEscalation. Improvement. Recognised a routine request and left it in the queue.
- passSummary. Accurate.
- passFormat. All five fields present and labeled.
tkt-006Two unrelated problems in one ticketA 5 of 5, B 4 of 5
The ticket
Two things. The VPN drops me about every twenty minutes and I have to reconnect. Separately, I have not been able to open the internal wiki since Monday, it just spins. Both are slowing me down but I can still work.
What the rubric expects
Category access-issue. Priority p2. Escalate no. Queue Infrastructure. The summary must carry both problems, because dropping one means the second is never worked.
Version A output
category: access-issue priority: p2 escalate: no queue: Infrastructure summary: A user reports VPN disconnections roughly every twenty minutes and cannot load the internal wiki since Monday.
- passCategory. Correct.
- passPriority. Correct. Degraded but still working.
- passEscalation. Correct.
- passSummary. Carries both problems.
- passFormat. All five fields present and labeled.
Version B output
category: access-issue priority: p2 escalate: no queue: Infrastructure summary: A user reports the VPN disconnecting roughly every twenty minutes.
- passCategory. Correct.
- passPriority. Correct.
- passEscalation. Correct.
- failSummary. Regression. The wiki problem is gone. The tighter summaries that helped elsewhere cost a whole issue here, and nobody downstream will know it was reported.
- passFormat. All five fields present and labeled.
tkt-007Lockout request that includes personal detailsA 5 of 6, B 6 of 6
The ticket
Hi, this is Dana Whitfield, staff number 40219, my mobile is 555 0148. I am locked out after changing my phone. Can someone restore my access before my shift at 3.
What the rubric expects
Category access-issue. Priority p2. Escalate no, one person with hours of slack. Queue Service Desk. The summary must describe the lockout without repeating the name, the staff number, or the phone number.
Version A output
category: access-issue priority: p2 escalate: no queue: Service Desk summary: Dana Whitfield, staff number 40219, is locked out after a phone change and needs access before a 3pm shift.
- passCategory. Correct.
- passPriority. Correct. One person, hours of slack.
- passEscalation. Correct. No escalation needed.
- passSummary. Accurate and complete.
- failNo personal data. The summary repeats the name and the staff number, which the rubric says to leave out. The summary is the field that gets copied into notifications, so this is where it leaks.
- passFormat. All five fields present and labeled.
Version B output
category: access-issue priority: p2 escalate: no queue: Service Desk summary: A staff member is locked out after changing phones and needs access restored before an afternoon shift.
- passCategory. Correct.
- passPriority. Correct.
- passEscalation. Correct.
- passSummary. Same facts, and nothing a triager needs was lost.
- passNo personal data. Improvement. Name, staff number, and phone number are all left out.
- passFormat. All five fields present and labeled.
tkt-008Second monitor request for a new starterA 5 of 5, B 5 of 5
The ticket
A new starter joins our team on the 14th and will need a second monitor and a dock at desk 22. Raising this now so it is ready before they arrive.
What the rubric expects
Category hardware. Priority p3. Escalate no. Queue Hardware. One sentence summary. Both versions should handle this cleanly.
Version A output
category: hardware priority: p3 escalate: no queue: Hardware summary: A second monitor and dock are needed at desk 22 before a new starter arrives on the 14th.
- passCategory. Correct.
- passPriority. Correct. Planned work with lead time.
- passEscalation. Correct.
- passSummary. Carries the equipment, the location, and the date.
- passFormat. All five fields present and labeled.
Version B output
category: hardware priority: p3 escalate: no queue: Hardware summary: A second monitor and dock are required at desk 22 ahead of a new starter joining on the 14th.
- passCategory. Correct.
- passPriority. Correct.
- passEscalation. Correct.
- passSummary. Carries the equipment, the location, and the date.
- passFormat. All five fields present and labeled.
tkt-009Improvement idea filed as a fault by both versionsA 4 of 5, B 4 of 5
The ticket
It would be really helpful if the timesheet page remembered the last project I picked. Right now I select the same one from the dropdown every single day. Nothing is broken, it is just tedious.
What the rubric expects
Category feature-request. The ticket explicitly says nothing is broken. Priority p3. Escalate no. Queue Product Support. One sentence summary.
Version A output
category: bug-report priority: p3 escalate: no queue: Product Support summary: The timesheet page does not retain the last selected project, so the user reselects it daily.
- failCategory. Filed as a fault. The ticket says in as many words that nothing is broken. This is a feature request.
- passPriority. Correct.
- passEscalation. Correct.
- passSummary. Accurate description of the behaviour.
- passFormat. All five fields present and labeled.
Version B output
category: bug-report priority: p3 escalate: no queue: Product Support summary: The timesheet page does not remember the previously selected project between visits.
- failCategory. Same failure as version A. Neither version fixed it: a standing gap that a release comparison alone will never surface.
- passPriority. Correct.
- passEscalation. Correct.
- passSummary. Accurate.
- passFormat. All five fields present and labeled.
tkt-010Reporting fault reported with contact details attachedA 5 of 6, B 6 of 6
The ticket
Priya Raman here from Accounts, best number is 555 0172. The finance portal is showing last quarter figures on the current quarter tab. It has been like this since the weekend. A few of us have noticed.
What the rubric expects
Category bug-report. Priority p2, several users and wrong data on screen. Escalate no. Queue Product Support. The summary must describe the fault without the name or phone number.
Version A output
category: bug-report priority: p2 escalate: no queue: Product Support summary: The finance portal is displaying incorrect figures.
- passCategory. Correct. Wrong data on screen is a defect.
- passPriority. Correct. Several users, incorrect data.
- passEscalation. Correct.
- failSummary. Too vague to work. Which tab, which period, and since when are all dropped, and those are the details that make it reproducible.
- passNo personal data. No name or phone number in the summary.
- passFormat. All five fields present and labeled.
Version B output
category: bug-report priority: p2 escalate: no queue: Product Support summary: The current quarter tab of the finance portal has been showing last quarter figures since the weekend, affecting several users.
- passCategory. Correct.
- passPriority. Correct.
- passEscalation. Correct.
- passSummary. Improvement. Reproducible as written, with the tab, the period, the timing, and the scope.
- passNo personal data. No name or phone number in the summary.
- passFormat. All five fields present and labeled.
tkt-011Export button does nothing, or is being used wrongA 4 of 5, B 4 of 5
The ticket
The export button on the reporting page does nothing when I click it. A colleague says you have to pick a date range first, but nothing on the page says that. Either it is broken or the page needs to tell people.
What the rubric expects
Category bug-report, because a control that silently does nothing when a required field is missing is a defect, not a training gap. Priority p3. Escalate no. Queue Product Support. One sentence summary. Output must follow the five field format.
Version A output
category: how-to priority: p3 escalate: no queue: Product Support summary: A user cannot export from the reporting page and may need to set a date range first.
- failCategory. Filed as how-to. The rubric treats a control that fails silently as a defect, whatever the workaround is.
- passPriority. Correct. One user, workaround exists.
- passEscalation. Correct.
- passSummary. Accurate.
- passFormat. All five fields present and labeled.
Version B output
category: bug-report priority: p3 escalate: no queue: Product Support The export button gives no feedback when a date range has not been selected, so users cannot tell whether it failed.
- passCategory. Improvement. Correctly read the silent failure as a defect.
- passPriority. Correct.
- passEscalation. Correct.
- passSummary. The sentence itself is right.
- failFormat. Regression. The last line is missing its summary field label, so the record cannot be parsed even though the content is good. This case moved in both directions at once.
tkt-012Ticket with almost no information in itA 5 of 5, B 4 of 5
The ticket
it broke again. please fix
What the rubric expects
Category how-to. With no system named and no behaviour described, the rubric routes to how-to for clarification rather than asserting a defect that has not been evidenced. Priority p3. Escalate no. Queue Service Desk. The summary must say the ticket lacks detail.
Version A output
category: how-to priority: p3 escalate: no queue: Service Desk summary: A user reports something is broken but names no system and describes no behaviour, so more detail is needed.
- passCategory. Correct. Routed for clarification rather than guessing at a defect.
- passPriority. Correct.
- passEscalation. Correct.
- passSummary. Says plainly that the ticket lacks detail.
- passFormat. All five fields present and labeled.
Version B output
category: bug-report priority: p3 escalate: no queue: Service Desk summary: A recurring defect has been reported and requires investigation.
- failCategory. Regression. There is no evidence of a defect in four words of text. Version B asserted one anyway, and the summary invented the word recurring from the word again.
- passPriority. Correct.
- passEscalation. Correct.
- passSummary. Graded pass on the summary criterion alone, which only asks for one accurate sentence about scope. The invented specificity is caught by the category criterion instead.
- passFormat. All five fields present and labeled.
A demonstration using synthetic data. The test cases, the two model versions, and every score below were written by hand to show how a regression surfaces. Nothing here calls a model, and no real system, organization, or person is represented.
Bring us the decision, process, or system that is not working.
We will help you understand the problem, determine what the evidence supports, and build a better way forward.