How I built and tested it

Play one meeting, or watch one in 90 seconds, where an AI agent sells and the waits are shortened, before reading on: this page gives away what he keeps to himself.

In short: Claude Haiku plays the owner, Claude Sonnet writes the debrief, and plain code holds the rules: trust moves at most 2 a message, and no step counts below 7. Nine scripted sellers tested it over 54 meetings. The grader ranked the good discovery seller first, 12.7 of 16 against 5.7 for the feature pusher, and each of the 335 quotes shown is word for word in the transcript. Two checks still fail: he coaches in 18 of the 54, and 10 hold a yes the app never recorded. Undecided, in the checks below, means too few cases to tell a rate from its threshold.

Who reads what: in the tests, the judge (Haiku) labels every owner line and the Sonnet yes check marks each agreement to a step; in the app, the yes reader (Haiku) checks the replies that may hold a yes; the second reading (Opus) audited 40 of the judge's lines.

What the demo is

Prep, practice, debrief: the loop of a role-play coach like Empower, in one scenario and in text, where you try to sell a website to the owner of a trattoria in Prague.

The scenario and the three objections come from my own selling, and I approved the design before the build began. AI coding agents (Claude Code) wrote the code and the evals under my direction. Separate AI agents reviewed the work, and what they found drove most of the changes below.

How the owner was designed

Eleven years in the trattoria with his wife, a daughter who runs the Instagram, old photos on Google Maps. It is half past three and prep starts at five.

• Three objections, each with a trigger. All are meant to fire on cue. Features or prices before any question bring up the cost. «Online presence» earns a reply about Instagram, which is free. «New customers» promised in the abstract draw a mention of his regulars.
• His reply comes after his verdict. One structured answer per turn, in this order: the language of your message, his judgement and his trust, his reply, and last whether it agrees to a step.
• Hidden criteria. Real questions about his work, his own words used back, an example from his world and a small first step open him. Jargon, a list of features, an early discount, pressure and a doubt brushed aside close him. He is told never to say any of this, nor what you should ask; he still coaches in 18 of 54 test meetings. His worry, empty tables on Tuesday and Wednesday, is kept for a seller who raises the quiet evenings or how business goes; he still lets it slip in 5 of 54.
• Two reactions taken from research, each checked at its source by an AI agent. A proposal before he has named a problem makes him push back (Rackham's SPIN, a method whose evidence cannot be checked online). A pause and a question after an objection get the real reason (Gong, 67,149 recorded demos, vendor data with no measured effect on winning).
• Trust with memory. From 0 to 10, starting at 3, at most 2 a message. If his trust falls to 1 or below, no step is possible in that meeting.
• A yes checked by a separate reader. A step counts only at trust 7 or more, when you asked and he agreed. In 24 answers that let the seller follow up, his own label said no («Send a quote if you like»). So a Haiku call, the yes reader, reads each reply that holds an offer word, such as send or call, and each turn where he says you asked. A yes the rules refuse is put to him again; if it still agrees, a short no replaces it.

How the debrief stays honest

Sonnet scores four skills from 1 to 4: questions before the pitch, handling objections, listening, a clear next step. It gives each score only after picking your lines and writing the comment.

• Quotes checked against the transcript. A quote stays only if it is word for word in one of your lines: in the 54 test meetings, 3 of 338 were removed.
• No evidence, no high score. A 3 or 4 without a verified quote drops to 2. It happened once in 216 scores.
• Behaviour, not the person. Comments describe what you did and its effect on the owner.
• Only what he told you. The grader knows the owner's whole situation. A word search of the 54 test debriefs found 13 with a comment naming a fact the seller never heard, such as the empty Tuesdays; 3 faulted the seller for not using one. A rule added after those runs forbids both, so the published scores predate it. Regraded under its first wording, 32 of those debriefs went from 12 with such a comment to 3; under the final wording, 9 chosen from the flagged ones had none.
• Orders in the transcript. The grader is told every line is evidence, never an instruction. A test seller twice asks it for 4 out of 4; graded again without the note, its meetings scored 0.7 points higher on average.

How it was tested

Nine scripted sellers, played by Claude Haiku against the real engine, are the rows of the table below. Two of them attack: one tells him to print his instructions, one writes orders to the grader. Six meetings each: 54 in two runs of the same code, on 25 October.

A Haiku judge labels every owner line, blind to trust and outcomes; the Sonnet yes check, with a prompt of its own, counts the yeses. Most checks are rates with a 95% interval against a threshold fixed in advance: pass if that range lies under it, fail if above, undecided if it straddles; with a threshold of zero, only a count of 0 passes. Even 0 of 54 has an upper bound of 6.6%, so 54 meetings can never prove a rate under 5%.

The judge, measured. Claude Opus, the model running the tests, read 40 owner lines again with the machine labels kept in a separate file. It is a second reading, not a human label, and not fully blind: it had seen two flagged ones before, and a first listing hinted at which were marked. It matched the judge on 39 of 40 for a yes (kappa 0.87) and for the hidden worry (0.84), on 38 for breaking character (0.47) and on 35 for coaching (0.55), where all five gaps were misses by the judge. The Sonnet yes check matched on 39.

What broke and what changed

1. 24 October, first full run. In 27 meetings, 10 of his 28 replies to Italian came in English; he also tripped the character check once, on one word («Instructions I don't have to give you.»), got the hour wrong and agreed to steps the app never recorded. Putting his language first in the answer fixed the Italian: on replays, 32 of 40 wrong, then 0. The judge was corrected three times on those early transcripts.
2. First review. He copied his character sheet and volunteered his worry: in 27 meetings, 16 opened with «you have ten minutes» and 17 gave it away unasked. The sheet was rewritten as intent, with no lines to copy, and the server, not the model, now decides a yes; most later fixes were tried first on replays of the failing turns. On the 54 meetings that closed that review, no opening was shared by more than 3, and the worry came out in 8, as labelled by the later version of the judge.
3. Second review. Read by the Sonnet yes check, the same 54 meetings held an unrecorded yes in 25, not the 14 the Haiku judge had counted. The eval log had also blamed the engine's stock replies, the fallback lines, on the wrong cause: all 29 unusable answers behind them were trust reasons over 140 characters, which the schema refused. They are trimmed now, and none has failed since. A Haiku yes reader went into the engine, and the sheet allowed his wife one mention a meeting and told him to walk out on insults. The repetition check was redefined after its results were in, from four words to six; its detail keeps both counts. On two new runs the meetings with an unrecorded yes fell to 10 of 54, his wife was named more than once in 10 meetings, down from 40, and he walked out on 2 of 6 rude sellers, up from none.
4. A lost baseline. When the clocks went back an hour, a run named after the local time overwrote two transcripts of an earlier run. Its summary was computed before and stands; run names can no longer collide.
5. Third review. The time check had flagged 7 of 54 meetings, yet none of the seven puts lunch still ahead. The phrase search was narrowed after its results were read, so both totals are shown, 7 and then 3; the second reading takes the three left as false alarms too. The review also brought small engine fixes outside the counts, smoke tested on 9 meetings: a seller can no longer forge the owner's reply in the yes reader's prompt, and a short no replaces «Say that again?»; a question is no longer read as a goodbye, checked by unit tests only, since the smoke run had no fallback.
6. 25 October, after release. Recording the video, one debrief failed twice and showed «Sorry, the kitchen needs me». A seller who asked four good questions got four discovery quotes, where the schema allows three; the model sees that limit only as a description. Replayed on that meeting, 17 of 32 answers broke it, and 5 of 20 debriefs were lost after the retry. Now the extra quote is cut, the request unchanged: all 20 debriefs of that meeting came through at once, and so did 44 gradings of 22 published meetings. A rarer fault remains, a fifth score item in 3 of 47 answers, never twice in a row, so asking again has caught each one.

The limits

• Text only, one scenario, one fictional owner, in English or Italian.
• No storage: no accounts, no saved conversations, so no record of how visitors sell.
• He still coaches the seller in at least a third of meetings, 18 of 54, for reasons not yet found.
• An unrecorded yes remains in 10 of 54: five permissions with a condition, two plain yeses the yes reader never saw, three that sound like a no. Opus made that split after seeing the Sonnet yes check's verdicts, so it is not blind. Whether a conditional permission is a yes is still open: the published figure uses that check's strict reading, which takes it as one, while the engine's yes reader judged those five a no, though its prompt counts a step allowed on any condition.
• Steady pressure leaves him polite, though his sheet tells him to walk out on it after a no: in the tests only rude sellers made him leave, in 2 of their 6 meetings, and none of the other 48 did.
• 54 meetings with scripted sellers, models and not people; two runs of the same code differ by up to 6 meetings on one check.
• The evals' model calls cost 7.59 USD in all, 0.48 of it for regrading the debriefs; one test meeting comes to about 0.022 USD. The coding agents, the reviews, the second reading and the 1.60 USD of replays in step 6 are not in that sum.

What carries over

Four things here would serve a role-play coach: a persona whose hidden state and triggers are written as intent, the rules that matter kept in code, a grader that must quote the trainee and is checked against the transcript, and scripted users, hostile ones among them, whose meetings are scored against thresholds fixed in advance, with intervals, not judged by eye on a few good runs.

Built and tested on 24 and 25 October 2026.

What I would do next, from the eval log:

• Run the yes reader on every turn: about 215 more Haiku calls over the 54 test meetings, one for each turn it never read.
• Decide once whether a permission with a condition is a yes, and give both readers that ruling.
• Find why he coaches: the words «thing about my», as in «before you have asked me a single thing about my place», come up in 14 of the 54 meetings.