Research partner / Otago Polytechnic
Teaching judgment that takes a career to build, measured turn by turn
Research partnership + every practice message scored
1.7x
rise in questions that open the client up, from first sessions to final ones
The research objective
Can career-built judgment be learned by trial and error?
Otago Polytechnic is a Tacit research partner, and the partnership set out to improve how the program teaches its hardest skill. Tacit captures expert judgment into scenarios: simulated clients with histories, things they hold back, and rules for how they respond to a skilled move versus a clumsy one. The study measured whether scenarios built that way can teach students, through trial and error, judgment that normally takes years of real clients to build.
The study set out to compress a career of practice reps into an afternoon and measure whether students actually improved. Every measured gain does double duty: it is a better-trained coach, and it is proof the scenarios carry the judgment itself.
The challenge
The core coaching skill had no safe rehearsal.
Otago Polytechnic, a Tacit research partner, teaches postgraduate sport and exercise programs through its Institute of Sport, Exercise and Health, where students specialize in physical conditioning and exercise prescription. Both specializations hinge on the hardest part of coaching to teach: asking a great question, then engaging with what comes back.
Judgment takes a career
The communication skills that separate great coaches from competent ones build over years of client conversations. There was no way to compress those reps, or to measure them.
Lectures cannot rehearse judgment
A student can recite questioning frameworks and still freeze when a client minimizes an injury or pushes back on an exercise choice.
Real clients are a risky practice surface
Practicing difficult conversations on real athletes carries real consequences, and role-playing classmates do not resist the way clients do.
Every client is a different exam
On a real placement, each student faces a different person, so no two performances can be compared and improvement cannot be measured.
Failure modes stay invisible
What a student does when a client challenges their expertise never surfaces in classroom observation or a written assessment.
The solution
Simulated clients, scored turn by turn.
Tacit built a library of AI clients for the program and ran two studies on it. The first taught students how to ask (February to March 2026, 27 students); the second taught them how to engage with the answer (May 2026, 19 students). Every student message was scored against an adapted version of motivational interviewing, the questioning-and-listening method health coaches train on.
Clients built with ex-students
Ex-students of the program built the clients in Tacit's scenario designer, drawing on New Zealand strength-and-conditioning prescription data. The result is a graded ladder of AI clients: from Tessa, a postpartum mum who volunteers everything, to Wiremu, a heart-attack survivor hiding grief behind five-to-ten-word answers.
Practice, review, retry
Students practiced before they were taught: a solo conversation with an AI client, a scored breakdown of what happened, a strategy huddle with peers, then a second attempt at the same kind of client.
Scored while it happened
In the first study, every conversation was scored message by message after the fact. By the second, six rubrics scored each message live and hints suggested the next move in real time. In both, mid-session prompts asked students what they were noticing.
Data wrote the next lesson
The first study taught questioning, and its data showed that most students never reflected a client's words back. So the second study taught exactly that: how to engage with the answer.
Results
Novices learned the judgment. Proof the scenarios carry it.
1.7x
The share of questions inviting exploration rather than a yes or no rose from 0.35 in first sessions to 0.61 in final ones, across 27 students and 858 scored messages.
82%
of students with two or more sessions used more of the framework's recommended behaviors by their last session, a median gain of +0.18 on a 0-to-1 scale.
13 of 17
students who completed both rounds of the second study scored higher after an 87-minute lecture on engaging with the answer: a median gain of 14.5 points per student on a 0-to-100 scale.
18 of 18
sessions run after that lecture improved while the conversation was still going, climbing a median 26.5 points from first score to final. The strongest number in the research, because each comparison stays inside one conversation.
~80%
Of the 18 real-time hints shown during the second study, roughly 8 in 10 were acted on in the student's very next message. One student followed all six of theirs and finished 39 points above their first session's final score.
Depth, reached
A student who opened their first session with a yes/no screening checklist closed their final one naming what drove a simulated athlete: 'Fear of failure. His fear of not making the team is driving everything.'
Next lesson, found
The first study's data showed 59% of students never once reflected a client's words back. The second study taught exactly that, beginning with reflective listening.
Caveat, declared
Students happened to draw easier simulated clients before the second lecture than after it. The gain above is reported with that caveat attached, and the next study is designed to remove it.
Why this works
If a novice can learn a skill inside an environment, the judgment lives in that environment. Each study measures whether students pick up, within an afternoon, communication judgment that normally takes a career of client conversations to build. Every measured gain is evidence the scenarios carry that judgment, and a model can train in the same environments.
Customer voice
“Going through those early stages of developing coaching/therapy confidence and competence is really tough. I just wish there was something around like this when I was starting up. I think this is priceless as it allows you to get reps at the sort of communication skills you'll need to truly master your craft.”
Dave Hadfield, Guest Lecturer in Coaching and consultant. Previously 25 years advising the elite players and coaches of New Zealand rugby teams, and the Toronto Blue Jays in Major League Baseball since 2017
Work with us
Put your experts' judgment in an environment.
Book a demo and we will walk through the Otago Polytechnic partnership, then map the same practice-and-measure loop onto the judgment your own organization runs on.