After a panel interview on a team with a few dozen interviewers, two panelists who used the same scorecard and heard the same answer have written it up as a “strong yes” and a “no,” and both of them meant well. They read the scorecard’s scale differently, which is how interviewer bias usually shows up on a large team, and on a loop with only one of them in the room, the candidate’s result would have come down to which one it was.
With five interviewers, a recruiter spots that in the room and sorts it out over coffee. With dozens spread across roles and time zones, and new ones joining faster than anyone can sit in on their interviews, no one in any single debrief sees the pattern, and it repeats in every loop they share.
So I’d measure progress on interviewer bias at scale somewhere other than the training log: in whether two interviewers who hear the same answer give it the same rating. That agreement can be built and then kept as the bench grows, and this guide hands over how.
Why interviewer bias grows with the bench.
Every interviewer brings the familiar biases into the room. Inside one interview you might catch them as they happen, but a large bench runs too many interviews for anyone to sit in on. A talent team can see the trace each bias leaves on an interviewer’s scorecards over a few months instead, and that trace is the part of the problem a team can measure.
| Bias | In a single interview | Across one interviewer’s scorecards |
|---|---|---|
| Affinity | A candidate from the interviewer’s old employer gets an easy follow-up and a warm write-up. | Ratings run higher on every competency for candidates whose former employers or schools the interviewer shares. |
| Halo or horns | One strong answer, or one weak one, colors every rating after it. | Competency ratings move together, so candidates score all high or all low and rarely a mix. |
| Leniency or severity | The interviewer treats a “3” as a pass, where a colleague treats it as a polite no. | Their ratings sit well above or well below the rest of the loop on the same candidates. |
| Central tendency | Unsure, the interviewer avoids both ends of the scale. | Almost every rating is the middle score, so their scorecards rarely separate one candidate from another. |
Bias training teaches the names in the first column. It can’t give forty interviewers one picture of a “3” on stakeholder management, and the right-hand column is where the missing picture shows up, so I’d manage that column.
Order effects, where the previous candidate becomes the bar, need a different fix. Scoring each candidate with only their own notes open is that fix. It’s set out in full in a post on contrast bias in interviews.
Calibrate the bench on one recorded interview.
A calibration meeting where interviewers describe what a good answer would sound like ends in easy agreement, because no one in it has rated anything yet. Calibrating is meant to check that everyone who interviews for a role reads the scorecard’s scale the same way, and that only shows once each of them has put a rating on a real answer.
I’d run it on a recorded interview instead. Pick a real interview for the role, one where the candidate gave a full answer on the competency the bench argues about most. Send it to everyone who interviews for that role, hiring manager included, and have each person rate it alone against the scorecard and send the rating in before the session.
Put the ratings in one sheet and open the session on the competency with the widest spread. Ask the people at each end to name the moment in the recording their rating rests on. When both name the same answer, they’re reading the level description differently, so rewrite it before the session ends, in words that would have put them on the same level.
When they name different moments, the bench disagrees about the candidate’s words themselves. Play both moments back, settle what was said, and have the two re-rate. The level description may be fine as it stands, and no one leaves having been told they were wrong.
The rubric-writing side, turning a level into something two people can check, is covered in a post on interview rubrics.
Read a quarter of ratings by interviewer.
A calibration session fixes the scale for one competency on one day. To see where it has split since, export a quarter of submitted scorecards for one role family from your applicant tracking system and sort them by interviewer instead of by candidate. Four reads are worth making:
- Lean. Compare each interviewer’s ratings with the rest of the loop on candidates they both met. An interviewer who happened to meet stronger candidates looks lenient on raw averages, so shared candidates are the only fair comparison.
- Spread. Count how often each interviewer uses the top and bottom of the scale. A scale used only in the middle gives the debrief very little to work with.
- Lockstep. Look for interviewers whose competency ratings always move together, which is the halo row from the table above.
- Recommendation against ratings. Find the scorecards where the overall recommendation points one way and the competency ratings point the other. In those, an impression outranked the interviewer’s own evidence.
Then be careful with what you’ve found. A gap between two interviewers doesn’t say which of them was wrong, and a handful of scorecards says almost nothing about anyone. I’d keep it off leaderboards. The sheet is for a private conversation with the interviewer, and for choosing the competency the next calibration session works on.
The pattern a talent team most wants to find at scale is whether candidates from one group fare differently from others. That analysis involves protected characteristics and the law where you hire, so it belongs with your legal and people analytics teams. Your part is making sure every decision has a submitted scorecard with written evidence behind it, so their review has something to read.
Bring new interviewers onto the bench on purpose.
A hiring push is when the bench grows fastest, and when its scale is least known. Each new interviewer brings their own reading of the scorecard, and their first ratings carry as much weight in a debrief as anyone’s.
I’d give every new interviewer two checks before they interview alone. First, they rate the calibration recording, and their ratings go into the same sheet as the bench’s, so you can see where their scale sits before a candidate depends on it. Then a calibrated interviewer sits in on their first interviews for the role and submits a separate scorecard, and the two are read side by side before the new interviewer’s ratings reach a debrief on their own.
This costs a calibrated interviewer some hours in a busy month, which is exactly when a team will want to skip it. I’d drop almost anything else first, because a new interviewer with an unchecked scale shapes every loop they join for the rest of the push.
Where Metaview fits in calibrating a large interview bench.
In the session itself, the bench is split on one answer, and the useful thing on screen is that answer. For the interviews that both the candidate and the interviewer agreed to have recorded, under a consent process your team runs, the Metaview Notetaker joins as a visible participant and records and transcribes the conversation, then lays it out as structured notes.
Because Metaview captures every spoken word, the session can replay the disputed answer instead of rebuilding it from several people’s memories of it.
becoming a great interviewer is a skill that’s learned over time and requires regular, personalized feedback, which can be tough to scale.”
Feedback on a rating needs something specific to point at. A recording every interviewer on the loop has rated gives it that, because the conversation can stay on one answer and the level description it was measured against.
To pick the recording, Metaview Reports can surface the role’s recorded interviews where candidates talked about a stakeholder who pushed back on them, and from those you choose the one where the disputed competency got the fullest answer.
Which interviewer is right about a candidate stays a judgment for the people in the room. The recording makes sure they’re arguing about the same answer.
Play the same recording to the bench again.
A couple of months after the first session, send the same calibration recording back to everyone who rated it, and have each person rate it again, alone, before anyone discusses it. Put the new ratings beside the old ones in the same sheet and read the two rounds together:
- If the ratings on the disputed competency land closer together than last time, the rewritten level description is doing its job. Pick the next competency with a wide spread and calibrate on that.
- If they spread as widely as before, the description still reads two ways. Rewrite it again with the two people at either end in the room.
- If most of the bench has converged and one interviewer still sits apart, the rubric can stay as it is, and the next step is a private talk with that person about how they read the scale.
- If the newest interviewers sit furthest from everyone else, one of the two onboarding checks is being skipped. Find out which before the next hiring push.
Read the two rounds by one figure: the spread on the disputed competency, from the lowest rating to the highest, for an answer that hasn’t changed. A narrower spread the second time means your interviewers are reading the scale more alike, and that’s subjectivity you can measure coming down across the whole bench.
Get to the interview answer your bench should rate first.
Use Metaview Reports to surface the recordings where your bench’s disputed competency came up, and open the next session on the fullest answer.
Frequently asked.
Who should rate the calibration recording?
Include the interviewers from other teams who join the loop now and then, because theirs is the scale least likely to have been seen. Someone who never assesses a given competency can leave that line blank, and a blank tells you more than a guess.
Should interviewers still take unconscious bias training?
Yes, and I’d time it to be useful. A short refresher just before an interviewer’s first calibration session gives them the vocabulary the session uses, which is a better moment for it than a once-a-year module.
What if the hiring manager’s ratings sit furthest from the rest of the bench?
Treat it like anyone else’s gap: in private, and about how they read the scale. It counts for more, because the hiring manager’s view weighs heavily in the debrief, so I’d raise it before the next candidate reaches them.
How often should a team calibrate its interviewers?
Run it when a role opens with a mostly new bench, and again whenever the re-rating shows a competency spreading out. A fixed date on the calendar is easy to skip in a busy month, and a trigger tied to the bench changing is harder to argue with.
Does calibration work for competencies with no right answer?
It works as long as each level describes evidence instead of a feeling. “Shows ownership” is hard to calibrate, while “described a decision they made and what they’d change about it” is easy, because two people can check a recording for it.
