Point sourcing software at one of your open roles and it hands back a first list of candidates for that role. Is that list as good as the one the best sourcer on your team would build?
Asked about their own searches, recruiting leaders and hiring managers on teams without AI in hiring estimated that 49% of those searches begin aligned on requirements.¹ When a hiring search starts without that agreement, there’s no shared standard to judge any list against.
I would put your best sourcer first. Every tool’s first twenty profiles get read beside the twenty your sourcer builds for the same role, and speed decides only between the tools whose lists pass.
Below is that trial, run on a role you’re hiring for now, from the must-haves to the fairness check at the end. If a tool passes, your sourcer starts the next role like this one from its list, and still reads every name on it before anyone is contacted.
Test sourcing software on a role that is still open.
A role you’re hiring for now has current must-haves, and a hiring manager who can settle a disagreement about a profile. No one knows the right answer yet, so the lists get read without the hire in mind.
A head of talent could reasonably prefer a role you’ve already filled, because its hire gives a known answer. That works when the tool reads the same applicants your hires came from, which is the test behind a rubric for screening tools. A sourced list is mostly people who never applied, so a filled role’s hire tells you whether a tool found one person, and little about the other nineteen.
So I would keep a filled role out of this trial. Save its hires for a separate check of what a search of your own records turns up.
Write must-haves any two people would apply the same way.
Before any vendor runs a search, sit down with the hiring manager and write the must-haves out. Keep what you agreed apart from what’s still open, which is the split the intake template is built around. Only what you agreed goes into the trial, because each tool would guess at an open question in its own way.
Test each must-have against a profile. If two readers could disagree on whether it’s met, it’s too vague: one like “strong communicator” gets read two ways, so rewrite it as something a profile can show.
Chris Adams, Uber’s first sourcer, calls this step calibration. In this 10x Recruiting episode he describes taking about twenty candidate profiles into a live session with the hiring manager, before the search goes wide, and agreeing what to take out of the requirements and what to leave in.
Respondents on teams where AI is core to hiring estimated that they and their counterpart are aligned on requirements at the start of 68% of their searches.¹ That is their own estimate, and the survey carries no outcome data beyond self-report. Your trial needs that agreement in writing, because it gives every list the same standard.
From those must-haves, write one brief in plain words. It’s the brief every tool in the trial gets, and the one your sourcer builds their list from.
Get a first twenty from your best sourcer.
Ask your best sourcer, whoever on your team sources this kind of role best, to build their own first twenty from the written brief before they see any tool’s list. Without that list there’s no bar, or there’s one bent toward whichever tool someone saw first, and the overlap you count later stops meaning much.
Have them mark the people they would contact first, so you can see later whether a tool found the same people. Their list is the bar every tool’s list gets read against.
Whether an agent finds better people than a good sourcer would is left as an open question in a comparison of AI sourcing and Boolean search. Your sourcer’s twenty is how you answer it for one role.
Even a list as strong as your sourcer’s can rest on criteria your team has never read.
What should each vendor show you about its search?
Before anyone opens a profile, each vendor should show you the criteria its search ran on for your brief, the sources it searched for this role, and whether your team can read and correct those criteria.
That only tells you something if every vendor searched from the same words. So send each one the written brief, word for word. If a vendor rewords it, its list comes from a different search.
Reading the criteria tells you whether the search itself let a weak profile in. A criterion may have misread the brief, the market may hold few people who meet every must-have, or the tool may have read something into a profile that isn’t there. Only the first shows up in the criteria, and correcting them fixes it.
Where a criterion misreads the brief, have it corrected and the search rerun before anyone reads a profile, and note the correction beside that tool’s result. Note any criterion your team could read but couldn’t change, too.
If a vendor can’t show you the criteria, rule that tool out. Without them, you can’t tell whether a strong list came from a sound search.
Where your candidates’ data goes, and whether a vendor trains its models on it, are questions for vendor due diligence. Ask them in writing, apart from this one search.
Is each list as good as the one your best sourcer built?
A tool’s list passes when as many of its twenty meet every written must-have as your sourcer’s twenty do. That rule decides whether a list passes, and the other records below explain why it passed or fell short.
Set each tool’s twenty beside your sourcer’s and read every profile against the must-haves, your sourcer’s own included. If someone else can strip the tool names off the lists first, the read is blind. Write a line on which must-haves each profile meets.
Where your sourcer’s read and a tool’s pick disagree, the must-haves decide. Anything they leave open goes to the hiring manager.
Record these for each tool. Only the first decides whether its list passes.
| What you record | What it tells you | If it falls short |
|---|---|---|
| Must-haves met | Whether the list passes | Take each profile that fails a must-have back to the criteria the search ran on and find the line that let it in |
| Overlap with your sourcer’s list | Whether it found the people your sourcer marked to contact first, and anyone good your sourcer missed | Name each marked pick it missed, then check whether a line in its criteria or a source it never searched kept each one out |
| Sources searched | Whether it looked where the people your must-haves describe can be found | Check each miss against the sources the vendor named before the read, because a source it never searched explains a miss before any criterion does |
| Each candidate’s reason | Whether each candidate’s reason matches what the profile shows | Ask the vendor where each reason you can’t trace came from, and note it beside that tool’s count |
If a tool searched your applicant tracking system (ATS) and found no one your team already knows, an empty ATS result can say as much about your records as about the market. Check your records before you mark the tool down.
What does Metaview AI Sourcing do with the written brief?
Give Metaview AI Sourcing one plain-language brief and it searches your team’s past Metaview conversations, the candidates brought over from your connected ATS, and the web, returning each candidate with the reason it was surfaced.
Metaview captures every spoken word of the intake calls it records, so AI Sourcing can run your first search off the call where your hiring manager set out the role.
Accuracy and the data underneath it is our obsession. It’s how we built our notetaker. It’s how we built our sourcing agent.”
Every tool in the side-by-side gets the same written brief, so treat a run started from a recorded call as a second run and read it against the same must-haves.
For highly specialized technical roles at Lemonade, Talent Acquisition Partner Yael Golan Ben Zaken says, “Metaview creates a shortlist for me. It’s small but very strong, and exactly what I’m looking for.”
That’s one recruiter’s account, on one kind of role, of the lists she gets once she has given the search feedback over time. A first trial on your own role shows where a tool starts.
How much should speed count?
Speed only decides between tools whose lists both pass, and between those, take the faster one, because your sourcer can start reading sooner.
Asked how often a faster competitor takes a qualified candidate from them, 67% of respondents said every month.¹ That figure covers the whole hiring process, and a tool’s search time is one early step in it.
Deciding on speed alone is a fair call for a team with more open roles than sourcers. I would still run the timing second. A fast list that fails the must-haves leaves your sourcer doing the search after all, and a tool’s cost per qualified candidate counts only the candidates the hiring manager wants to meet.
When you time the tools, start the clock when each one gets the written brief and stop it when its twenty are ready to read. Any time spent correcting a tool’s criteria and rerunning its search counts toward its total.
Before you pick on either count, check that the trial was run evenly.
Was the trial fair to every tool?
It was fair if these held, and where one didn’t, its fix comes ahead of any comparison.
- The must-haves came first. If they were written or changed after the first lists came in, rebuild them from the intake notes or recording. Then rerun every tool and read every list again against them.
- Your sourcer’s list came first. If your sourcer saw a tool’s list before finishing their own, their list may lean toward it, so treat every comparison in this trial as unproven and collect the sourcer’s list first next time.
- Every tool got the same brief. If a vendor reworded it during a demo or ran its search from a different document, rerun that tool from the written brief before you compare anything.
- The hiring manager settled what the must-haves left open. If a vendor, or anyone else, made that call on a disputed profile, take it to the hiring manager with the must-haves beside it.
I would keep the tool whose first twenty held up beside your sourcer’s list, from a vendor that showed you its criteria before anyone opened a profile. If no tool gets there, your sourcer keeps building those lists. If one does, your sourcer opens the next role like this one from that tool’s list and reads it against the must-haves the hiring manager wrote.
Judge every sourcing tool by the list your best sourcer builds.
Book a demo of Metaview AI Sourcing with your must-haves in hand.
Frequently asked.
What if sourcing software misses candidates a basic search finds?
Check the sources the tool searched first, then have the vendor correct the criterion that kept those people out and rerun the search once, noting the correction. If they still don’t come back, note it in the tool’s overlap record.
What if your best sourcer’s list doesn’t meet every must-have?
Keep their count as the one a tool has to match, since it’s what your team finds today. If no list meets a must-have, the market may hold few people who do, so raise it with the hiring manager for the next search and leave this trial’s must-haves as they are.
Can you test sourcing software on a niche or hard-to-fill role?
Yes, and I think it tells you more there. When your sourcer can name most of the people who could do the job, each one missing from a tool’s twenty can be set beside the sources it searched, so a gap in reach shows up in the first run.
What if the role’s requirements change partway through the trial?
Then the first brief describes a different search, and the first run ends there. Write the new must-haves with the hiring manager and run every tool again. Your sourcer has now seen every tool’s first list, so count overlap in the second run as unproven.
Will sourcing software contact candidates during a trial?
Agree in writing with each vendor, before the first search, that no tool sends anything to a candidate during the trial. Reaching people is a second test for the tools whose lists pass, and before anyone writes to a name from any list, check who stays off limits.
Will your sourcer still review every profile after the trial?
Yes. The tool runs the search, and your sourcer still reads every list before anyone is contacted, because that read is where the judgment stays.
Sources.
¹ Metaview, 2026 AI & Hiring Alignment Report: a 19-question survey of 505 recruiting leaders and hiring managers at companies with 200 or more employees in North America and Europe, the Middle East, and Africa, run with Cint.
