Most evaluations of an artificial intelligence (AI) screening tool end the same way. The vendor ticks every box on a long feature checklist, the demo runs on a role it has seen before, and the buying team still can’t say whether the AI screening tool’s judgment would hold up on their own applicants. A feature checklist rewards whoever built the most features. It can’t tell you whether the ratings agree with your recruiters.

Whether the ratings agree with your recruiters is a question with an answer you can ask for. In Metaview’s aggregate data on AI-rated applications, applicants rated “Great” advance at 17.2% and applicants rated “Poor” at 5.6%, counting only applicants a recruiter went on to progress or reject. Ratings and recruiter decisions move together, and recruiters still advance some people from the bottom band, so the rating doesn’t settle the decision on its own.

We’d score what a vendor can prove and let a set of non-negotiables end an evaluation before anyone averages anything. This guide gives you 30 questions across six areas, what a strong answer looks like for each, the evidence to ask for, a 1 to 5 scale, and the rule for when to stop. It works whether the tool reads applications, runs screening calls, or does both.

Evaluate an AI screening tool on what it can prove.

An AI screening tool is software that reads applications, holds screening conversations with candidates, or both, and gives your recruiters a rating with a reason for each person. Evaluating one comes down to three questions: does it fit the way your team screens, can you trust its judgment, and can you live with it once it’s running? The 30 questions below cover all three across six areas.

Run the evaluation the same way for every vendor. Shortlist three tools at most, give each one the same roles and the same past applicants, and record a score, a note on the evidence, and where that evidence came from for every question. Bring a recruiter, someone from your people or legal team, and the person who owns your applicant tracking system (ATS), because each of them will spot gaps the others miss.

Four rules keep the scores honest:

• Keep claims and evidence apart. Write down what the vendor said and what you saw in separate columns.
• Don’t score a feature because it exists. Score how well it works, what it takes to adopt, and what controls sit around it.
• Mark your non-negotiables before the first demo, so a polished pitch can’t move them.
• Pilot your top choice on real roles, with the kinds of candidates you see and the way your recruiters work, before you sign.

How it fits the way your team screens.

This area checks whether the tool does the daily work of screening the way your team needs it done: set up per role, reading what candidates send, applying your rules, and handing recruiters a queue they can work.

Question 3 is the only non-negotiable in this area, and it’s the one we’d raise first in any demo. Ask it plainly: is there any configuration in which the tool rejects a candidate without a person pressing a button? A tool that can act on its own is a different purchase, with a different approval path and different people who need to sign it off. We looked at how this works in practice in how AI reads every inbound application before a human decides.

For question 1, look for criteria you can read. In Metaview’s Application Review, which reads and rates every inbound application, each role has an Ideal Candidate Profile built from the job description, and a person reviews and approves it before evaluation starts.

On question 2, also ask how the tool treats text written to game it. We looked at what the research says in whether candidates can prompt-inject a resume.

Question 5 is where high-volume tools earn or lose trust. Siadhal Magos, Metaview’s co-founder and chief executive, has described what goes wrong when people read every application by hand:

Whether its judgment holds up.

This is the area where a demo tells you least, because the demo runs on candidates the vendor chose. Five questions test whether the ratings are relevant, consistent, monitored, explained, and open to challenge.

Every question here except 7 is a non-negotiable. Question 6 is the one most vendors answer with a slide, so ask for the numbers instead. Metaview’s aggregate data shows the shape of answer to look for, where advance rates track the rating:

The middle bands sit where you’d expect, at 12.1% for “Good” and 7.8% for “Okay”, so every step down the rating is a step down in how often recruiters advance someone. Read it carefully, though. Recruiters could see the rating when they decided, so this shows the two moving together and leaves open whether the rating is right. Question 6 asks for a blind test on your own applicants for exactly that reason, and numbers like these start the conversation with a vendor rather than end it. We looked at the overrides in more detail in what happens when recruiters override the AI’s application score.

Questions 9 and 10 are easier to test, because you can see them. Pick an applicant yourself, read the reason, and overrule it.

How it connects to your stack and rolls out.

Here the questions move to the work around the tool: the systems it has to talk to, the effort to get it running, and who looks after it once it’s live.

Question 11 is the non-negotiable here, and it’s where vague answers hide. “We integrate with your ATS” can mean candidates come in, decisions go back, or both, and the answer often differs from one ATS to the next. Ask for your system by name, and trace one real candidate end to end before you score it.

What happens to candidate data.

Every question in this area is a non-negotiable, and most of the answers live in documents rather than demos. Send the requests in your first week, so legal and security review can run alongside the product evaluation instead of after it.

Question 20 deserves the most patience. Ask to read the method behind any fairness claim: what population the testing used, what it compared, and how often it runs. A vendor that has done the work will have it written down. We set out how bias gets into screening, with or without software, in our guide to catching and reducing candidate screening bias.

What candidates and recruiters have to live with.

A screening tool sits in front of every applicant and every recruiter, so small frictions add up fast. This area checks the experience on both sides.

Questions 21 and 25 are the non-negotiables here, and both protect candidates who would otherwise drop out for reasons unrelated to whether they could do the job. Completion rate is the number vendors reach for in this area. It’s worth having, but it can’t answer these questions alone. We covered what it can and can’t tell you in our piece on screening completion rate, and where the screen should sit in your process in where to put AI screening in your hiring process.

This episode of 10x Recruiting, Metaview’s podcast, covers why being human is still an edge in hiring, which is the thinking behind question 25.

What it costs, and who stands behind it.

The last area checks whether the tool will still be the right choice in its third year: the full cost, the volume it can carry, the reporting you’ll get, the support behind it, and the vendor itself.

Question 26 is the non-negotiable here. Question 30 is the easiest to skip and one of the most useful, because a reference who compared several tools before buying can tell you what the demos didn’t.

Workleap is one example of that kind of buyer. Its recruiters were reviewing 200 to 300 candidates per role by hand, each taking 30 to 45 seconds, and senior recruiter Johnny Drexhage tested quite a few tools before the team settled on Metaview’s Application Review. He said that “what immediately stood out to me is how intuitive and effective it was from the very start,” and he reported that “It’s reduced my screening time by up to 50%.” The full story is in the Workleap case study.

How to score it, and when to stop.

Score every question from 1 to 5. The scale only works if you hold one line: claims top out at 3. Anything higher has to be seen working off script, or proven in writing.

Then work it out in this order:

1. Leave anything you couldn’t assess blank rather than giving it a 1, and note how many questions each tool was scored on. A tool scored on 20 questions can’t be compared with one scored on 30.
2. Check the non-negotiables first: questions 3, 6, 8, 9, 10, 11, 16 to 21, 25, and 26. Any of them below 4 is a reason to stop, whatever the average says, because a non-negotiable needs evidence rather than a claim. A blank non-negotiable stops you too, until it’s assessed.
3. Average each of the six areas, then all 30. Read the area averages before the overall one, because a strong total can hide a weak area.
4. Use the overall average to decide the next step. Four or above: take it to a pilot and start on commercial terms. From three up to four: pilot it only with conditions, and write down each gap, its fix, and how the pilot will test it. Below three: drop it unless there’s a strong strategic reason and a plan to close the gaps.

Where Metaview lands on the non-negotiables.

Here’s how Metaview answers several of the non-negotiables. Ask us for the evidence behind each one, and put the rest of the questions to us in the demo, the same as you would with any vendor.

• Who decides (question 3): Application Review never auto-rejects, and a recruiter makes every accept and reject call.
• Reasons (question 9): Application Review rates each applicant’s fit, from “Great” to “Poor”, and shows the reasoning. Screening, Metaview’s voice-led screening agent, returns a fit rating with a documented reason, plus the recording and a transcript mapped question by question. For the screening calls your recruiters still run themselves, Metaview’s Notetaker captures every spoken word where consent has been given, so the reason for a decision sits next to what was said.
• Candidate data (questions 16 to 19): Metaview is System and Organization Controls (SOC) 2 Type II certified and compliant with the General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA), with 256-bit encryption and SSO. Interview data is hosted in the Amazon Web Services environment in the United Kingdom, retention can match your company’s policy, and you can delete recordings at any time.
• Fairness (question 20): Screening adheres to the European Union’s AI Act, New York City Local Law 144, and guidelines for bias and adverse-impact testing.
• Your ATS (question 11): Metaview’s integrations list covers more than 100 applicant tracking systems, and which way data flows differs by system and by product. Ask us about yours and trace a candidate through it, the same test we’d tell you to run on anyone.

Run it on your own roles.

Take the 30 questions into your next vendor call and score them live, while someone else drives the product. Anything you end up scoring from memory afterward was probably a 3 at best.

Then treat a strong result as a reason to run a pilot. Buy only after the pilot has shown the tool working on your own roles and applicants, with your recruiters deciding who moves forward.