17 questions to ask a cyber assessment vendor before you buy

Most cyber assessment demos are the same. You get a polished walkthrough, a challenge library count, a mention of AI, and a slide about anti-cheat. Everyone looks competent because everyone is demoing the parts that look good.

Here are the questions that actually separate them. Where the answer would rule us out too, I have said so.

On the content

1. Can I see a challenge from the library that matches the role I am hiring for, right now, on this call?

The single most useful question, and the one that most reliably ends conversations. A vendor with real depth in your discipline shows you one. A vendor with a thin library will offer to send something later, or show you a general “cyber” challenge and describe how it could be adapted.

2. Who writes the challenges, and what did they do before?

Ask for names and backgrounds. Content written by practitioners looks different from content written by an instructional design team working off a syllabus. In security the difference shows up immediately in whether the scenarios are plausible.

3. Show me a question that a candidate could not answer by searching for it.

If everything in the library is recall, you are buying a quiz. The answer should live inside an artefact the candidate has to work through, not in their memory or a browser tab.

4. How many of your challenges are multiple choice?

Not a disqualifier by itself. Multiple choice anchored to real output can be a legitimate format. But if the answer is “most of them”, you are looking at a knowledge test with a security skin.

5. Is your content public anywhere?

This one matters more than it sounds. If the challenges come from a training platform’s public library, walkthroughs exist on YouTube and Reddit, and the candidate who has seen the room beats the candidate who can do the job. Ask directly whether their assessment content is the same content they sell as training.

6. What happens when I need something the library does not cover?

Every library has gaps. The question is what happens next: can something be built, how long does it take, who builds it, and does it cost extra.

On the assessment itself

7. Who builds the assessment for my specific role, and how long does it take?

Some vendors sell you a platform and the work of designing a valid assessment lands on your team. That is not necessarily wrong, but it is a real cost that never appears in the quote, and it is usually paid by the senior engineers you least want to interrupt.

Ask the follow-up too: can we run this entirely ourselves? For CyberHire the answer is yes: send us the job spec and we build the assessment for you, at the right level and in your branding. Your team invites candidates and reads the results. If you want to change anything, you can edit it in your own workspace, but you never have to.

8. How do you calibrate difficulty to seniority?

If the same assessment goes to Tier 1 and Tier 3 candidates, it will be wrong for at least one of them. Ask specifically how a junior assessment differs from a senior one.

9. What does the candidate actually get, and how long does it take them?

Ask for the candidate’s view, not the admin’s. Assessments that take three hours will not be completed by people with other offers, and you lose the strongest candidates first.

10. What happens if a candidate has an accessibility requirement?

Ask what accommodations exist and how they are requested. A vendor who has not thought about this has not deployed at scale, and you inherit the legal exposure.

On scoring and output

11. Can I see the actual answers, or only the score?

Scores without underlying evidence are unfalsifiable. You want to be able to look at what a candidate wrote when you are deciding between two of them, and you want it available months later when someone asks why you rejected a candidate.

12. How is partial credit handled?

Binary right-or-wrong scoring throws away the difference between a candidate who reasoned correctly and slipped, and one who guessed. In security that distinction is most of the signal.

13. Do you report per-skill breakdown or just a total?

A single number ranks a spiky candidate and a uniformly average one identically. They are not the same hire.

On integrity

14. What exactly do you detect, and what do you do about it?

Push past “AI-powered proctoring”. Ask what signals are captured, what is presented to the reviewer, and crucially whether the system makes a judgement or reports evidence. Any vendor whose product automatically rejects candidates on a cheating score is selling you a false-accusation machine.

15. What is your false positive rate, and what happens when you are wrong?

Most vendors will not have a good answer. That is informative. The follow-up matters more: is there a human review step before a candidate is disadvantaged?

For what it is worth, we do not publish a false positive rate either. Our integrity flags are evidence for the reviewer, not a verdict: the platform records what happened and a person on your side decides what it means.

16. If a candidate uses an LLM well, does your assessment still work?

The best answer is not “we detect it”. The best answer is that the tasks are built so an LLM does not help much, because the answer only exists in the artefact in front of the candidate. Detection is an arms race. Task design is not.

On the commercials

17. What am I actually committing to, and what happens if hiring stops?

Annual platform subscriptions assume steady hiring. If you hire in bursts, you will pay through the quiet months. Per-role pricing assumes you hire occasionally. If you hire constantly, it gets expensive. Neither is wrong. They are just shaped for different buyers, and vendors will not always volunteer which one you are.

Also ask what is not in the number: implementation fees, ATS integration, additional seats, annual uplift clauses, overage charges when you exceed an assessment allowance.

Our own answer, so you can hold us to the same standard: you buy CyberHire either per role or as an ongoing hiring programme. A role campaign covers one live role for a 90-day window, with 250 completed assessments as standard, and the allowance can be extended if a role draws more applicants than expected. Usage is counted on completed assessments, not invitations sent, so candidates who never start do not cost you anything. Teams hiring continuously can agree a tailored annual programme instead. Both are quoted by talking to us, around your actual hiring plan.

The questions where we would come off badly

A list like this from a vendor is usually built so the vendor wins every question. Here are the ones where we do not.

“Will my board recognise the name?” No. The established assessment and cyber training brands have a decade or more of presence. We are not one of them, and if procurement comfort is a significant factor for you, that is a real point against us.

“Can you also handle our non-technical hiring?” No. We do cyber security roles. If you want one platform covering sales, marketing, finance and engineering as well, a broad generalist testing platform covers far more ground than we do, and running two vendors has a real overhead.

“Do you do training as well as hiring?” No. If you want assessment and upskilling under one contract, the larger cyber training platforms do that properly and we do not do it at all.

“How many customers do you have, and can I speak to three?” We are early. A vendor with hundreds of deployments can hand you a reference list and we cannot match that yet. Ask us anyway, and weigh the answer.

That list is not modesty. It is the fastest way to work out whether a conversation is worth either side’s time, and any vendor unwilling to give you their version of it is wasting yours.

How to run the evaluation

Three practical suggestions.

Use a real role. Evaluate against a live vacancy with real candidates, not a hypothetical one. A demo on a made-up role tells you about the demo.

Sit the assessment yourself. Twenty minutes as a candidate tells you more than an hour of vendor slides. If it feels like a quiz to you, it will feel like a quiz to a senior candidate, and they will judge your company for it.

Ask what happens when it goes wrong. A candidate’s environment crashes halfway through. Someone disputes a result. A hiring manager disagrees with a score. The answers tell you whether the vendor has actually run this at scale or has only ever demoed it.

The wider method, if you are still working out what a good assessment looks like, is in how to assess cybersecurity candidates. If you want to see the output before committing to anything, we publish a sample report.

One honest sentence

The most useful question on this list is the first one, because a vendor who cannot show you a role-relevant challenge on the call does not have one.