17 questions to ask a cyber assessment vendor before you buy

Most cyber assessment demos are the same. You get a polished walkthrough, a challenge library count, a mention of AI, and a slide about anti-cheat. Everyone looks competent because everyone is demoing the parts that look good.

Here are the questions that actually separate them. Where the answer would rule us out too, I have said so.

On the content

1. Can I see a challenge from the library that matches the role I am hiring for, right now, on this call?

The single most useful question, and the one that most reliably ends conversations. A vendor with real depth in your discipline shows you one. A vendor with a thin library will offer to send something later, or show you a general “cyber” challenge and describe how it could be adapted.

2. Who writes the challenges, and what did they do before?

Ask for names and backgrounds. Content written by practitioners looks different from content written by an instructional design team working off a syllabus. In security the difference shows up immediately in whether the scenarios are plausible.

3. Show me a question that a candidate could not answer by searching for it.

If everything in the library is recall, you are buying a quiz. The answer should live inside an artefact the candidate has to work through, not in their memory or a browser tab.

4. How many of your challenges are multiple choice?

Not a disqualifier by itself. Multiple choice anchored to real output can be a legitimate format. But if the answer is “most of them”, you are looking at a knowledge test with a security skin.

5. Is your content public anywhere?

This one matters more than it sounds. If the challenges come from a training platform’s public library, walkthroughs exist on YouTube and Reddit, and the candidate who has seen the room beats the candidate who can do the job. Ask directly whether their assessment content is the same content they sell as training.

6. What happens when I need something the library does not cover?

Every library has gaps. The question is what happens next: can something be built, how long does it take, who builds it, and does it cost extra.

On the assessment itself

7. Who builds the assessment for my specific role, and how long does it take?

Some vendors sell you a platform and the work of designing a valid assessment lands on your team. That is not necessarily wrong, but it is a real cost that never appears in the quote, and it is usually paid by the senior engineers you least want to interrupt.

8. How do you calibrate difficulty to seniority?

If the same assessment goes to Tier 1 and Tier 3 candidates, it will be wrong for at least one of them. Ask specifically how a junior assessment differs from a senior one.

9. What does the candidate actually get, and how long does it take them?

Ask for the candidate’s view, not the admin’s. Assessments that take three hours will not be completed by people with other offers, and you lose the strongest candidates first.

10. What happens if a candidate has an accessibility requirement?

Ask what accommodations exist and how they are requested. A vendor who has not thought about this has not deployed at scale, and you inherit the legal exposure.

On scoring and output

11. Can I see the actual answers, or only the score?

Scores without underlying evidence are unfalsifiable. You want to be able to look at what a candidate wrote when you are deciding between two of them, and you want it available months later when someone asks why you rejected a candidate.

12. How is partial credit handled?

Binary right-or-wrong scoring throws away the difference between a candidate who reasoned correctly and slipped, and one who guessed. In security that distinction is most of the signal.

13. Do you report per-skill breakdown or just a total?

A single number ranks a spiky candidate and a uniformly average one identically. They are not the same hire.

On integrity

14. What exactly do you detect, and what do you do about it?

Push past “AI-powered proctoring”. Ask what signals are captured, what is presented to the reviewer, and crucially whether the system makes a judgement or reports evidence. Any vendor whose product automatically rejects candidates on a cheating score is selling you a false-accusation machine.

15. What is your false positive rate, and what happens when you are wrong?

Most vendors will not have a good answer. That is informative. The follow-up matters more: is there a human review step before a candidate is disadvantaged?

16. If a candidate uses an LLM well, does your assessment still work?

The best answer is not “we detect it”. The best answer is that the tasks are built so an LLM does not help much, because the answer only exists in the artefact in front of the candidate. Detection is an arms race. Task design is not.

On the commercials

17. What am I actually committing to, and what happens if hiring stops?

Annual platform subscriptions assume steady hiring. If you hire in bursts, you will pay through the quiet months. Per-role pricing assumes you hire occasionally. If you hire constantly, it gets expensive. Neither is wrong. They are just shaped for different buyers, and vendors will not volunteer which one you are.

Also ask what is not in the number: implementation fees, ATS integration, additional seats, annual uplift clauses, overage charges when you exceed an assessment allowance.

The questions where we would come off badly

A list like this from a vendor is usually built so the vendor wins every question. Here are the ones where we do not.

“Can we run this entirely ourselves, without you involved?” Not really. CyberHire is a managed service. We build the assessment and run the process. If you have an in-house team who want to own assessment design and operate a platform themselves, a self-serve tool is a better fit and we are the wrong shape.

“Will my board recognise the name?” No. HackerRank, Hack The Box and Immersive Labs are established brands with a decade or more of presence. We are not, and if procurement comfort is a significant factor for you, that is a real point against us.

“Can you also handle our non-technical hiring?” No. We do cyber security roles. If you want one platform covering sales, marketing, finance and engineering as well, a broad platform like TestGorilla covers far more ground than we do, and running two vendors has a real overhead.

“Do you do training as well as hiring?” No. If you want assessment and upskilling under one contract, Hack The Box and Immersive Labs both do that properly and we do not do it at all.

“How many customers do you have, and can I speak to three?” We are early. A vendor with hundreds of deployments can hand you a reference list and we cannot match that yet. Ask us anyway, and weigh the answer.

That list is not modesty. It is the fastest way to work out whether a conversation is worth either side’s time, and any vendor unwilling to give you their version of it is wasting yours.

How to run the evaluation

Three practical suggestions.

Use a real role. Evaluate against a live vacancy with real candidates, not a hypothetical one. A demo on a made-up role tells you about the demo.

Sit the assessment yourself. Twenty minutes as a candidate tells you more than an hour of vendor slides. If it feels like a quiz to you, it will feel like a quiz to a senior candidate, and they will judge your company for it.

Ask what happens when it goes wrong. A candidate’s environment crashes halfway through. Someone disputes a result. A hiring manager disagrees with a score. The answers tell you whether the vendor has actually run this at scale or has only ever demoed it.

The wider method, if you are still working out what a good assessment looks like, is in how to assess cybersecurity candidates. If you want to see the output before committing to anything, we publish a sample report.

One honest sentence

The most useful question on this list is the first one, because a vendor who cannot show you a role-relevant challenge on the call does not have one.