What 8,600 practice questions reveal about exams
We measured our whole question bank: stem length, domain spread, open-book share. The findings say more about the exams than about the writing. Free to cite.
In short
- Mean stem length across the corpus is 43 words, but the range runs from 25 to 130 — a factor of five, driven by the exam rather than the writer.
- Scenario-led exams like PMP cannot have short questions: the scenario is the question. Recall-led exams like LPIC-3 cannot support long ones.
- Multi-select questions cluster almost entirely in the IT certifications and are absent from the inspection and safety exams.
- 960 questions across the corpus are marked open-book, and every one of them belongs to an API inspection exam.
- Blueprint domain counts range from 3 to 13, which changes what 'revise the weak domain' can even mean.
Almost everything written about certification exams quotes the awarding body’s published outline. That is fine as far as it goes, but it describes the specification rather than the thing. We had a corpus sitting there — every question we have written, across ten professional certifications — so we measured it.
The full report is published and recomputed on every deploy. Here is what stood out.
How long is a typical certification practice question?
The mean stem across the whole corpus is 43 words. That number is close to useless on its own, because the per-exam means run from 25 words to 130 — a factor of five.
The spread is not stylistic. It is structural, and it falls out of what each exam is trying to test:
- PMP: 130 words. A situational-judgement exam has to describe the situation before it can ask anything. The scenario is the question, and you cannot compress it without destroying what is being assessed.
- API 510, 570, 653: 42-43 words. Enough to establish a vessel, a reading and a condition, then a decision.
- MB-330: 29 words. Configuration questions name a module and a setting.
- LPIC-3: 25 words. A command-recall exam has nothing to describe.
The practical consequence is about reading speed, not writing quality. On a 180-question PMP paper at 130 words a stem, you will read roughly 23,000 words under time pressure — before you look at a single option. Nobody budgets for that, and it is the most common reason people run out of clock on a paper they knew.
Which exams allow more than one answer?
Multi-select questions cluster almost entirely in the IT certifications, and are absent from the inspection and safety exams in this corpus.
That is not an accident of drafting. Selecting exactly the right set of commands, packages or settings is the skill an IT exam is testing, and a single-answer format cannot express it. An inspection exam asks whether a condition is acceptable — a question with one defensible answer, or it is a badly written question.
If you are moving between the two worlds, this is the format shock: an IT candidate sitting an inspection paper keeps looking for the second right answer, and an inspection candidate sitting an IT paper stops reading at the first one.
Where does open-book sit?
960 questions across the corpus are marked open-book, and every single one belongs to an API inspection exam — the only certifications here that split a sitting into a closed-book half and an open-book half with the references on screen.
We mark them in the bank rather than shuffling them together, because rehearsing the real rhythm is most of the value. Practising open-book lookups without a clock teaches you the wrong skill: on the day, the constraint is not whether you can find the clause but whether you can find it fast enough.
How many domains does an exam have?
Between 3 and 13, which sounds like trivia and is not.
At three domains — PMP’s People, Process and Business Environment — a weak domain is a third of the paper, and “revise your weak domain” means weeks of work. At thirteen, a weak domain might be thirty questions, and the same advice means an evening. The number changes what the advice can even mean, and generic study guidance never mentions it.
It also changes what a practice score tells you. A 70% on a three-domain exam hides much less than a 70% on a thirteen-domain one, where you can be strong in eleven areas and comprehensively lost in two without the total moving much.
What should you actually do with this?
Three things:
- Time your reading, not just your answering. Work out the word count of a full paper for your exam and rehearse at that volume. The exams with long stems punish this hardest.
- Learn your exam’s answer format before the day. Whether more than one option can be right is not a detail you want to discover in the first sitting.
- Read every explanation, including on questions you got right. A guess that lands looks identical to knowledge in a score. The explanation is the only place the difference shows up.
The underlying data is free to cite under CC BY 4.0, and if you want a cut we have not published, ask.
Where to start
- Browse every certification — ten exams, each with a free ten-question sample
- The full question bank report — domain tables, stem lengths, coverage
- API 510, 570 or 653: which should you take first?
Frequently asked questions
How long is a typical certification practice question?
Across 8,600 questions the mean stem is 43 words. That average is close to useless on its own: the per-exam means run from 25 words to 130, and the difference is a property of the exam format rather than of the writing.
Why are PMP questions so much longer than LPIC-3 questions?
Because a situational-judgement exam has to describe the situation before it can ask anything. The scenario is the question. A command-recall exam has no scenario to describe, so its stems are short by necessity, not by choice.
Do more answer options make a question harder?
Not reliably. Most of this corpus sits at four options because that is what the exams use. Difficulty comes from how close the distractors are to being right, which is a property no option count can capture.
What share of practice questions should have explanations?
All of them. Explanation coverage is the clearest line between a bank you learn from and one you merely score against — a score tells you that you were wrong, an explanation tells you why.
Where can I see the underlying data?
The full report is published at /research/question-bank-report/ and recomputed on every deploy. It is free to cite under CC BY 4.0. No question text is published, only counts and distributions.
References
- CertCrafter question bank composition report — CertCrafter
CertCrafter is independent and is not affiliated with or endorsed by the certification bodies named above.
- #exam strategy
- #original research
- #question design
- #study methods