How to choose a quality review sample for a capacity pilot

01

Define the decision and standard

Write down the decision the review must inform: continue, revise, pause or expand the pilot. Define an acceptable output using the requirements for that work, and agree which defects are serious enough to require immediate escalation. Keep any mandatory inspections and approvals in place.

Choose a reviewer with the authority and knowledge to assess the work. Record the item, work type, reviewer, date and result, including any correction. Decide in advance how disagreements are resolved and who can pause the pilot when a significant problem appears.

02

Select items across the pilot

Decide how many items can be checked and how they will be selected before work begins. Spread checks across the pilot period, shifts and people, including relevant work types. A random selection from eligible output can reduce selection bias; document how it was made and retain the list of eligible items.

Add targeted checks for high-risk exceptions, but report them separately from the random sample. A targeted sample deliberately contains more risky work and its defect rate should not be presented as the rate for all output. If every item must be checked under a requirement, sample review does not replace that requirement.

03

Interpret a small sample carefully

Hypothetical example: a pilot completes 100 eligible requests. A reviewer randomly selects 20 and finds two with a defined defect. The observed sample rate is 2 divided by 20, or 10%. It is not proof that exactly ten of the 100 requests have defects, and the small sample leaves substantial uncertainty. Review and correct the two identified items.

If the same reviewer also checks five unusually complex requests and finds one defect, report those five separately. Investigate the issue without combining the two groups into a claimed overall rate. Zero defects in a small sample would not prove the remaining work is defect-free either.

04

Act on findings and follow late issues

Compare the findings with the pre-agreed decision rules and baseline, using compatible definitions and selection methods. Examine the severity and cause of each defect, downstream delays and customer feedback alongside output. If the sample is too small to support the decision, extend observation or increase review where feasible.

Some failures appear after delivery. Define a follow-up window and track later corrections or complaints linked to the original work. Record the decision, remaining uncertainty and owner, along with the next review date.

05

Keep the selection record

Save the eligible-item list and the selection method before reviewers inspect the sample. A simple approach is to assign each eligible item a random number, sort the list and review the first items in the planned sample. Record exclusions and keep risk-targeted reviews separate, since they were intentionally selected differently.