12 March 2026 · 6 min read
Sampling bias hides in “representative” packs
Ask a review team how they chose their sample and you will often hear “we picked a representative mix.” That phrase comforts stakeholders. It rarely describes a method.
Representative of what? Business units that answer email quickly? Controls with tidy folders? Periods without system outages? Bias creeps in when convenience masquerades as coverage.
Write the exclusion, not just the percentage
A useful sample sheet lists what you deliberately left out and why. “Excluded weekend settlements because the overnight batch was under change freeze” tells a second reader more than “15% of transactions.”
In the Evidence Control Workshop we practise risk-weighted picks and force cohorts to defend exclusions aloud. The discomfort is the point — silent exclusions are how packs look complete while skipping the awkward corners.
Friendly controls are not neutral
Reviewers return to controls they already understand. Over a year, the same three owners appear in every pack. Rotate deliberately. Track which owners have not been sampled in two cycles. That tracker is dull; it is also one of the highest-leverage artefacts a compliance evidence review practice can keep.
What this is not
This is not a licence to sample so lightly that you cannot support a conclusion. If your regulator or internal methodology requires full coverage for a control type, say so and do it. Bias control is about honesty in the space where judgement is allowed.