5.5 Human Review & Confidence Calibration

Task Statement 5.5 is about a decision every production AI system eventually faces: how much human review do you need, and where should it go? Reduce human review too aggressively and errors slip into production; keep too much and you've automated nothing. The skill is designing review workflows that put scarce human attention exactly where it's needed — and the lesson opens with a trap that makes systems look far safer than they are.

That trap is the AGGREGATE-METRICS trap. Your extraction system reports '97% overall accuracy,' and that sounds great — surely safe to automate most of it. But the aggregate HIDES the distribution. Break that 97% down by document type and you might find: invoices 99.5%, handwritten forms 60%, scanned PDFs 72%, international documents 45%. The headline number is a weighted average dominated by the easy, common cases, while whole categories are failing badly. It's like saying a hospital has a '97% survival rate' while one ward is at 45% — the average conceals the disaster. An aggregate metric can make a system look production-ready when specific segments are nowhere near it.

So this lesson is about looking BENEATH the aggregate — validating by segment, sampling intelligently, calibrating confidence, and routing review where uncertainty is highest. Let's take them in order.

The fix for the aggregate trap is to validate accuracy by DOCUMENT TYPE and FIELD SEGMENT before trusting automation. Don't ask 'is the system accurate?'; ask 'is it accurate on invoices? on handwritten forms? on the date field? on the total field?' Only when each segment meets your bar is it safe to reduce human review for that segment — and the failing segments (handwritten at 60%) keep their human review. Validation-before-automation means proving accuracy per segment first, not flipping the whole system to automatic on the strength of an average.

How do you MEASURE ongoing accuracy without reviewing everything? Stratified random sampling. Divide the population into strata (by document type, say) and sample from each — and crucially, include the HIGH-CONFIDENCE items in your sampling, not just the low-confidence ones. Why sample the items the model is sure about? Because that's where NOVEL error patterns hide: a new failure mode might produce confidently-wrong outputs that you'd never catch if you only review low-confidence cases. Sampling each stratum, including the confident ones, surfaces problems an only-review-the-uncertain approach would miss.

You can have the model output a CONFIDENCE score per field, then route review by it — but only after addressing the calibration problem you've now met several times (4.1, 4.6). Raw self-reported confidence is POORLY CALIBRATED: a reported 0.95 might correspond to 88% real accuracy in one context and 99% in another. So you can't read the raw number as a probability.

The fix is to CALIBRATE field-level confidence against a LABELED VALIDATION SET. You take items where you KNOW the right answer, see what the model's confidence levels actually correspond to in reality, and build a calibration curve mapping reported confidence to true accuracy. Now '0.9 calibrated' actually means ~90% accurate, and you can set routing thresholds that mean something. Calibration turns a vibe into a measured probability.

With calibrated confidence, route to concentrate scarce human attention: send the HIGHEST-uncertainty items — and ambiguous or contradictory source documents — to human review FIRST, as a dynamic priority queue, rather than distributing reviewers evenly across everything. The lowest-uncertainty, well-calibrated-as-accurate items can be auto-accepted. The principle: spend human review where the risk is, and a calibrated confidence score is what tells you where that is.

The 5.5 traps test the aggregate trap, segment validation, calibration, and review prioritization. The signature scenario: '97% accuracy, so let's automate everything above 95% confidence.'

• •Trusting the aggregate. ✗ Automating because overall accuracy is 97%. ✓ Validate by document type and field — segments may be failing (handwritten 60%, international 45%).
• •Raw confidence for automation. ✗ Auto-accepting everything above 0.95 self-reported confidence. ✓ Calibrate against labeled data first — raw scores are poorly calibrated.
• •Sampling only low-confidence items. ✗ Reviewing only what the model is unsure about. ✓ Stratified sampling that includes high-confidence items catches novel errors.
• •Even reviewer distribution. ✗ Spreading reviewers evenly across all items. ✓ Route highest-uncertainty / ambiguous items first.

You now know the aggregate trap, segment validation, stratified sampling, confidence calibration, and uncertainty-first routing. The exercise has you build a review workflow that's honest about where the system actually fails.

Human review and calibration tell you when to trust automated output. The final lesson, 5.6, closes Domain 5 — and the course — with provenance: preserving where information came from when synthesizing across many sources.