4.4 Validation & Retry Loops

Lesson 4.3 ended on a crucial gap: tool_use guarantees your output is well-FORMED, but not that it's CORRECT. A schema-valid extraction can still have line items that don't sum, values in the wrong field, or fabricated data. Task Statement 4.4 is about closing that gap — validating the output and, when it's wrong, retrying in a way that actually fixes it.

The naive approach is a plain retry: it failed, so run it again and hope. But hope isn't a strategy — a plain retry usually reproduces the SAME mistake, because the model has no idea what went wrong. Picture a student who got a math problem wrong; handing the identical problem back unchanged just gets the same wrong answer. But hand it back with 'your total of £450 doesn't match the line items, which sum to £500' and now they know exactly what to fix. That targeted feedback is the heart of this lesson.

The technique is retry-WITH-error-feedback: when validation catches a problem, you send the model back (1) the original document, (2) its failed extraction, and (3) the SPECIFIC validation error. With that, the model can self-correct precisely instead of guessing again. But there's an important limit — some errors retries can never fix, no matter how good the feedback — and knowing which is which is the most-tested idea here.

This is the heart of 4.4. Retry-with-feedback is powerful, but it has a hard boundary, and the exam tests whether you know it. The deciding question: is the information actually THERE to get right?

Retries CAN fix problems where the right answer is derivable from what's in front of the model: format mismatches (a date in the wrong format), structural errors (fields nested wrong), misplaced values (the right number in the wrong field), and math errors (a sum that's off). In all of these, the correct answer is recoverable — the feedback just points the model at it. Retries CANNOT fix a fundamentally different problem: information that is simply ABSENT from the source. If the document never states the customer's VAT number, no amount of retrying or feedback will conjure it — it isn't there. The same applies to anything requiring EXTERNAL knowledge or documents the model wasn't given.

So when validation fails, first ask: is this a recoverable error (retry with feedback) or missing information (don't retry — flag for a human, or return null)? Retrying for absent data is a waste that, worse, can pressure the model into fabricating the missing value. Match the response to the cause.

You can make validation easier by designing the schema to SURFACE its own errors. Instead of computing the total separately and comparing, have the extraction return BOTH a calculated_total (summed from the line items) AND the stated_total (the figure printed on the document). Now a discrepancy is visible right in the output — your validator just compares two fields, and the model itself is prompted to notice when they disagree.

The same self-correction idea extends further. Add a conflict_detected boolean for sources that contradict each other, so inconsistency is flagged rather than silently resolved. Add a detected_pattern field that records WHICH code construct triggered a finding — so when developers dismiss certain findings, you can analyze the dismissal patterns and refine your criteria (tying back to the false-positive work in 4.1). Schema fields can do more than hold data; they can build in the checks that catch semantic errors.

And recall the distinction from 4.3: schema SYNTAX errors are already eliminated by tool_use, so they're not what validation-retry is for. Validation-retry exists to catch SEMANTIC errors — the sums, misplacements, and conflicts that well-formed JSON can still contain. The two lessons are partners: 4.3 guarantees the form, 4.4 verifies the substance.

The 4.4 traps test the retry-able-vs-not distinction and the difference between naive and feedback-driven retries. The signature question gives two documents — one with a recoverable error, one with missing info — and asks how to handle each.

• •Naive retry. ✗ Re-running the same prompt and hoping. ✓ Retry WITH the document, the failed output, and the specific error.
• •Retrying for absent info. ✗ Retrying when the needed value isn't in the source. ✓ Flag for a human or return null — retrying can't supply missing data and may cause fabrication.
• •Treating syntax and semantic errors the same. ✗ Using retry loops for JSON syntax. ✓ tool_use already eliminates syntax (4.3); retry-validate is for semantic errors.
• •Not surfacing the error. ✗ Computing totals entirely outside the model. ✓ Have the schema return calculated vs stated values so discrepancies are visible.

You now know retry-with-feedback, the recoverable-vs-absent boundary, and schema-based self-correction. The exercise has you build a validation-retry loop and prove the boundary by feeding it both kinds of failure.

Validation-retry verifies one document at a time. The next lesson, 4.5, scales up to VOLUME — the Message Batches API for processing thousands of documents cost-effectively, and when batch is (and isn't) the right choice.