Feedback Loop Design & Dismissal Pattern Analysis

Advanced

Implement validation, retry, and feedback loops for extraction quality · Difficulty 3/5

feedback-loopsdismissal-patternsquality-improvement

Prerequisites

Production AI systems need feedback loops that systematically improve output quality over time by tracking patterns in failures and dismissals.

The detected_pattern Field

Add a detected_pattern field to structured findings that records which code construct triggered the finding. When developers dismiss findings, you can analyze which patterns produce false positives.

Example

{
  "finding": "Potential null pointer dereference",
  "severity": "high",
  "detected_pattern": "optional_chaining_missing",
  "location": "src/api/handler.ts:45"
}

If developers consistently dismiss optional_chaining_missing findings, you know to refine the prompt criteria for that pattern.

Systematic Improvement

Track dismissals: Record which findings developers dismiss

Aggregate by pattern: Group dismissals by detected_pattern

Identify low-precision patterns: Patterns with >30% dismissal rate

Refine prompts: Add explicit criteria or few-shot examples for those patterns

Re-enable: Test updated prompts before re-enabling in production

Connection to False Positive Management

This is the data-driven version of the disable-fix-re-enable strategy. Instead of disabling entire categories, you disable specific patterns within categories based on dismissal data.

Key Takeaways

✓Add detected_pattern fields to enable systematic analysis of false positive patterns
✓Track developer dismissals aggregated by pattern to identify low-precision areas
✓Use dismissal data to drive targeted prompt refinement, not broad category changes
✓This is the data-driven version of disable-fix-re-enable

Related Concepts

Classification Consistency & False Positive Reduction

Use absolute criteria with concrete examples for each classification level

Explicit Criteria over Vague Instructions

Replace vague goals with specific, categorical criteria the model can apply deterministically