5.4 Context in Large Codebase Exploration

Task Statement 5.4 is about a reliability problem unique to long exploration sessions — and it's NOT about failures or errors. It's about a subtle DEGRADATION that creeps in as an agent works through a large codebase over many turns: it slowly stops referring to the specific things it discovered and starts giving vague, generic answers instead. Nothing crashes; the quality just quietly erodes.

Here's the telltale symptom. Early in a session, the agent gives precise answers: 'the PaymentProcessor class in billing/processor.py calls validate_card() before charging.' Hours and many turns later, ask a related question and you get: 'in a typical payment system, you'd usually validate the card first...' — generic textbook patterns instead of the specific classes and files it actually found earlier. The agent has drifted from concrete knowledge to vague generalities. It's like a researcher who, after a long day buried in documents, starts answering from general impressions rather than the specific notes they took that morning.

This is context degradation, and the crucial — and most tested — insight is its CAUSE: it is NOT a token-limit problem. The agent didn't run out of room; the quality of its attention to the specifics degraded as verbose output piled up. That distinction dictates the fix, because the obvious 'just use a bigger context window' does NOT solve it. Let's see what actually does.

If the problem is that key findings get diluted as conversation context fills with verbose output, the fix is to move those findings OUT of the conversation entirely. That's the scratchpad-file pattern, and it's the primary cure for context degradation. The agent writes its key discoveries — important classes, file paths, dependency chains — to a FILE on disk, outside the conversation context, and refers back to that file when it needs the information later.

Why does this work? Because the file is immune to degradation. The conversation context can fill with noise and the model's attention to in-context details can fade, but a fact written to scratchpad.md is exactly as precise on turn 200 as it was on turn 5 — the agent just re-reads it. It's the codebase-exploration analogue of the case-facts block from Lesson 5.1: in both, you protect the information that must stay accurate by keeping it OUTSIDE the part of context that gets diluted or summarized. The researcher who writes careful notes and consults them, rather than trusting memory after a long day, doesn't drift.

Three more techniques manage context across a long exploration, and each ties back to earlier domains. SUBAGENT DELEGATION: spawn subagents to investigate specific questions — 'find all the test files,' 'trace the refund-flow dependencies' — while the main agent keeps the high-level coordination. The key insight (from Lesson 1.3) is that the primary benefit here is context ISOLATION, not just parallelization: the subagent's verbose exploration happens in ITS context, and only a clean summary returns to the main agent, so the main context never fills with the noise that causes degradation.

SUMMARY INJECTION between phases: before moving from one exploration phase to the next, summarize the key findings and inject that summary into the next phase's starting context — carrying forward the essentials without the accumulated noise. And use /compact PROACTIVELY: rather than waiting until context is nearly full, compact during a long session to reduce the verbose discovery output before it causes drift. Compaction isn't only an emergency measure at the limit; it's a maintenance habit for long sessions.

Finally, STRUCTURED STATE MANIFESTS for crash recovery: have each agent export its state (what it explored, what it found, what phase it's in, next steps) to a known location, and on resume the coordinator loads that manifest and injects it. This makes a long, multi-phase exploration RESUMABLE after an interruption without re-exploring everything — the reliability complement to the scratchpad pattern.

The 5.4 traps almost all hinge on the same insight: context degradation is an attention-quality problem, not a capacity one, so capacity-based 'fixes' don't work.

• •Bigger window for degradation. ✗ 'Use a larger context window' to stop the drift to generic patterns. ✓ It's attention quality, not capacity — use scratchpad files.
• •Delegation = parallelization only. ✗ Thinking subagents only speed things up. ✓ Their primary benefit here is context ISOLATION, keeping the main context clean.
• •Restart without saving state. ✗ Crashing and re-exploring everything from scratch. ✓ Export structured state manifests so you can resume.
• •/compact only at the limit. ✗ Waiting until context is full to compact. ✓ Compact proactively during long sessions.

You now know context degradation and its real cause, the scratchpad-file primary fix, and the supporting techniques (subagent isolation, summary injection, proactive /compact, state manifests). The exercise has you reproduce the drift and cure it.

Scratchpad files and context isolation keep long explorations reliable. The next lesson, 5.5, is about reliability of a different kind — designing human review and confidence calibration so you know WHEN to trust automated output.