Agent Memory

Phase 10 · Document 04 · Agents and Tools Prev: 03 — Planner-Executor · Up: Phase 10 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

An agent's context grows every step — each tool call and result is appended (01) — so a long-running agent runs straight into the finite context window: it gets slower, pricier, and eventually loses early information or hits the limit. Agent memory is how the harness manages what the (stateless) model sees across a long task and across sessions: what to keep in context now, what to summarize, what to push to an external store and retrieve later. Get it right and an agent stays coherent and affordable over hundreds of steps; get it wrong and it forgets the goal, repeats work, blows the budget, or OOMs the window. This doc is the agent-specific application of the cross-cutting memory model you already studied (agent-memory reference) — and it's where context engineering meets the agent loop.

2. Core Concept

Plain-English primer: the model is stateless — memory is managed context

The anchoring law again (what-happens §0, agent-memory ref): the model remembers nothing; every form of "memory" is the harness deciding what text to put back into the context window each step. For an agent specifically, that means managing a context that grows with every tool call against a fixed window.

The standard taxonomy (mapped to mechanisms — see the memory reference for the cross-tool view):

For most agents the live problem is working + episodic memory under a growing window; semantic/procedural are the long-term store you retrieve from.

The core challenge: a context that outgrows the window

A 50-step agent accumulates 50 tool requests + 50 (often large) results. That:

• blows the window (hard limit → truncation/OOM),
• costs more every step (you re-send the whole growing context each call, what-happens §1.D/§10),
• degrades quality (lost-in-the-middle; the goal buried under tool noise).

So the agent harness needs the context-reduction toolkit (what-happens §3), applied to the agent loop.

The agent memory toolkit (manage the growing context)

• Tool-result trimming/elision — old, already-acted-on tool outputs are the biggest, least-durable thing; shorten or stub them once used (01, Phase 9.07). Usually the first and cheapest win.
• Compaction/summarization — replace a long stretch of past steps with a model-written summary of decisions, findings, and open tasks; continue from the summary (what-happens §3.1). Lossy — restate critical constraints after.
• A scratchpad / external state — keep durable task state (plan, findings, TODOs) in a structured store or a file the agent reads/writes rather than relying on the chat history. (Anthropic's "memory tool" / file-based agent memory works this way: the agent writes notes to disk and reads them back, so memory survives compaction.)
• Retrieval (semantic/long-term) — push facts to an external store and retrieve only the relevant ones per step (RAG over memories, Phase 9) — the always-loaded-index + retrieved-bodies pattern (agent-memory ref).
• Subagents (context isolation) — push a bulky sub-task into a separate context window and return only a summary, keeping the main context clean (03, what-happens §3.6).

Working vs long-term: two delivery strategies

Same split as the memory reference: always-loaded (in the context now — working memory, the system prompt, a small scratchpad) vs retrieved-on-demand (an external store you query — semantic/episodic long-term memory). Keep always-loaded small and high-signal; push the long tail to a retrieved store. The mistake is dumping everything into the always-loaded context (cost + lost-in-the-middle) or, conversely, relying on retrieval for the active goal (which must stay in working memory).

Cross-session persistence

Memory across sessions is the harness persisting state — the transcript (episodic) and any written notes/facts (semantic/procedural) — and reloading/retrieving them next time (what-happens §8 resume, agent-memory ref). The model still remembers nothing; continuity is replayed/retrieved text.

3. Mental Model

   model is STATELESS → "memory" = harness choosing what text to put in the window each step
   PROBLEM: agent context GROWS every tool call → window limit + cost↑ + lost-in-the-middle

   TYPES:  WORKING (window: goal+recent steps)  EPISODIC (transcript/summaries)
           SEMANTIC (facts, retrieved)           PROCEDURAL (skills, retrieved/loaded)

   TOOLKIT (manage the growing context — what-happens §3):
     trim/elide old tool results  →  COMPACT (summarize past steps)  →  SCRATCHPAD/files (durable state)
     →  RETRIEVE long-term memories (RAG)  →  SUBAGENTS (isolate bulky sub-tasks)

   DELIVERY: ALWAYS-LOADED (small, high-signal: goal/scratchpad) vs RETRIEVED (long tail) [agent-memory ref]
   cross-session = harness persists transcript + notes, reloads/retrieves (resume) [what-happens §8]

Mnemonic: the model is stateless; agent memory = managing a growing context against a fixed window — trim results, compact, use a scratchpad/files, retrieve long-term, isolate via subagents. Keep working memory (the goal) always-loaded and small; retrieve the rest.

4. Hitchhiker's Guide

What to look for first: does context grow unbounded as the agent runs? If yes, you need a reduction strategy (trim results + compaction) before the window/cost bites — plus keeping the goal/plan reliably in working memory.

What to ignore at first: elaborate long-term memory systems (Letta/mem0) and multi-store architectures. Start with tool-result trimming + compaction + a simple scratchpad; add retrieval/long-term memory when the task genuinely needs cross-session facts.

What misleads beginners:

• Treating chat history as reliable memory. It's lossy under compaction and bounded by the window — keep durable task state in a scratchpad/file the agent re-reads (what-happens §3.1).
• Never trimming tool results. Old large outputs dominate the window and cost — elide them once used (01).
• Dumping everything always-loaded. Cost + lost-in-the-middle; push the long tail to retrieval (agent-memory ref).
• Compaction losing the goal/constraints. After a compaction, restate the goal and hard constraints (they may have been summarized away).
• Polluting main context with exploration. Use subagents to isolate bulky sub-tasks (03).

How experts reason: they keep working memory small and goal-focused, trim/elide spent tool results, compact at boundaries (restating constraints), keep durable task state in a scratchpad/file, retrieve long-term facts on demand, and isolate heavy sub-work in subagents. They treat the context window as a scarce, actively-managed resource, not an append-only log.

What matters in production: context size/cost per step trajectory, whether the agent keeps the goal/constraints across compactions, cross-session continuity correctness (resume), and that long-term memory retrieval is relevant (not noise).

How to debug/verify: trace context size per step (08); if it grows unbounded → add trimming/compaction; if the agent "forgot" the goal/a constraint → it was compacted away (use a persistent scratchpad + restate); if it repeats work → episodic memory/scratchpad missing.

Questions to ask: does context grow unbounded? are old tool results trimmed? is durable task state in a scratchpad/file (compaction-safe)? is long-term memory retrieved (not all loaded)? does resume restore the right state?

What silently gets expensive/unreliable: unbounded growing context (cost + window limit), compaction dropping the goal/constraints, exploration polluting main context, and over-stuffed always-loaded memory (lost-in-the-middle).

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• The model is stateless — agent "memory" is the harness managing what text is in the window each step (what-happens §0).
• Agent context grows every tool call → hits the window, raises cost, and degrades quality (lost-in-the-middle) (01).
• Four memory types: working (window), episodic (transcript/summaries), semantic (facts, retrieved), procedural (skills) — agent-memory ref.
• Manage growth with the reduction toolkit: trim/elide old tool results → compact → scratchpad/files → retrieve → subagents (what-happens §3).
• Keep durable task state in a scratchpad/file so it survives compaction (the chat history is lossy).
• Working memory (the goal) stays always-loaded and small; the long tail is retrieved on demand.
• Restate the goal/constraints after compaction — they may have been summarized away.
• Cross-session memory is harness persistence (transcript + notes) reloaded/retrieved on resume (what-happens §8).

9. Observations from Real Systems

• Claude Code / coding agents rely on tool-result trimming + compaction + a persistent scratchpad (and CLAUDE.md) to run long sessions affordably (what-happens §3/§4, agent-memory ref).
• Anthropic's memory tool / context-management lets agents write notes to files and compact, so memory survives the window (context engineering).
• Letta (MemGPT) and mem0 formalize long-term memory (self-editing blocks; extract→store→retrieve) for agents that need cross-session facts.
• Deep-research agents isolate bulky exploration in subagents and keep only summaries in the lead agent's context (03, 07).
• The classic failure: an agent that "forgot" its goal mid-task — the goal was compacted away and not held in a scratchpad (what-happens §3.1).

10. Common Misconceptions

11. Engineering Decision Framework

MANAGE AGENT MEMORY:
 1. WORKING: keep the GOAL + plan + recent steps in context, small and high-signal. Restate goal after compaction.
 2. TRIM: elide/stub old, already-acted-on tool results (biggest cheap win). [01]
 3. COMPACT: summarize past steps at boundaries when context grows large. [what-happens §3.1]
 4. SCRATCHPAD: persist durable task state (plan/findings/TODOs) to a file/store the agent R/W — survives compaction.
 5. LONG-TERM: push facts/skills to an external store; RETRIEVE only the relevant per step (RAG). [Phase 9]
 6. ISOLATE: push bulky sub-tasks to SUBAGENTS (own context, return summary). [03]
 7. CROSS-SESSION: persist transcript + notes; reload/retrieve on resume. [what-happens §8]
 8. INSTRUMENT context size/cost per step; trim/compact before it bites. [08]

12. Hands-On Lab

Goal

Take a long-running agent and add a memory strategy (trim + compact + scratchpad) — proving it stays coherent and affordable while a naïve append-only agent degrades.

Prerequisites

• The bounded agent from 00/01; a multi-step task long enough to grow context (e.g., explore 15+ items and summarize).

Steps

1. Baseline (append-only): run the task appending every tool result; plot context tokens and cost per step. Observe growth, rising cost, and (on a long enough task) degraded/forgotten goal or window limit.
2. Trim tool results: elide/stub tool outputs once the agent has acted on them (01); re-measure context/cost — expect a big drop.
3. Compaction: when context exceeds a threshold, summarize past steps into a compact summary and continue; restate the goal/constraints in the summary (what-happens §3.1). Verify the agent still knows its goal after.
4. Scratchpad: add a file/dict the agent reads/writes for durable state (plan, findings, TODOs); show the goal/plan survive even after aggressive compaction (because it's in the scratchpad, not just history).
5. (Long-term) retrieval: store findings as memories; on a follow-up task, retrieve relevant ones instead of replaying everything (Phase 9).
6. Compare: tabulate context tokens/cost-per-step and task success for append-only vs managed; show managed stays flat-ish and coherent.

Expected output

A context-tokens/cost-per-step plot for append-only vs managed memory, plus evidence the managed agent retains its goal across compaction (via scratchpad) — demonstrating active context management.

Debugging tips

• Goal lost after compaction → not in the scratchpad / not restated in the summary.
• Cost still grows → tool results not actually trimmed, or compaction threshold too high.

Extension task

Add subagents for a bulky sub-task and show the main context stays small (03); add long-term retrieval across two sessions.

Production extension

Wire a persistent scratchpad/memory store + retrieval, compaction at boundaries, and context-size/cost-per-step metrics into observability (08); handle resume (what-happens §8).

What to measure

Context tokens & cost per step (append-only vs managed); goal-retention across compaction; task success; long-term retrieval relevance.

Deliverables

• A managed-memory agent (trim + compact + scratchpad).
• A context/cost-per-step comparison vs append-only.
• A goal-retention-across-compaction demonstration (scratchpad).

13. Verification Questions

Basic

1. Why does an agent's context grow, and what three problems does that cause?
2. Name the four memory types and their mechanisms.
3. Why isn't chat history reliable durable memory?

Applied 4. Which memory tools address a context that's hitting the window? In what order? 5. Why keep working memory small and retrieve the long tail?

Debugging 6. An agent forgot its goal mid-task. Cause and fix. 7. Cost rises every step on a long task. What memory levers help?

System design 8. Design memory for a long-running research agent: working/episodic/semantic + scratchpad + subagents.

Startup / product 9. How does agent memory management affect cost-per-task and reliability for a long-running-agent product?

14. Takeaways

1. The model is stateless — agent memory is the harness managing a growing context against a fixed window.
2. Four types: working (window), episodic (transcript/summaries), semantic & procedural (retrieved long-term).
3. Manage growth: trim spent tool results → compact → scratchpad/files → retrieve → subagents.
4. Keep the goal in working memory (and a scratchpad); restate constraints after compaction; retrieve the long tail.
5. Cross-session memory is harness persistence reloaded/retrieved on resume — treat the window as a scarce, managed resource.

15. Artifact Checklist

• A managed-memory agent (trim + compact + scratchpad).
• A context/cost-per-step comparison vs append-only.
• A goal-retention-across-compaction demo (persistent scratchpad).
• A long-term retrieval across sessions (semantic memory).
• Context-size/cost-per-step metrics wired toward 08.

Up: Phase 10 Index · Next: 05 — Sandbox and Approval