Skip to content

Part 3, Memory โ€‹

Knowledge-share outline โ€‹

A short talk (about 6โ€“8 slides):

  1. Why memory matters: agents without memory ask users to repeat themselves; agents with bad memory leak data or behave unsafely.
  2. Short, medium, and long context: the three memory horizons and what lives in each.
  3. Four kinds of memory: conversation history, working memory, user-preference memory, domain knowledge.
  4. Remember / forget / retrieve / ignore: the four-way choice every memory decision reduces to.
  5. Memory vs. retrieval: pulling a record from a database is not the same as remembering something about a user.
  6. Safety: what must never enter memory; what must be confirmed before storage.
  7. Where middleware fits: one place to enforce memory rules, so nothing slips through by accident.
  8. What you will build in the lab: a session-scoped order-memory middleware.

At a glance โ€‹

Start stateEnd stateFallback baseline
A context map from Part 2A working order-memory middleware that automatically stores the active order reference and makes it available to the agent on the next turnbaseline/03-1-order-memory.py, baseline/03-2-agent-memory.py

Learning goals โ€‹

  • Distinguish short-term conversation state (what the agent is tracking right now) from long-term memory (what persists across sessions).
  • Name the four kinds of memory and describe where each one lives.
  • Apply the remember / forget / retrieve / ignore choice to at least two concrete examples from the Dunder Mifflin scenario.
  • Build a middleware layer that automatically stores the current order reference so the agent can answer follow-up questions without being told the order ID again.

Why it matters โ€‹

Darryl asks about package DM-1037 in one message, then asks "is it fragile?" in the next. Without memory, the agent has no idea what "it" refers to and either fails or asks Darryl to repeat himself. With good session memory, the agent tracks the active order across turns invisibly. With bad memory, it could retain PII, surface a different user's package data, or never forget something that should expire. This part is about getting the memory decisions right before any state is stored.

Concept: a memory model for agents โ€‹

There is no single "correct" memory architecture, only design choices and trade-offs. A useful starting model separates memory by how long it lives and what it is for.

Short, medium, and long context โ€‹

HorizonLives forPackage-manager example
ShortThe current turn"it" in "is it fragile?" refers to the order just mentioned
MediumThe current conversation / sessionthe order or route under discussion, so the user need not repeat it
LongAcross sessionsStanley's saved display preference

Four kinds of memory โ€‹

Name them so your requirements can be precise:

  • Conversation history: the running transcript of this chat (short / medium).
  • Working memory: the few facts the agent is actively using to answer now (short).
  • User-preference memory: durable choices a user made, scoped to that user (long).
  • Domain knowledge: facts about the package system itself (routes, label formats); shared, not personal.

Retrieval vs. memory

Pulling a record from the package database to answer a question is retrieval of domain knowledge. It is not the same as remembering something about a user. Keep the two separate in your implementation and your specification.

The four-way choice โ€‹

For each candidate fact, decide:

ChoiceMeaningWhen to use
RememberStore it, with a defined scope and expiryUser-level preferences; session context that would be painful to repeat
ForgetDiscard at a defined pointAnything session-scoped that should not outlast the conversation
RetrieveLook it up on demand from domain storagePackage data, route data (facts about the world, not about the user)
IgnoreNever store it at allPII, credentials, anything that must not persist

What you will build โ€‹

A small Python middleware module that intercepts agent messages, extracts any package or order reference (for example, DM-1037), and stores it in a session-scoped memory dict. On the next turn, the stored reference is injected into the agent's context so the agent can answer follow-up questions without being told the order ID again.

Requirement examples: good vs. bad memory decisions โ€‹

Bad (ambiguous or unsafe)Good (specific, testable, bounded)
"Remember the current order.""Within a conversation, store the most-recently-mentioned order ID so follow-up questions resolve without repeating it; discard at session end."
"Keep useful context.""Track the active route in session memory so Darryl can ask 'is it on schedule?' without re-stating the route number."
"Store everything for support.""Never store PII, credentials, or safety-status overrides."
"Cache user answers.""Store a user's display preference, scoped to that user, until they change or clear it."

Lab steps โ€‹

Running commands

All terminal commands below must be executed in a terminal. All Python commands assume you are in your dm-ws/ root with the virtual environment active. If you are not sure how to open a terminal in VS Code, see VS Code tips.

Step 1: Understand the memory structure you will implement โ€‹

You are extending the agent.py you built in Part 1. You will add two new files alongside it in the dm-ws/ root: order_memory.py (the memory module) and agent_memory.py (the agent that uses it).

The memory model uses a session store (a Python dict keyed by session ID). Each entry holds:

python
{
    "active_order_id": "DM-1037",  # the most-recent order mentioned in this session
}

This is medium-horizon (session-scoped) working memory. It is discarded when the session ends.


Step 2: Create order_memory.py โ€‹

In VS Code, create a new file order_memory.py in the dm-ws/ root folder. Add this content:

python
"""Session-scoped order memory.

Stores the most-recently-mentioned Dunder Mifflin order ID within a
conversation so follow-up questions resolve without repetition.
"""
import re
from typing import Any

# In-process session store: maps session_id -> context dict.
# Replace with a Redis or DB-backed store for production.
_store: dict[str, dict[str, Any]] = {}

# Matches order IDs like DM-1037 or dm-1037
_ORDER_PATTERN = re.compile(r"\bDM-\d+\b", re.IGNORECASE)


def extract_and_store(session_id: str, text: str) -> None:
    """Extract an order ID from *text* and store it for *session_id*."""
    match = _ORDER_PATTERN.search(text)
    if match:
        context = _store.setdefault(session_id, {})
        context["active_order_id"] = match.group(0).upper()


def get_context(session_id: str) -> dict[str, Any]:
    """Return the stored memory context for *session_id*, or an empty dict."""
    return dict(_store.get(session_id, {}))


def clear(session_id: str) -> None:
    """Discard the memory context for *session_id*."""
    _store.pop(session_id, None)

Expected output: the file saves without errors.


Step 3: Create agent_memory.py โ€‹

Rather than editing your Part 1 file, create a fresh agent_memory.py alongside it. The memory-relevant changes compared to agent.py:

  • uuid and from order_memory import ... are imported at the top.
  • ask() now accepts a session_id, calls extract_and_store on the way in, and appends the memory hint to the user message.
  • The __main__ block is a multi-turn Read-Evaluate-Print-Loop (REPL) instead of a single --ask invocation, so the session (and its stored order ID) persists across turns.

(It also loads data/routes.json alongside the packages and combines both into the system prompt โ€” a small context upgrade, not part of the memory pattern itself.)

In VS Code, create a new file agent_memory.py in the dm-ws/ root folder and paste this complete file:

python
from __future__ import annotations

import json
import os
import uuid
from pathlib import Path

from dotenv import load_dotenv
from openai import AzureOpenAI
from order_memory import extract_and_store, get_context

load_dotenv()

# -- Load data ----------------------------------------------------------------
_root = Path(__file__).parent
PACKAGES = json.loads((_root / "data/packages.json").read_text())
ROUTES   = json.loads((_root / "data/routes.json").read_text())

# -- Azure OpenAI client ------------------------------------------------------
client = AzureOpenAI(
    api_key=os.environ["AZURE_OPENAI_KEY"],
    azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
    api_version="2024-05-01-preview",
)
MODEL = os.environ.get("AZURE_OPENAI_DEPLOYMENT", "gpt-5.5")

SYSTEM = f"""You are the Dunder Mifflin package manager assistant.
You help warehouse staff and managers look up package status, routes, and conditions.

Current package data:
{json.dumps(PACKAGES, indent=2)}

Current route data:
{json.dumps(ROUTES, indent=2)}

Answer questions based only on the data above. If something is not in the data, say so.
"""


def ask(question: str, session_id: str = "") -> str:
    # On the way in: extract any order reference from the user's message
    extract_and_store(session_id, question)

    memory_ctx = get_context(session_id)
    active_order = memory_ctx.get("active_order_id")
    context_hint = (
        f"\n[Memory] Active order this session: {active_order}" if active_order else ""
    )

    if context_hint:
        print(f"  {context_hint.strip()}")

    response = client.chat.completions.create(
        model=MODEL,
        messages=[
            {"role": "system", "content": SYSTEM},
            {"role": "user",   "content": question + context_hint},
        ],
    )
    return response.choices[0].message.content or ""


if __name__ == "__main__":
    session_id = str(uuid.uuid4())
    print("Dunder Mifflin package agent โ€” type 'quit' to exit.\n")
    while True:
        question = input("You: ").strip()
        if question.lower() in ("quit", "exit", "q"):
            break
        print(f"Agent: {ask(question, session_id)}\n")

Where to put the extract call

extract_and_store belongs on the way in (the user's message), not on the way out. That way, the current turn already has access to the order it just mentioned.

Why does this work without sending conversation history?

A typical chatbot sends every previous message back to the model on each turn; the model can then see the full exchange and resolve references like "it" by re-reading the history. This agent does not do that: the messages list contains only the system prompt and the current user message.

It still works because the memory middleware extracts the relevant fact (the order ID) and injects it explicitly into the current message as a hint. The model does not need to re-read the conversation; it receives the answer directly in the prompt.

This is a deliberate design choice. It is lightweight and predictable, but it is also limited: if Darryl asked "why did you say that two turns ago?", the agent could not answer. Full conversation history handles that naturally; the memory middleware does not. Keep that trade-off in mind when you decide what goes in memory and what belongs in history.

Expected output: the file saves without errors.


Step 4: Test the middleware โ€‹

Run the agent from the dm-ws/ folder:

bash
python agent_memory.py

Then try these three turns in order.

Turn 1 (introduce the order):

What is the status of order DM-1037?

Expected: the agent looks up DM-1037 and returns its status. Your middleware stores "DM-1037" in the session store.

Turn 2 (follow-up without repeating the ID):

Is it fragile?

Expected: the context hint [Memory] Active order this session: DM-1037 is appended to the message before it reaches the model, so the agent resolves "it" to DM-1037 and answers correctly.

Turn 3 (verify isolation, new session):

Type quit to end the current session. Start the agent again (python agent_memory.py) and immediately ask:

Is it fragile?

Expected: no stored order; the agent asks for clarification rather than guessing. This confirms session memory is correctly scoped and does not persist between runs.


Step 5: Write unit tests โ€‹

Create test_memory.py in the dm-ws/ root folder:

python
from order_memory import extract_and_store, get_context, clear


def test_stores_order_id():
    extract_and_store("sess-1", "What about order DM-1037?")
    assert get_context("sess-1")["active_order_id"] == "DM-1037"


def test_clears_on_session_end():
    extract_and_store("sess-2", "order DM-0001")
    clear("sess-2")
    assert get_context("sess-2") == {}


def test_no_order_leaves_context_empty():
    extract_and_store("sess-3", "Hello, is the system online?")
    assert "active_order_id" not in get_context("sess-3")

Run them from the dm-ws/ root folder:

bash
python -m pytest test_memory.py -v

Expected output: three tests collected, three passed.


Step 6: Reflect on what you built โ€‹

Answer these for your notes (you will reuse this template in Part 4 to write it as a spec requirement):

  1. What is stored? The most recently mentioned order ID.
  2. Why? So follow-up questions resolve without repetition.
  3. For whom? Scoped to the session; not shared across users.
  4. For how long? Until the process exits (or clear() is called explicitly).

Checkpoint โ€‹

  • order_memory.py exists.
  • agent_memory.py exists.
  • All three unit tests pass.
  • A follow-up turn without an order ID resolves correctly using the stored reference.
  • A new run of the agent does not see a previous run's stored order.
  • You can state what is stored, why, for whom, and for how long for at least two memory decisions.

See How checkpoints work. If you're not there yet, switch to the baseline below.

Fallback baseline โ€‹

Try it yourself first. If you run out of time, copy the provided baseline files so you start Part 4 in the right place:

powershell
copy baseline\03-1-order-memory.py order_memory.py
copy baseline\03-1-order-memory.py 03-1-order-memory.py
copy baseline\03-2-agent-memory.py agent_memory.py
bash
cp baseline/03-1-order-memory.py order_memory.py
cp baseline/03-1-order-memory.py 03-1-order-memory.py
cp baseline/03-2-agent-memory.py agent_memory.py

baseline/03-1-order-memory.py is the completed memory module and baseline/03-2-agent-memory.py is the agent that uses it. (The second copy keeps the module under its original filename too โ€” the baseline agent loads it by that name, while your own agent_memory.py from the lab imports order_memory.)

Troubleshooting โ€‹

  • "The agent still asks for the order ID on turn 2": Confirm you replaced the original user_message argument to the model call with question + context_hint. The hint must actually reach the model.
  • "The wrong order ID is stored": re.IGNORECASE normalizes case; .upper() ensures consistent storage. Confirm both are present in the regex.
  • ModuleNotFoundError: No module named 'order_memory': Make sure you are running python agent_memory.py (and pytest) from the dm-ws/ root folder, where order_memory.py lives alongside it.
  • "Memory leaks between tests": Each test uses a unique session ID (sess-1, sess-2, sess-3). If you reuse IDs across test runs, call clear(session_id) in teardown. The in-process dict does not expire automatically.
  • "I can't tell memory from retrieval": If it is a fact about the package system (status, route), it is domain knowledge you retrieve from a file or database; if it is a fact about what the user is currently discussing, it is working memory you store.

Optional stretch tasks โ€‹

Intermediate: Extend the middleware to also extract and store a route number (for example, R-2) when one is mentioned. Add unit tests for it.

Intermediate: Create new middleware to extract and store route and order numbers when these are mentioned, but persist it to disk. Add unit tests for it and make sure it is persisted between sessions.

Advanced: Replace the in-process dict with a Redis-backed store using redis-py. Verify that two separate Python processes share session memory through Redis.

Expert: Add a confirmation gate: before storing anything beyond the session reference (for example, a user preference), surface what you are about to store and ask for confirmation. Log the decision. This previews the human-in-the-loop pattern from Part 10.

Apply it: De LeesLantaarn โ€‹

Apply it to your own solution

For De LeesLantaarn, what belongs in session memory for a member who is browsing right now (the title they are viewing, an in-progress reservation) versus longer-term memory that shapes recommendations (reading history, favourite genres)? Just as important: what should the agent forget at the end of a session so a shared library terminal never leaks one member's activity to the next?

What we will not cover โ€‹

  • Persistent memory stores (database-, or file-backed): the in-process dict keeps the focus on the concept. Persisting raises all kinds of new challenges. Check out the stretch task if you are curious.
  • Long-term user-preference memory (storage and retrieval): you will specify that in Part 4.
  • Authentication or access control on the memory store.
  • Memory expiry policies beyond "clear on session end".
  • A production-ready memory architecture.

Previous: Part 2, Context engineering ยท Next: Part 4, Writing useful specs & personas