AI Memory System Design

Memory is what separates an AI that feels like a tool from one that feels like a collaborator. Without memory, every session starts from zero. This skill designs the memory layer: what to remember, how to store it, how to retrieve and use it, and when to forget.

Context

The four types of AI memory:

Type	What it stores	Scope	Example
Working memory	Current session context	One session	Current conversation history
Episodic memory	Summaries of past sessions	Per user, long-term	"Last week, user asked about RAG for legal documents"
Semantic memory	Facts and preferences learned about the user	Per user, persistent	"User is an AI PM at a Series B SaaS company"
Procedural memory	How to do tasks better for this user	Per user, persistent	"User prefers bullet-point summaries, not prose"

Working memory is automatic — handled by the context window. This skill designs the other three types.

Step 1 — Define memory requirements

MEMORY REQUIREMENTS:
What must be remembered: [List specific information]
Memory scope: [Per user / Per team / Per agent run]
Memory horizon: [N days / Indefinite / Session only]
Privacy constraints: [Retention limit, user control requirements]
Write trigger: [Every session end / On key event / On user instruction]

Step 2 — Design the memory architecture

Episodic Memory: Structured summaries of each session including topics, outcomes, and open questions. Stored in a vector database for semantic retrieval. Semantic Memory: Factual statements about the user with confidence levels. Sources: explicit user statements, inferred from sessions, user corrections. Procedural Memory: User preferences and behavioural patterns with strength scores. Sources: explicit instructions ("always do X"), inferred patterns.

Step 3 — Design the retrieval system

At session start, load:

All high-confidence semantic facts (5–15 facts)

All preferences above strength 3 (3–8 preferences)

Top 3–5 most relevant past sessions by semantic similarity

Memory volume limit: Total injected memory must not exceed N tokens. Prioritise: semantic > procedural > episodic.

Step 4 — Design the memory management UX

Users can view, edit, delete any stored memory

Users can pause memory, export memory, clear all memory

When AI stores a new fact, it discloses: "I've noted that [fact]"

No silent memory updates

Step 5 — Define privacy and retention

Episodic memory: retained for N days/months

Semantic memory: retained while account is active

On account deletion: all memory deleted within N days

User rights: access, rectification, erasure, portability

Memory data never used for training without explicit opt-in

Quality check before delivering

All four memory types addressed

Memory injection format specified

Memory volume limit defined

User control includes delete and export

Memory write disclosure required

Retention periods defined

Suggested next step: Start with semantic memory only — facts about the user. It's the simplest to implement and the highest trust signal.