Stanford's Memorilla Squeezes LLM Memory Into 16 Tokens

Researchers behind a project hosted at memory.stanford.edu have released Memorilla, a memory system that boils an entire collection of documents down to 16 "memory tokens" for a frozen large language model. According to the paper posted on arXiv on 15 October 2026, the approach beats the baselines on five of eight benchmarks, including Mem0 and MemoRAG, while feeding the model far fewer tokens than full-context prompting.

What is Memorilla and what problem does it solve?

A frozen LLM is a model whose weights are no longer updated after training. Its only way to use new knowledge, such as past user conversations, long documents or an agent's earlier actions, is to receive that information in its context window. According to the arXiv abstract, gathering and combining the information a query needs while keeping that context short remains hard for existing memory systems.

Memorilla attacks this by separating remembering from recalling, as the project page explains. A frozen encoder stores each incoming document, such as a chat or a book chapter, as an embedding, a numerical vector. For each question, learned memory slots read the question and the entire store, then become memory tokens that the frozen LLM reads alongside the question. Only this memory module is trained, in stages that move from paraphrasing Wikipedia to general question answering and summarization, then to the benchmarks' training splits, according to the project page.

The project page lists four design requirements:

• collection-wide access: each query reads the whole collection, not a few pre-selected passages;
• query-dependent recall: what memory returns depends on the question;
• a fixed token budget: the model receives the same few memory tokens however large the collection grows;
• reusable embeddings: each document is encoded once, so the store grows without re-encoding the past.

How well does it perform against full context and RAG?

Every method in the comparison answers through the same frozen Qwen3-8B, according to the project page. Memorilla obtains the highest mean score, 49.68, and leads on five of eight benchmarks, while giving the model on average 394 times fewer document tokens than full context. The page stresses that this mean combines different metrics.

The headline result concerns FactKG, a fact-verification benchmark. Memorilla reaches 84.33% accuracy against 80.43% for full-context inference, using 108 times fewer tokens to represent the source documents, according to the arXiv abstract. The project page specifies that the score covers 9,041 test claims and that, as claims combine more facts, the RAG-50 baseline falls from 78.7% to 51.8% while Memorilla stays above 80%.

Other figures reported on the project page:

• PersonaMem-v2, a personalization test: 44.60% short-answer accuracy against 30.90% for the strongest baseline, xRAG; full context reaches 27.92% with 1,974 times more document tokens;
• LaMP-7: ROUGE-L rises from 43.13 without memory to 48.37;
• TriviaQA: best exact match at 57.0, with 743 times fewer document tokens than full context;
• NarrativeQA: up to 4,212 times fewer tokens than full context;
• latency: 27 times lower than full context with 512 documents, measured end to end on one H100-80GB with Qwen3-8B at batch size 1.

Two caveats appear on the page itself. PersonaMem-v2 and PersonalizationV4 receive additional benchmark-specific fine-tuning, and the RAG baseline was not timed for latency.

How is Memorilla different from RAG?

Retrieval-augmented generation (RAG) selects a few passages and pastes their exact text into the prompt. According to the project page, RAG and Memorilla can share the same document embeddings and the same frozen LLM, but Memorilla reads the whole collection, so it can connect information that resembles the question only weakly. In geometric mean across the eight benchmarks, it uses 113 times fewer document tokens than RAG-50, according to the same page.

The trade-off is stated openly. RAG gives the model the exact text it selects, which suits questions whose answer is a name found verbatim. On NarrativeQA, Memorilla gains most over RAG-50 on "why" and "how" questions, while RAG-50 wins on "who" and "which", and the project page notes that a fixed number of memory tokens can lose fine details such as exact names.

Can it help AI agents remember?

The arXiv abstract also tests continual learning in agents, where the number of documents grows with every turn. In a TextWorld environment, a frozen Qwen3-4B agent explores up to 65 rooms to complete a quest within 50 turns, with Memorilla condensing its history into eight memory tokens trained by reinforcement learning, according to the project page. The paper reports that the agent wanders less between already explored rooms and that its average success rate rises from 18.6% with full-history in-context learning to 24.2%.

What does this change for a business?

For a company building assistants on an open model it cannot retrain, the result points to a way to give the model a long memory of customers or documents at a fixed prompt cost. Treat it as research for now: the results come from the authors' own benchmarks and from Qwen3 models, and the sources we could read do not detail licensing or production tooling. An international team would still do well to test it against its own data and against a plain RAG pipeline, especially where exact names and figures matter.