Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning

Authors: Sikuan Yan, Xiufeng Yang, Zuchao Huang, Ercong Nie, Zifeng Ding, Zonggen Li, Xiaowen Ma, Jinhe Bi, Kristian Kersting, Jeff Z. Pan, Hinrich Schuetze, Volker Tresp, Yunpu Ma

Affiliations: Ludwig Maximilian University of Munich, Munich Center for Machine Learning, Technical University of Munich, University of Cambridge, University of Hong Kong, Technical University of Darmstadt, University of Edinburgh

Venue: arXiv:2508.19828v5 [cs.CL]

Year: 2026 (submitted January 2026)

Code: Not explicitly mentioned in the paper text

Pages: 20 (main paper + appendix)

PDF 文件: [Memory-R1 Paper](file:///C:/Users/admin/.openclaw/workspace/attachment/papers/20260629_memory_r1_enhancing_llm_agents_manage_utilize_memories_rl.pdf)

研究摘要 (Research Summary)

The fundamental challenge that Memory-R1 addresses lies at the intersection of two critical limitations in modern Large Language Models (LLMs): their stateless nature and their finite context windows. Despite the remarkable capabilities that LLMs have demonstrated across natural language processing tasks, they remain fundamentally unable to maintain persistent knowledge across extended interactions. Any information that falls outside the context window is effectively forgotten, creating a severe bottleneck for long-horizon reasoning tasks that require synthesizing information across multiple sessions or conversations. This limitation is not merely a technical inconvenience—it fundamentally constrains the ability of LLM agents to serve as genuine long-term assistants, collaborators, or memory-bearing entities.

The existing landscape of memory-augmented systems has largely approached this problem through heuristic-driven retrieval-augmented generation (RAG) pipelines, where external memory banks are appended to the model's input prompt. Yet these approaches suffer from a fundamental retrieval dilemma: static heuristics may return too few entries, omitting crucial context, or too many, flooding the model with irrelevant information and degrading performance. More critically, existing systems lack a learned mechanism for deciding what to store, update, or discard—operations that humans perform effortlessly but remain elusive for artificial systems.

Memory-R1 reframes this challenge through the lens of reinforcement learning (RL), arguing that the missing ingredient for adaptive memory in LLM agents is not more sophisticated retrieval algorithms, but rather an outcome-driven learning signal that teaches the model to manage and utilize memory effectively. The paper introduces a novel framework consisting of two specialized agents: a Memory Manager that learns structured operations—ADD, UPDATE, DELETE, and NOOP—to maintain and evolve an external memory bank, and an Answer Agent that applies a learned Memory Distillation policy to filter retrieved memories and reason over the most relevant entries. Both agents are fine-tuned using Proximal Policy Optimization (PPO) or Group Relative Policy Optimization (GRPO), with rewards derived from downstream answer correctness rather than manual annotations of memory operations.

The intellectual contribution of this work extends beyond the specific architecture to a broader methodological insight: reinforcement learning can align LLM behavior with high-level objectives in structured decision-making tasks, and memory management is precisely such a task. The authors demonstrate that with only 152 training question-answer pairs, Memory-R1 achieves state-of-the-art performance on the LoCoMo benchmark, outperforming strong baselines including Mem0, A-Mem, and MemoryOS. The improvements are substantial—relative gains of 28% in F1, 34% in BLEU-1, and 30% in LLM-as-a-Judge scores—yet what makes these results particularly compelling is the minimal supervision required. This data efficiency suggests that the RL framework is not merely memorizing patterns but learning genuine principles of memory management that transfer across contexts.

The broader significance of Memory-R1 lies in its demonstration that reinforcement learning can serve as a general paradigm for adaptive memory in LLM agents. By optimizing outcome-based rewards, the model learns when to add new information, when to consolidate existing memories through updates, when to prune contradictory content, and how to distill relevant facts from noisy retrieval. This opens a new research direction where memory systems are not hand-engineered but learned, adapting to the specific needs of tasks and domains through interaction and feedback.

理论框架 (Theoretical Framework)

The theoretical foundations of Memory-R1 build upon a rich intellectual lineage that spans memory-augmented systems, reinforcement learning, and the emerging paradigm of outcome-driven optimization for large language models. The work draws directly from the retrieval-augmented generation (RAG) paradigm, which has become the dominant approach for extending LLM capabilities beyond their context windows. However, Memory-R1 fundamentally challenges the static nature of traditional RAG by introducing learned memory operations and selective memory utilization, moving beyond the simple append-retrieve-read pipeline that has characterized most prior work.

The conceptual architecture of Memory-R1 centers on two distinct but complementary agents. The Memory Manager operates as a policy πθ that takes extracted information x from a dialogue turn and retrieved memories Mold from the current memory bank as input, then outputs an operation o with updated content m′. This formulation can be expressed as (o, m′) ∼ πθ(· | x, Mold), where the policy must learn to navigate the complex decision space of memory operations. The four operations—ADD, UPDATE, DELETE, and NOOP—form a minimal yet expressive framework for modeling memory dynamics, derived from database CRUD operations but adapted for the semantic richness of natural language memories. The ADD operation inserts new information into the memory bank, UPDATE merges new information with existing entries while preserving identifiers, DELETE removes contradictory or obsolete information, and NOOP leaves the memory unchanged when the new information is redundant or irrelevant.

The training of the Memory Manager employs either PPO or GRPO, both of which are actor-critic methods designed for stable policy optimization. The PPO objective takes the form:

where ρ_θ = π_θ(o, m′ | x, M_{old}) / π_{old}(o, m′ | x, M_{old}) represents the importance ratio between the current and old policies, A is the advantage estimated from answer-based rewards, and ε is the clipping threshold that prevents overly aggressive policy updates. This clipping mechanism is crucial for stability, as memory operations are discrete decisions with sparse rewards, and unconstrained policy updates could easily destabilize training.

The GRPO formulation extends this approach by sampling a group of G candidate actions per state and computing their relative advantages. The objective becomes:

where each candidate i yields reward r_i, and its advantage A_i = (r_i - \text{mean}(r)) / \text{std}(r) is computed relative to the group mean and standard deviation. This group-relative normalization eliminates the need for an explicit value function, simplifying the architecture while maintaining PPO-style stability. The KL divergence term D_{KL}[π_θ ∥ π_{ref}] regularizes updates to prevent the policy from drifting too far from the reference model π_{ref}, preserving linguistic coherence and preventing catastrophic forgetting of the base model's capabilities.

The reward design for both agents follows an outcome-driven philosophy that represents a key theoretical contribution. Rather than labeling individual memory operations as correct or incorrect—which would require expensive human annotation and may not generalize—the authors use downstream answer correctness as the reward signal. For the Memory Manager, after applying operation o with proposed content m′, the updated memory bank is passed to a frozen Answer Agent, and the reward is computed as R_{answer} = EM(y_{pred}, y_{gold}), where EM denotes exact match between the predicted answer and ground truth. This sparse but meaningful reward creates a credit assignment challenge—the Memory Manager must learn which operations lead to better downstream answers without receiving direct feedback on the operations themselves. This is a form of delayed reward optimization, where the effects of memory operations manifest only through the answers generated by a separate agent.

The Answer Agent operates as a second policy π_θ that maps the question q and retrieved memories M_{ret} to an answer y, sampled as y ∼ π_θ(· | q, M_{ret}). The same PPO and GRPO formulations apply, but the action space is now the generation of natural language answers rather than discrete memory operations. The reward for the Answer Agent is simply the exact match score between the generated answer and the gold answer, directly tying the optimization objective to the task of interest.

The theoretical assumptions underlying this framework are worth examining carefully. The decoupled training of the two agents assumes that memory management and answer generation can be optimized separately, which simplifies training but may miss opportunities for richer coordination. The sparse reward signal assumes that exact match is a sufficient proxy for answer quality, which may not capture semantic nuance or partially correct answers. The reliance on retrieved memories assumes that the retrieval mechanism is sufficiently good to surface relevant candidates, though the Memory Distillation policy is designed to compensate for retrieval noise. Finally, the outcome-driven reward assumes that the effects of memory operations are detectable in downstream answers, which may not hold for operations whose benefits are deferred or subtle.

Despite these limitations, the theoretical framework represents a significant advance over heuristic memory systems. By framing memory management as a reinforcement learning problem, the authors provide a principled mechanism for learning adaptive memory policies that can optimize for task-specific objectives. The group-relative advantage estimation in GRPO is particularly well-suited to this setting, as it provides a natural baseline for comparing the quality of different memory operations without requiring an explicit value network. The KL regularization ensures that the learned policies remain close to the base model's behavior, preventing the agents from diverging into unnatural or ungrammatical outputs.

技术架构 (Technical Architecture)

The Memory-R1 framework is organized as a two-stage pipeline that mirrors the dual-agent architecture: Stage 1 constructs and updates the memory bank through the Memory Manager, while Stage 2 answers user questions through the Answer Agent with Memory Distillation. This separation is not merely an engineering convenience but a fundamental design choice that reflects the distinct temporal scales and action spaces of the two tasks. Memory management occurs at the granularity of individual dialogue turns, requiring incremental updates to a persistent store, while answer generation occurs at the granularity of questions, requiring synthesis across potentially distant memories.

The data flow through the system begins with a multi-turn dialogue containing multiple sessions, where each session consists of several back-and-forth exchanges between two users. For each dialogue turn, an LLM first extracts key information worth remembering, producing a concise summary of the new facts. These facts are then used as a query to retrieve semantically related entries from the existing memory bank via standard retrieval-augmented generation techniques. The retrieved memories, along with the new facts, form the input to the Memory Manager, which must decide whether to add the new information as a fresh memory, update an existing entry to incorporate the new facts, delete a contradictory memory, or do nothing if the information is redundant.

The Memory Manager's decision is implemented as a structured operation with both a categorical choice (ADD, UPDATE, DELETE, NOOP) and content generation (the updated memory text m′). When the operation is ADD, the new memory is inserted with a fresh identifier. When it is UPDATE, the system merges the new information into an existing memory entry, preserving its identifier but modifying its content. When it is DELETE, the identified memory is removed from the bank. When it is NOOP, the memory bank remains unchanged. This structured output format ensures that memory operations are interpretable and reversible, facilitating debugging and analysis.

The construction of the memory bank follows Algorithm 3 in the paper, which processes dialogue turns sequentially. For each turn, the system extracts facts, retrieves related memories, invokes the Memory Manager to determine the operation, and updates the bank accordingly. This sequential processing is important because the memory state evolves over time, and later operations may depend on earlier ones. The memory bank thus serves as a persistent state variable that accumulates knowledge across the entire dialogue history, compacting and consolidating information as it arrives.

For question answering, the system switches to the Answer Agent. Given a question, the system retrieves the top-k relevant memory candidates from the bank—typically 60 entries, following the Mem0 configuration—and concatenates them with the question to form a memory-augmented prompt. However, rather than passing all 60 memories directly to the generation model, the Answer Agent applies a Memory Distillation policy that selects only the most relevant entries. This distillation step is crucial because it filters out retrieval noise and reduces the cognitive load on the generation model, analogous to how humans focus on the most salient details when answering questions from memory.

The distillation process is learned through reinforcement learning, with the Answer Agent trained to identify which memories are essential for correct answering. The agent outputs not only the final answer but also the selected memories, creating a traceable reasoning chain that shows which pieces of information were deemed relevant. This transparency is valuable for debugging and interpretability, as it allows users to inspect the evidence that led to a particular answer.

The training procedure for Memory-R1 is performed in two stages with alternating optimization. When training the Memory Manager, the Answer Agent is frozen and used solely to provide outcome-based rewards. The Manager's operations are reinforced if the resulting memory state improves the Answer Agent's ability to answer correctly. Conversely, when training the Answer Agent, the Memory Manager is fixed to ensure stable memory inputs. This decoupled setup avoids the attribution ambiguity that would arise from simultaneously optimizing both agents, where it would be unclear whether improvements in answer quality stem from better memory management or better answer generation.

The implementation details reveal careful engineering choices. The authors use the VERL framework for reinforcement learning, with a total batch size of 128 and a micro-batch size of 2 per GPU. During RL training, the decoding temperature is set to τ = 1.0 to encourage exploration and collect diverse reward signals, which helps stabilize policy learning by ensuring that the agents encounter a wide variety of memory states and question types. For validation and testing, greedy decoding (τ = 0) is applied to ensure deterministic outputs and consistent metric evaluation. The maximum prompt and response lengths are set to 4096 and 2048 tokens respectively, providing sufficient context for multi-turn dialogues while keeping computational costs manageable.

The model backbones include LLaMA-3.1-8B-Instruct and Qwen-2.5 Instruct at three scales (3B, 7B, 14B), demonstrating the framework's robustness across different architectures. Training is conducted on 4 NVIDIA H100 GPUs (80GB each) for most experiments, with the 14B model requiring 8 GPUs. The learning rates for PPO are set to 1 × 10⁻⁶ for the actor and 1 × 10⁻⁵ for the critic, using a constant warmup schedule. These conservative learning rates reflect the challenge of fine-tuning large language models with reinforcement learning, where aggressive updates can easily destabilize the policy or cause catastrophic forgetting.

The data construction for training is equally thoughtful. For the Memory Manager, GPT-4o-mini is used to build a temporal memory bank from the preceding 24 turns of each dialogue, and the current turn is fused with this snapshot to form the training input. No explicit labels are provided for memory operations; instead, the RL framework discovers the correct operations through trial and reward. For the Answer Agent, 60 candidate memories are retrieved for each question using RAG over the temporal memory bank, and the agent learns to distill these into a concise, correct answer. This training setup requires only 152 question-answer pairs from the LoCoMo training split, making it remarkably data-efficient compared to supervised approaches that would require thousands of labeled memory operations.

实验评估 (Experimental Evaluation)

The experimental evaluation of Memory-R1 is designed to test three core hypotheses: that reinforcement learning improves memory management beyond heuristic baselines, that the framework generalizes across different model scales and benchmarks, and that each component contributes meaningfully to the final performance. The authors evaluate on three benchmarks—LoCoMo, MSC, and LongMemEval—using three metrics: token-level F1, BLEU-1, and LLM-as-a-Judge. These metrics capture different aspects of answer quality: F1 measures lexical overlap with ground truth, BLEU-1 measures unigram precision, and LLM-as-a-Judge uses a separate language model to assess semantic correctness, relevance, completeness, and contextual appropriateness.

LoCoMo serves as the primary benchmark, containing long multi-session dialogues with about 600 turns and 26k tokens, covering single-hop, multi-hop, open-domain, and temporal reasoning questions. The authors use a 1:1:8 train/validation/test split with 152 training questions, 81 validation questions, and 1307 test questions. Models are trained only on LoCoMo and evaluated zero-shot on MSC and LongMemEval, providing a rigorous test of cross-task generalization. The baseline comparisons include LoCoMo (RAG), A-Mem, Mem0, MemoryOS, and Memory-SFT—a supervised fine-tuning variant of the Memory-R1 architecture trained on GPT-5-generated trajectories.

The main results on LoCoMo, presented in Table 1, demonstrate that Memory-R1 consistently achieves state-of-the-art performance across both model families. On LLaMA-3.1-8B-Instruct, Memory-R1-GRPO delivers the strongest overall performance, improving F1 by 28.5%, BLEU-1 by 34.0%, and LLM-as-a-Judge by 30.2% relatively over the strongest baseline MemoryOS. The improvements are particularly pronounced on multi-hop and temporal questions, which require synthesizing information across multiple sessions—precisely the scenario where effective memory management is most critical. Memory-R1-PPO also yields substantial gains, though slightly lower than GRPO, suggesting that the group-relative advantage estimation in GRPO provides more stable learning for this task.

Table 1: Evaluation results on LoCoMo benchmark with LLaMA-3.1-8B-Instruct. Best results in bold.

When applied to Qwen-2.5-7B-Instruct, Memory-R1-GRPO again emerges as the top performer, surpassing MemoryOS by margins of 24.5% (F1), 24.1% (BLEU-1), and 20.0% (LLM-as-a-Judge). The consistency of improvements across both LLaMA and Qwen architectures suggests that the benefits of RL-based memory management are not specific to a particular model family but generalize across different pretraining distributions and architectural choices. Notably, while Memory-SFT benefits from guidance by a powerful teacher model (GPT-5), the reinforcement learning approach still outperforms it, highlighting the effectiveness of outcome-driven optimization over purely supervised imitation. This is a significant finding because it suggests that RL can discover policies that exceed the quality of the demonstrations used to train supervised models, particularly in tasks where the optimal behavior is difficult to specify through examples alone.

The scalability analysis across Qwen-2.5 model sizes (3B, 7B, 14B) reveals that Memory-R1 consistently outperforms the base model at every scale, with both PPO and GRPO delivering clear gains in all metrics. These improvements persist as models scale, demonstrating that reinforcement learning remains effective in teaching memory management regardless of backbone capacity. This scalability is important because it suggests that the benefits of Memory-R1 are not limited to small models that might be expected to benefit most from external memory assistance; even larger models with more internal capacity can improve their long-horizon reasoning through learned memory management.

The cross-task generalization results are equally compelling. When applied zero-shot to MSC and LongMemEval—benchmarks on which the models were never trained—Memory-R1 continues to achieve consistent improvements across all metrics. This zero-shot transfer highlights the robustness of the learned memory policies and shows that they generalize beyond the training distribution to new dialogue styles and question types. The gains extend across single-hop, multi-hop, open-domain, and temporal questions, demonstrating Memory-R1 as a generalizable framework for adaptive memory augmentation.

The ablation studies provide deeper insight into the contribution of each component. Removing the RL-fine-tuned Memory Manager consistently degrades performance, with F1, BLEU-1, and LLM-as-a-Judge scores dropping substantially under both PPO and GRPO. This confirms that outcome-driven RL enables more effective memory operations than scripted control. Similarly, removing the RL-fine-tuned Answer Agent degrades answer quality, though the effect is more pronounced for GRPO than PPO. The Memory Distillation mechanism also contributes meaningfully, with GRPO showing larger gains from distillation than PPO, increasing overall F1 from 41.0 to 45.0 and BLEU-1 from 34.4 to 37.5. These results indicate that filtering irrelevant memories reduces noise and improves reasoning, particularly when combined with the group-relative optimization of GRPO.

The comparison between reward designs reveals an interesting trade-off. Using LLM-as-a-Judge as the reward signal achieves the highest J score (63.58) but performs poorly on F1 and BLEU-1 because it encourages longer, more descriptive answers that are penalized under string-overlap metrics. For example, when asked whether John and James studied together, the EM-based model outputs "Yes," while the J-based model produces a verbose explanation. Although both are semantically correct, the latter is penalized under F1 and BLEU-1, making direct comparison with baselines difficult. The authors wisely adopt the EM reward for their main experiments, which yields balanced improvements across all three metrics.

案例研究 (Case Studies)

The paper provides several representative examples that illuminate how Memory-R1 operates in practice, revealing both the strengths of the learned policies and the subtle failures of heuristic approaches. The first example concerns a user who mentions adopting a dog named Buddy in one session, and later mentions adopting another dog named Scout. The vanilla memory manager, lacking the nuanced understanding that comes from reinforcement learning, misinterprets this sequence as a contradiction. It sees two different dog names and assumes that the second statement replaces the first, issuing a DELETE operation for Buddy and an ADD operation for Scout. This fragmentation leaves the memory bank with only one dog, preventing the system from correctly answering questions about how many dogs the user has.

In contrast, the RL fine-tuned Memory Manager recognizes that these events are complementary rather than contradictory. It consolidates the information with a single UPDATE operation, producing a memory that states "Andrew adopted a dog from a shelter and named him Buddy because he is his buddy, and later adopted another dog named Scout." This consolidated memory preserves the full temporal sequence and enables the Answer Agent to correctly answer that Andrew has two dogs. The case reveals how reinforcement learning teaches the model to understand the semantic relationship between statements—distinguishing between contradictory information (which should replace old facts) and complementary information (which should enrich existing memories).

A second example involving Joanna's allergies demonstrates even more sophisticated reasoning. Joanna mentions that she is allergic to most reptiles and animals with fur, and later adds that she recently discovered an allergy to cockroaches as well. The vanilla memory manager misinterprets this in two ways: it views the new allergy information as contradicting the broader statement about reptiles and animals with fur, failing to recognize that cockroaches are a specific instance of the broader category. It also interprets Joanna's expressed fondness for turtles as incompatible with her allergy to them, incorrectly assuming that emotional attachment and physical limitations cannot coexist. As a result, it deletes valuable emotional context about Joanna's admiration for turtles and her general enthusiasm toward pets.

The Memory-R1 manager, trained through reinforcement learning, handles this case correctly. It recognizes that the new allergy information is a more specific elaboration of the broader allergy statement, not a contradiction. It also understands that Joanna's fondness for turtles and her allergy to them are complementary facts that can coexist—she likes turtles but cannot keep them due to her allergies. The manager updates the relevant memories using targeted UPDATE operations, preserving both factual accuracy and emotional nuance. This case demonstrates that the RL framework has learned to reason about overlapping and evolving information, favoring memory consolidation over fragmentation. The model has effectively learned a form of commonsense reasoning about human preferences and limitations, understanding that people can both like things and be unable to have them.

The Answer Agent case study provides equally compelling evidence of the benefits of learned memory distillation. When asked whether John lives close to a beach or the mountains, the original model consumes all retrieved memories indiscriminately and defaults to "mountains," likely influenced by irrelevant mentions of mountaineering or hiking. In contrast, Memory-R1 filters the retrieved memories, selecting only the beach-related entries—such as John's nostalgic memory of having a film camera as a kid and taking many pictures at the beach, and his family photo at the beach expressing a commitment to continue their efforts. With these relevant memories distilled, the agent correctly answers "beach."

This case highlights how Memory Distillation helps the model discard noise and focus on true signals. The retrieved memories contain 60 entries, many of which are irrelevant or only tangentially related to the question. The original model, overwhelmed by this volume of information, makes an error by fixating on a misleading pattern. The RL-trained Answer Agent, however, has learned to identify the specific memories that directly support the correct answer, effectively performing a form of evidence-based reasoning. This behavior is reminiscent of how humans answer questions from memory—they do not recall every related detail but instead focus on the most salient facts that directly address the query.

These cases collectively reveal that Memory-R1 has learned more than just surface patterns. The Memory Manager has acquired an understanding of semantic relationships, temporal dynamics, and the coexistence of contradictory-sounding facts. The Answer Agent has learned to filter noise and identify relevant evidence. Both capabilities emerge from the outcome-driven reward signal, which reinforces behaviors that lead to correct answers without prescribing how those behaviors should be implemented. This emergent intelligence is a hallmark of effective reinforcement learning, where the optimization objective shapes capabilities that the designers may not have explicitly anticipated.

综合价值与局限 (Synthesis — Value and Limitations)

Memory-R1 represents a significant conceptual advance in how we think about memory systems for large language models. The paper's most important theoretical contribution is the demonstration that memory management can be learned through reinforcement learning rather than hand-engineered through heuristics. This shifts the paradigm from designing memory operations to teaching them, opening the door to adaptive memory systems that improve with experience and optimize for task-specific objectives. The framework provides a principled mechanism for aligning memory behavior with downstream performance, addressing a fundamental limitation of existing RAG-based approaches where retrieval and memory operations are decoupled from the ultimate goal of correct answering.

The practical impact of this work is substantial, particularly for applications requiring long-term interaction and persistent knowledge. Virtual assistants, customer service agents, educational tutors, and collaborative tools all stand to benefit from memory systems that can learn to maintain accurate, relevant, and coherent knowledge over extended conversations. The data efficiency of the approach—achieving state-of-the-art results with only 152 training examples—is especially promising for real-world deployment, where large labeled datasets are often unavailable or expensive to obtain. The minimal supervision requirement suggests that Memory-R1 could be adapted to new domains with relatively small amounts of task-specific data, making it a practical choice for niche applications.

The paper's strengths are numerous and compelling. The empirical results are robust, with consistent improvements across multiple benchmarks, model scales, and evaluation metrics. The ablation studies are thorough, isolating the contributions of each component and demonstrating that the full system is more than the sum of its parts. The case studies provide concrete, interpretable examples of how the learned policies differ from heuristic baselines, making the benefits of reinforcement learning tangible and understandable. The framework's generalization across model architectures and datasets suggests that the benefits are not tied to specific implementation choices but reflect fundamental principles of learned memory management.

However, the limitations are also worth examining honestly. The evaluation is restricted to dialogue-centric datasets, which, while covering a wide range of reasoning types, may not fully capture the complexity of real-world memory tasks. Extending Memory-R1 to multimodal data—where memories might include images, videos, or sensor readings—would introduce challenges that go beyond the current text-based framework. The separate training of the Memory Manager and Answer Agent, while necessary for stability under sparse rewards, makes the process less straightforward and may miss opportunities for richer coordination between the two agents. An end-to-end multi-agent reinforcement learning approach could potentially simplify training and enable more sophisticated interactions, though this would require advances in credit assignment and stability for multi-agent systems.

The reliance on exact match rewards, while practical for benchmarking, may not fully capture the nuances of answer quality. Partially correct answers, answers that are correct but phrased differently from the ground truth, and answers that provide additional helpful context are all treated as equally wrong under the exact match metric. This limitation is acknowledged by the authors through their use of LLM-as-a-Judge as an auxiliary metric, but the training itself is constrained by the binary nature of exact match rewards. More sophisticated reward models that capture semantic similarity and partial correctness could potentially improve the learned policies further.

The framework also assumes that the memory bank is of manageable size and that retrieval can surface relevant candidates effectively. For very long conversations spanning thousands of turns, the memory bank could grow large enough to make retrieval challenging, and the benefits of Memory Distillation might be offset by the difficulty of finding relevant memories in the first place. The paper does not address memory bank compression or hierarchical memory organization, which may become necessary for truly lifelong agents.

Despite these limitations, Memory-R1 makes a compelling case for reinforcement learning as a paradigm for memory-augmented LLMs. It demonstrates that complex cognitive functions—deciding what to remember, what to forget, and what to focus on—can emerge from simple outcome-driven optimization, provided the right architecture and training setup. This is a powerful insight that extends beyond memory systems to other aspects of agent behavior, suggesting that many of the capabilities we associate with intelligent agents might be learnable through RL rather than explicitly programmed.

延伸阅读与思考 (Further Reading and Reflection)

Memory-R1 builds upon a rich foundation of prior work that has explored memory augmentation for language models from multiple angles. The retrieval-augmented generation paradigm, exemplified by systems like LoCoMo (Maharana et al., 2024), has established the basic approach of extending LLM capabilities through external memory stores. The LoCoMo benchmark itself, with its long multi-session dialogues and diverse question types, provides the evaluation framework that makes Memory-R1's advances measurable. The Tensor Brain framework (Tresp et al., 2023) offers a more theoretical perspective on memory, using bilayer tensor networks to model episodic, semantic, and working memory in a unified architecture. While Memory-R1 takes a more practical, engineering-oriented approach, the conceptual distinction between different memory types that Tensor Brain introduces could inform future extensions of the framework.

The memory management literature provides several important points of comparison. MemGPT (Packer et al., 2023) introduced working and long-term buffers with scheduling policies, treating memory as an operating system abstraction. Mem0 (Chhikara et al., 2025) investigated explicit in-context memory operations with the same {ADD, UPDATE, DELETE, NOOP} set that Memory-R1 adopts, but relied on vanilla LLMs without reinforcement learning. MemoryBank (Zhong et al., 2024) proposed a compositional memory controller for lifelong agent memory, while A-Mem (Xu et al., 2025) developed dynamic agentic memory with linking and updating mechanisms. Memory-R1 advances beyond these approaches by learning the memory operations rather than scripting them, achieving superior performance with less supervision.

The reinforcement learning connections are equally important. The foundational work on RLHF (Ouyang et al., 2022) established that LLMs can be aligned with human preferences through reinforcement learning. More recent work has extended RL to structured decision-making tasks for LLMs, including tool use (Schick et al., 2023; Qian et al., 2025), web navigation (Wei et al., 2025), and search optimization (Jin et al., 2025; Song et al., 2025). Search-R1 (Jin et al., 2025) is particularly relevant as a parallel effort that trains LLMs to issue web search queries using RL, demonstrating that the outcome-driven optimization philosophy can extend to information retrieval tasks. Memory-R1 can be seen as applying this same philosophy to the internal memory of an agent rather than external search tools.

Alternative approaches to the same problem include supervised fine-tuning of memory operations, which Memory-SFT implements as a baseline. The fact that Memory-R1 outperforms this baseline despite using the same architecture and training data suggests that RL's ability to explore and discover superior policies is a genuine advantage, not merely an artifact of additional training. Other alternatives include neural memory architectures that modify the model itself to have persistent state, such as recurrent mechanisms or memory-augmented neural networks. While these approaches avoid the need for explicit memory operations, they typically require more invasive architectural changes and may not scale to the largest language models as effectively as external memory systems.

Looking forward, several promising research directions emerge from this work. End-to-end multi-agent reinforcement learning, where the Memory Manager and Answer Agent are trained simultaneously rather than alternately, could enable richer coordination and potentially superior performance. The challenge here is credit assignment—determining whether a correct answer stems from good memory management or good answer generation—but advances in multi-agent RL could make this feasible. Another direction is the extension to multimodal memory, where the memory bank contains images, videos, audio, or structured data in addition to text. This would require new retrieval mechanisms and memory operations but could dramatically expand the applicability of the framework.

Continuous learning and memory consolidation represent another frontier. Current Memory-R1 processes dialogues in batches during training, but a truly lifelong agent would need to learn incrementally from each interaction, consolidating new memories into its existing knowledge without catastrophic forgetting. This raises questions about memory stability and plasticity—how to preserve important old memories while incorporating new ones—that parallel long-standing questions in neuroscience and cognitive psychology. The RL framework could potentially be extended to optimize these trade-offs dynamically, learning when to consolidate, when to compartmentalize, and when to forget.

The deepest unsolved challenge in this area is the development of general, scalable memory systems that can support not just question answering but creative problem-solving, planning, and reasoning. Human memory is not merely a store of facts but a dynamic system that supports imagination, counterfactual reasoning, and the synthesis of new ideas from existing knowledge. Whether external memory systems for LLMs can support these higher cognitive functions remains an open question, but Memory-R1 provides a promising foundation by demonstrating that memory operations can be learned and optimized for task performance.

The most surprising aspect of this work is the data efficiency: the fact that 152 training examples can produce such substantial improvements over strong baselines suggests that the RL framework is capturing genuine principles of memory management rather than overfitting to specific patterns. This efficiency implies that the space of useful memory policies is smaller and more structured than one might expect, and that relatively little experience is needed to discover effective strategies. It raises the intriguing possibility that memory management, like many cognitive skills, may have a learnable structure that can be acquired quickly once the right learning framework is in place. This is a hopeful sign for the broader project of building adaptive, memory-capable agents, suggesting that we may not need massive datasets or complex hand-engineered rules to achieve sophisticated memory behavior.

笔记创建时间: 2026-06-29
阅读方式: L2 深度阅读

Topics:

• "agent_architecture"
• "memory_mechanism"
• "reinforce_learning"
• "llm"
• "long_term_memory"
  References:
• "lmu_munich"
• "mem0"
• "locomo_bench"
• "grpo"