MELT: Testing Long-Lived Memory for Agentic AI

Most agent memory benchmarks test retrieval from a frozen transcript. MELT is an open benchmark for the harder problem: whether memory still works as facts change, old information expires, and systems maintain what they know over time.
ai
agents
memory
benchmark
evaluation
open-source
Author

Leonard Lin

Published

July 13, 2026

While building the memory system for ShisaD, we ran into an awkwardly basic problem: how do you tell whether an AI agent’s memory is actually working?

Most popular memory benchmarks use some version of the same recipe: load a long, fixed conversation transcript, ask questions about it, and score the answers. That is useful, but it mostly tests retrieval and reading comprehension over a frozen snapshot.

Real memory gets harder after the snapshot changes. What if a user changes jobs? A good memory system should know the new employer, stop presenting the old one as current, and still answer where the user worked last year. It should preserve durable health information, such as an allergy, through consolidation while allowing a one-off doctor’s appointment for a cough or cold to recede from everyday retrieval. It should also keep two genuinely conflicting claims separate and say “I don’t know” when the evidence is not there.

Existing benchmarks generally don’t exercise that full loop, so we built MELT: Memory Evaluation for Lifecycle Testing, an open benchmark runner for testing memory over time.

TipTL;DR

MELT (Memory Evaluation for Lifecycle Testing) tests whether an agent memory system can write, update, maintain, and retrieve the right information as its state changes. The current release includes:

  • A native lifecycle benchmark with 13 behavioral axes and 217 cases in the full fixture
  • Scripted and agentic memory tracks
  • Adapters for LongMemEval, LoCoMo, and RHELM under the same report format
  • Runnable system adapters for ShisaD and Memobase, plus a deterministic reference adapter
  • Reproducibility metadata, answer/judge controls, baselines, checkpointing, and resume

Links:

A Memory Benchmark Should Test Memory

Static-conversation benchmarks answer an important question: can the system find enough evidence in a long history to answer correctly?

They do not necessarily tell you whether the system stored the right things, handled a correction, preserved history, survived maintenance, or forgot something at the right time. In fact, surprisingly simple baselines can do very well on retrieval benchmarks. Raw embeddings, grep, or dumping the full transcript into a sufficiently long context window can sometimes match or beat much more elaborate memory architectures.

That does not make retrieval benchmarks bad. MELT runs LongMemEval and LoCoMo because they remain useful comparison points.

We also support RHELM, a Microsoft-developed benchmark designed around evolving user profiles, heterogeneous sources, and questions requiring temporal and cross-source synthesis. RHELM is closer in spirit to MELT’s goals, although it remains primarily a QA evaluation over a constructed history; MELT’s native lifecycle suite additionally tests explicit writing, correction, supersession, consolidation, decay, and forgetting.

MELT’s native lifecycle suite focuses on the behaviors that show up once an agent has to live with its memories:

Area What MELT asks
Write quality Did the system capture the useful fact rather than the surrounding noise?
Structured semantics Did it preserve source IDs, timestamps, and supersession metadata?
Correction Did a newer fact become current without erasing the older fact’s history?
Contradiction Can conflicting claims remain distinguishable instead of becoming one false memory?
Temporal recall Can it answer both “what is true now?” and “what was true then?”
Maintenance Did consolidation and decay preserve durable facts while letting stale ones recede?
Core memory Did identity, constraints, and long-lived preferences survive?
Multi-hop recall Can it bring together related evidence from separate sessions?
Abstention Does it decline to answer when the memory does not support an answer?
Retrieval hygiene Can it exclude high-similarity distractors and superseded evidence?
Scope and source Does it keep users/workspaces isolated and handle chat, email, documents, and summaries?
Operations Do retries, time advancement, and maintenance cycles leave memory in a valid state?

(The implementation reports these as 13 separate axes; the table groups a few of them for readability.)

Scripted Memory vs. Agentic Memory

MELT separates two questions that are easy to blur together.

In the scripted lifecycle track, the harness explicitly tells the system what to write, supersede, consolidate, or decay. This tests execution: if a memory system is given a precise lifecycle operation, does it perform it correctly?

In the agentic memory track, MELT sends raw timestamped events and lets the system decide what should become memory. This tests policy: does the system decide to retain a durable preference, ignore noise, update stale information, preserve a real conflict, or forget ephemera on its own?

Those are different claims. A system can have excellent storage semantics but poor extraction policy, or make good write decisions on top of a weak retrieval layer. MELT keeps track, axis, write surface, time horizon, query mode, retrieval granularity, evidence type, and lifecycle operation separate in the report rather than compressing everything into one suspiciously tidy number.

There is also an unavoidable observability problem. Some systems can export their internal memory units; others are black boxes. MELT supports both, but labels them differently. Export-capable systems get direct write/update/forget policy scoring. Black-box systems are judged through retrieval and behavior. The latter, while still useful, is inherently weaker.

Benchmark and Harness

MELT is both a native benchmark and a system-agnostic runner. A suite supplies sessions, operations, probes, and expected outcomes; a System Under Test (SUT) adapter translates MELT’s versioned operations into whatever interface the memory system exposes.

melt run -> suite / fixture -> SUT adapter -> memory system
                    |                              |
                    +------- scoring <-------------+
                                |
                                v
                     reproducible JSON report

The same runner can execute MELT lifecycle cases or normalize established benchmarks into a common report envelope. Today it ships with:

  • lifecycle: smoke (7 cases), mini (48), full (217), and stress (373)
  • longmemeval: built-in smoke plus user-supplied full datasets
  • locomo: smoke and full runs, with Category 5 audit warnings and conversation-level ingestion
  • rhelm: smoke and full runs, plus a dependency-free dataset downloader

The adapter contract covers reset, ingest, structured writes, consolidation, time-aware query, answer generation, and shutdown. The built-in ShisaD adapter uses JSONL over a subprocess; the Memobase adapter talks to its HTTP service. A new system can use either pattern (or an in-process adapter) without changing benchmark scoring code.

Early Results: One “Memory Score” Is Not Enough

Our current public results are still preliminary: development fixtures, single runs, and known caveats. They are useful diagnostics, not leaderboard claims.

On MELT’s 217-case lifecycle-v4 full fixture, the current runs look like this:

System Lifecycle pass Retrieval R@3 Correction Contradiction Temporal Maintenance
ShisaD 95.4% 90.3% 100.0% 100.0% 23.5% 100.0%
Memobase 44.6% 13.4% 56.7% 13.3% 23.5% 13.3%

(We were as surprised as anyone by ShisaD’s lifecycle score. Although MELT grew out of our work on ShisaD, the benchmark cases were developed independently of its implementation; these are the initial results, and we did no evaluation-specific tuning. That said, ShisaD’s memory system was explicitly designed around agentic memory lifecycle mechanics, so this is a test it should be structurally well suited to.)

The useful result here is not simply that one aggregate number is higher. The per-axis view immediately shows that temporal/as-of recall is a weakness for both real systems in this run. That is exactly the kind of failure a static aggregate can hide.

Now compare the same systems on full LoCoMo judged question answering, using a GPT-5.4 Mini answer model and GPT-5.4 judge:

System Questions Judged QA score
Memobase 1,986 42.0%
ShisaD 1,986 6.2%

The ordering flips completely.

This is not a contradiction, and it is definitely not evidence that either table is a universal ranking. The lifecycle suite tests controlled memory behaviors; the LoCoMo run tests whether retrieved context supports answers to questions over long conversations. ShisaD’s current retrieval context was effective on the lifecycle cases but weak for LoCoMo QA. Memobase’s was the reverse. Different tests exposed different systems.

There are important caveats:

  • Every result above is a single development run and is marked preliminary.
  • The lifecycle runs trigger a retrieval-bypass warning because top_k=3 is at least the number of canonical sessions in the built-in cases. Per-axis lifecycle assertions remain useful, but the retrieval number should not be treated as competitive.
  • Neither real adapter exposes full memory export, so their write-policy metrics use weaker black-box proxies.
  • LoCoMo has known dataset and judging problems; MELT reports its Category 5 and dataset-audit caveats.

Evals are hard, and agentic memory evals are still nascent. Memory spans many distinct behaviors, which makes any single score easy to overread. MELT is our attempt to add nuance and tease apart those different dimensions.

For those interested in learning more, we also maintain Agentic Memory Research, a broad survey of agentic memory patterns, academic research, and implementations. If you are building agent memory, kick the tires, open an issue, or send a PR: github.com/shisa-ai/MELT.