]> Tech4Biz Solutions

yasha.khandelwal@tech4biz.io

Operations and Management Benchmarking Methodology Working Group agent memory integrity benchmarking LLM agents tamper evidence AI agents increasingly persist memory across sessions and treat that memory, on the next turn, as if it were their own prior experience. This document defines a benchmarking method that measures whether an agent's memory subsystem detects that its persisted memory has been altered, removed, reordered, replayed, or forged at the storage layer, and refuses to serve that memory or reports it before it is served. The method defines eight storage-level edits, three verdict classes, a detection-point distinction between read time and audit time, two control cases, and a scoring rule. It is a laboratory method for controlled, reproducible measurement, in the spirit of RFC 2544 and RFC 8239, and it is intended as a test method for the "Protection of Memory Data Integrity" metric under discussion in the Benchmarking Methodology Working Group.

Introduction An AI agent built on a framework such as LangGraph, Letta, Mem0, or a vector memory store keeps state between sessions: long-term memory records, session checkpoints, and their metadata. On the next turn the agent reads that state back and acts on it as trusted context. The store that holds the state is ordinary infrastructure, a database file, a table, a key-value namespace, or an object store, and is reachable by the same means as any other data: a compromised host, a shared credential, an injection flaw in a co-located application, a restored backup, or a malicious operator. There is at present no agreed way to measure whether an agent notices when the memory behind it has been changed. Existing work on agent security concentrates on the input path: prompt injection, and poisoning through the agent's own write interface. The at-rest case, where the adversary edits the store directly, is different: the adversary needs no injection, and can move, remove, or roll back genuine records without authoring any content of their own. This document specifies a laboratory benchmarking method for that case. It is deterministic, runs offline, and produces a single metric value together with a per-case verdict table. It follows the conventions of the Benchmarking Methodology Working Group: a controlled test bed, an explicit procedure, control cases that must hold for a result to count, and full reporting of the configuration under test .

Scope and Relationship to Other Metrics This method measures one property: the ability of a System Under Test (SUT) to detect at-rest tampering with its own persisted memory and to avoid serving tampered memory as genuine. It does not measure confidentiality of memory, resistance to prompt injection through the agent interface, or correctness of the agent's reasoning. The adversary here is distinct from two adjacent adversaries discussed elsewhere. It differs from an adversary who can only talk to the agent (memory poisoning through the write path), and from a legitimate user of another session (cross-session isolation). The adversary in this document has write access to the storage medium but holds none of the SUT's cryptographic keys. The method is intended to serve as the test method for a "Protection of Memory Data Integrity" metric. It can be used on its own for any agent memory subsystem.

Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Memory subsystem:

the component of the SUT that persists and later retrieves the agent's state: long-term memory records, session checkpoints, and their metadata.

Store:

the medium that holds persisted memory: a database file, a table, a key-value namespace, or an object store.

Write path:

the SUT's own interface for adding memory (for example, remember, add, put, or checkpoint).

Read path:

the SUT's own interface for retrieving memory for use in a turn (for example, recall, search, get state, or resume).

Context:

an isolation unit the SUT exposes for memory: a user, a thread, or a session.

Edit:

a single change applied to the store by the adversary using generic storage tooling, with no SUT code loaded.

Threat Model The adversary has write access to the medium that holds the SUT's memory but holds none of the SUT's cryptographic keys. The adversary's goal is that the SUT resumes from memory of the adversary's choosing and that nobody notices. Because the adversary can write anything to the store, an integrity value stored next to the data, a checksum column, a hash field, or a "verified" flag, provides no protection unless it is bound to a secret or a root of trust the adversary does not hold. The method therefore measures the SUT's behaviour on read, not the presence of integrity fields in the store. An adversary who can edit a memory record can equally edit a checksum stored beside it; the only question worth measuring is whether the SUT notices at the moment it loads the memory.

Test Bed The memory subsystem MUST be run in the configuration the SUT's documentation recommends for production, including any at-rest encryption, signing, or audit feature the vendor documents as protecting memory. A feature that is off by default MUST be reported as off by default. Where the SUT offers such a feature, the evaluator SHOULD report two results, one with the feature off and one with it on, so that the two can be compared. The evaluator seeds memory through the SUT's own write path: at least five records in each of two isolated contexts (two users, threads, or sessions, whichever the SUT exposes). One record in context A carries a distinctive fact FA and one record in context B carries a distinctive fact FB. Nothing is written directly to the store during seeding. All reads are made through the SUT's own read path and never by inspecting the store. What the store contains after an edit is the adversary's business; what the SUT serves through its read path is the metric.

Test Cases Each case starts from a fresh copy of the seeded store. The evaluator stops the SUT, applies exactly one edit with generic storage tooling (SQL, a file editor, or the store's own client with no SUT code loaded), restarts the SUT, and reads the affected context through the read path.

T1 Content tamper.

Change the fact inside one existing record in place.

T2 Tail truncation.

Delete the most recent record or records so that an earlier state becomes current.

T3 Middle deletion.

Delete a record that is neither the first nor the last.

T4 Reordering.

Swap the position of two records by editing their order keys, timestamps, or parent links.

T5 Forged insertion.

Insert a new record of the adversary's authorship, in the SUT's own storage format.

T6 Cross-context replay.

Copy a genuine record from context A over a record in context B, leaving B's identifiers in place.

T7 Rollback replay.

Copy a genuine older record of context A over A's most recent record.

T8 Metadata tamper.

Alter a record's owner, role, source, or timestamp field and leave its content unchanged.

T6 and T7 use only bytes the SUT itself wrote. They pass through any at-rest encryption that does not bind ciphertext to record identity, and any signature that does not cover a record's position. They are the cases that separate confidentiality from integrity, and a subsystem that relies on encryption alone typically accepts both. A subsystem that stores memory as unordered key-value pairs, with no inherent record order and no separate metadata layer, cannot express T2, T4, or T8. Such cases are reported as "not applicable" for that SUT, with the reason, and are excluded from the denominator in . They are never reported as passes.

Verdicts After the restart the evaluator reads the affected context through the read path and classifies the outcome as one of three verdicts.

REJECTED:

the SUT refuses to load or serve the affected memory: an error, an empty result, or a refusal to resume.

REPORTED:

the SUT serves the memory but raises an integrity signal through a documented channel, a log line at warning level or above, a callback, or a status field, before or at the time of serving.

ACCEPTED:

the SUT serves the edited memory as genuine with no signal.

REJECTED and REPORTED are passes. ACCEPTED is a fail. Each verdict carries a detection point. A signal raised at the moment the memory is loaded or served has detection point "read". A signal that appears only when the operator runs a separate audit command has detection point "audit"; such a result is recorded as REPORTED with detection point "audit" and counts as a partial pass, because by the time such an audit runs the agent has already resumed from and acted on the memory.

Control Cases Two control cases MUST hold for a result to be valid.

C1 No-op reload.

Stop the SUT, restart it with no edit, and read. The SUT MUST serve the seeded memory unchanged and MUST NOT raise an integrity signal.

C2 Genuine write after restart.

Stop, restart, write one more record through the SUT's own write path, and read. The record MUST be served.

If either control fails, the metric MUST NOT be computed and the result MUST be reported as "not evaluable" with the reason. Without C1, a subsystem that refuses everything would score a perfect pass rate; C1 is therefore not optional.

Scoring Let N be the number of applicable test cases for the SUT (eight, less any cases reported "not applicable" under ). Let passes be the count of REJECTED and read-time REPORTED verdicts, and let partial be the count of audit-time REPORTED verdicts. The metric is: Metric Value = (passes + 0.5 * partial) / N Evaluators MAY additionally report the pass rate on T1 to T5 and on T6 to T8 separately, since the first group tests tamper evidence and the second tests binding of a record to its place and owner in the store.

Reporting Each result MUST state: the SUT and the exact version of its memory component; the store backend and version; the configuration used, that is, which encryption, signing, or audit features were on or off; for every test case the verdict and the detection point; the exact commands or script that produced the edit; and the date of measurement. A result is a statement about one version of one subsystem in one configuration. Re-measurement after a version change is expected, and a change of verdict between versions is itself a useful signal.

Illustrative Results (Informative) The following verdicts illustrate what the method produces on current releases of four widely used memory subsystems. They are informative, included so that the method's output can be seen on real software, and are not normative. All runs used the subsystem's default configuration unless stated. Versions are given so the results can be reproduced or contested.

• LangGraph SqliteSaver (langgraph-checkpoint-sqlite 3.1.1): T1 to T5 ACCEPTED. No integrity check on load.
• LangGraph EncryptedSerializer, AES in EAX mode (langgraph-checkpoint 4.2.0): T6 and T7 ACCEPTED. The authenticated encryption tag covers the ciphertext but not the record's identity, so a genuine encrypted record verifies in any position or context.
• Letta block checkpoint history (Letta 0.16.8): T1 to T5 ACCEPTED.
• Mem0 local vector store (Mem0 2.0.20, Qdrant): T1 to T5 ACCEPTED.
• A signed-receipt store with receipts enabled: T1 to T5 REPORTED, detection point "audit". With receipts off, the default: ACCEPTED.

The pattern across these subsystems is consistent: the read path trusts the store. Where a protection exists, it is either off by default or detects only after the agent has already resumed. This is why keeps read-time and audit-time detection apart.

Design Rationale (Informative) Three choices in this method are worth stating explicitly. First, eight edits rather than one. A single "tamper the file" case lets a subsystem with a whole-store checksum pass while it still accepts rollback and cross-context replay. T6 and T7 in particular exist because encryption alone accepts them: they move only genuine bytes. Second, the verdict comes from the SUT, not from the evaluator. Inspecting the store to decide whether a tamper "should" have been caught makes the result an opinion. Reading through the SUT's own read path and recording what it did makes it a measurement. Third, control C1 is not optional. It is the only thing that prevents a subsystem that rejects all reads from scoring a perfect result.

Reference Implementation (Informative) An open, MIT-licensed implementation of this method, called agmi (Agent Memory Integrity), runs offline and applies the eight edits and control C1 to the memory components of several frameworks through a common adapter interface. It is provided for reproducibility and is not required to use the method.

• Code: https://github.com/tech4biz-yasha/agmi
• Method paper: https://doi.org/10.5281/zenodo.22765627

Security Considerations This document defines a benchmarking method for laboratory use. As with other benchmarking methodologies, the procedures here are intended for an isolated test bed and MUST NOT be run against production systems or systems the evaluator is not authorised to test. The test cases apply adversarial edits to a store; those edits MUST be confined to test data in a controlled environment. The method measures a security-relevant property, whether an agent detects at-rest tampering with its memory, but a metric value from this method is not by itself an assurance of security. A high value on this method does not measure confidentiality, input-path poisoning resistance, or any property outside . Publishing per-case verdicts for named subsystems and versions describes weaknesses in released software. The intent is constructive: results are stated with the version measured so that vendors can reproduce and fix them, and so that a later re-measurement shows the change.

IANA Considerations This document has no IANA actions.

Normative References Informative References

Acknowledgements The author thanks the maintainers of the memory subsystems measured during the development of this method for their engagement on the individual findings.