Skip to content

Measurements

Every number on this page comes from a run on a real install. The date and the command are given, so you can run it again.

2026-10-13, agent-mem 1.0.0 (7d6cf39), Python 3.13, Apple Silicon Mac, a fresh data folder, 40 sessions of six Claude Code events each. Times are the whole agent-mem hook claude process as the agent sees it:

EventAveragep50p95p99
SessionStart70 ms69 ms79 ms87 ms
UserPromptSubmit67 ms65 ms74 ms103 ms
PreToolUse65 ms64 ms72 ms93 ms
PostToolUse66 ms64 ms76 ms98 ms
Stop65 ms64 ms73 ms81 ms
SessionEnd64 ms63 ms72 ms82 ms
python -c pass29 ms29 ms30 ms31 ms

About 29 ms of each is Python starting. The rest is the hook’s own work, well inside its 400 ms deadline. A PreToolUse with nothing to check (no rules, no file history) exits through the fast path about 45 ms sooner.

the script (shortened)
for i in range(40):
sid = f"bench-{i}"
for name, payload in six_events(sid): # SessionStart … SessionEnd, Claude Code payloads
t = time.perf_counter()
subprocess.run(["agent-mem", "hook", "claude"], input=json.dumps(payload),
capture_output=True, text=True,
env={**os.environ, "AGENT_MEM_DATA_DIR": tmp, "AGENT_MEM_NO_SPAWN": "1"})
times[name].append((time.perf_counter() - t) * 1000)

LongMemEval asks questions about long chat histories. The _s set has 500 questions over histories of about 115k tokens each; agent-mem answers retrieval: does the right session appear in the top 5 or 10? 470 questions (the 30 abstention questions have no session to find).

2026-10-12, SQLite backend, MiniLM-L12-v2, no model call:

CategorynR@5R@10All@5All@10
knowledge-update721.0001.0001.0001.000
multi-session1210.9750.9920.7770.876
single-session-assistant561.0001.0001.0001.000
single-session-preference300.9001.0000.9001.000
single-session-user640.9841.0000.9841.000
temporal-reasoning1270.9610.9840.7870.874
overall4700.97450.99360.87660.934

R@k: at least one evidence session in the top k. All@k: every evidence session in the top k.

Terminal window
agent-mem eval longmemeval_s.json --progress lme.progress.jsonl

tests/fixtures/memory_cases.json: 20 answerable questions, 20 that must get no answer, 10 pairs where a newer memory replaces an older one. 2026-10-13, fresh database, SQLite backend, MiniLM-L12-v2:

MetricValueMeaning
hit@11.00the top raw search result contains the expected answer
abstention0.45raw search returns nothing for an unanswerable question (raw search always lists candidates)
injected_hit@10.90what the prompt hook injects has the answer first
injected_abstention1.00the prompt hook injects nothing for every unanswerable question
mcp_hit@11.00mem_search’s first confident result has the answer
mcp_abstention1.00mem_search says “no confident match” for every unanswerable question
contradictions_resolved0/10the top result shows only the newer value

The last line is expected without a model: agent-mem keeps the old and the new fact, shows both with dates, and the agent reads the newer one as current. The daily review, which needs a model, retires the old one. The fixture runs without it.

Terminal window
agent-mem eval --fixture tests/fixtures/memory_cases.json

2026-10-13, agent-mem 1.0.0 (7d6cf39), a fresh data folder with the fake model provider. Six sessions of ten requests each fired real hook processes in parallel, rotating Claude Code, Codex and Copilot payloads, while a second process took the database’s write lock for 800 ms every 1.5 seconds, and the indexer, the observer, the viewer and MCP search ran alongside:

CheckResult
hook processes612 in 7.3 s
hook latencymedian 67 ms, p95 101 ms, max 272 ms; none over the 400 ms deadline
invalid hook output or stderrnone
viewer and search errors during the runnone
requests complete, every tool call in its request, spool drained, nothing quarantinedyes
observer queue afterwards60 of 60 done
health errorsnone

Fewer pre_tool events (152) than tool calls (240) is the fast path: calls with nothing to check are not stored.

WhatCommand
fixtureagent-mem eval --fixture tests/fixtures/memory_cases.json (from a checkout)
LongMemEval-Sdownload longmemeval_s.json, then agent-mem eval longmemeval_s.json
compare with mem0’s own searchagent-mem eval longmemeval_s.json --backend mem0
without vectorsadd --no-semantic
hook latencythe script above with your own payloads

agent-mem eval builds its own database in a temporary folder; it never touches your memory.