← Comparisons · 2026-07-05
comparison / evermemos
EverMemOS, on their own bench.
EverMemBench is the benchmark EverMemOS built, named after themselves, and published a leaderboard for — with their own system on top at 44.6. We ran their harness, their corpus, their judge, their protocol, unmodified, with a thin adapter in the same slot Mem0, Zep and MemOS occupy. Mazemaker: 48.7% category mean on the leaderboard protocol — four points clear, on their turf. Swap the answering model for a budget hybrid-reasoner and the same retrieval yields 51.5%.
The numbers
| System | Answerer | Category mean | Pooled |
|---|---|---|---|
| Mazemaker | gpt-4.1-mini (leaderboard protocol) | 48.7% | 51.0% (319/626) |
| EverMemOS (published) | gpt-4.1-mini | 44.6% | — |
| MemOS (published) | gpt-4.1-mini | 41.6% | — |
| Zep / Mem0 / MemoBase (published) | gpt-4.1-mini | below MemOS | — |
| Mazemaker | deepseek-v4-flash ($0.09/$0.18 per M) | 51.5% | 53.5% (335/626) |
The bench: a year of multi-person group chat — 10,222 messages, 254 days, multiple rooms — ingested into the memory system, then 626 questions (389 multiple-choice by exact option match, 237 open-ended by gpt-4.1-nano judge). Every leaderboard system answers with gpt-4.1-mini, so the memory system is the only variable. Mazemaker’s configuration: the full production engine — GPU BGE-M3, ColBERT + DAE rerank, Postgres + pgvector, retrieval depth k=30. The engine that produced these numbers is the engine curl | bash installs today, release 1.0.0-rc7 — including the five fixes this bench surfaced, which we shipped before publishing the win. The full story, including every bug: The arena we did not build.
What the budget row demonstrates
The flash row is not leaderboard-comparable — the other systems were not re-run with flash, and we say so everywhere we show it. What it demonstrates is that the retrieval quality has headroom the standard answerer cannot exploit: a model four to nine times cheaper than gpt-4.1-mini extracts four more points from the identical memories. The category that moves most is update tracking — “which value is current after three revisions” — at 50%, fourteen points above anything a non-reasoning answerer managed. If your agent stack runs on budget models, this row is the one that matters to you.
Reproduce
# EverMemBench official harness (their repo), Mazemaker adapter in eval/src/adapters/
python -m eval.cli --dataset dataset/004/dialogue.json --system mazemaker \
--user-id run1 --stages add
python -m eval.cli --qa dataset/004/qa_004.json --system mazemaker \
--user-id run1 --stages search answer evaluate --top-k 30
# answer model: openai/gpt-4.1-mini · judge: openai/gpt-4.1-nano (paper protocol)One full evaluation costs about $0.40 in API calls and ~45 minutes end to end. Every retrieval, answer and judgment lands in a JSON on disk.
Honest caveats
- Our 48.7% mean is computed over the ten question categories encoded in EverMemBench’s own question IDs. Their published table groups subtasks slightly differently; we have not reproduced their exact aggregation. The pooled 51.0% (319/626) has no such ambiguity.
- The open-ended half is scored by an LLM judge (gpt-4.1-nano, per their protocol). Judge-sensitivity is a known failure mode of memory benchmarks — we have written about it at length — but here the judge is theirs, held constant across systems.
- The flash number mixes answer models and must not be read as a leaderboard entry. It is a cost-efficiency measurement of one system, stated as such.
- Published numbers for EverMemOS, MemOS, Zep, Mem0 and MemoBase are from the EverMemBench publication; we did not re-run their systems.
- Our first same-day publication of this page showed 46.4 / 51.1 — produced with the ColBERT rerank channel accidentally disabled (a stale debugging default gated blob writes off; the channel silently contributes zero without them). We found it in our own wiring audit, backfilled the corpus, and re-ran everything: the corrected numbers above are with the channel live. Full handbrake story in the blog post.