Ranking beats structure.

Many agent memory systems call an LLM twice: once when a conversation comes in, to restructure it into summaries, clusters or memory cells, and again when a question arrives, to write search queries and route between memory tiers. SmartSearch asks whether either step is needed. It searches raw conversation history with a deterministic pipeline and uses a learned model in exactly one place: ranking.

“Recall is not the bottleneck. Ranking is.”

THE MAIN FINDING

Context windows now pass 100,000 tokens, but agents that run for weeks still produce more history than fits. Recent memory systems handle this by putting an LLM in two places. At ingestion, it turns conversations into structured memory. At query time, it writes search queries, picks which memory tier to search, or refines what comes back.

Both cost time and money. The ingestion cost is spread across every future question and rarely appears in per-query token counts, so these systems look cheaper at query time than they are overall. Learned routing adds training data and GPU inference on top.

When an LLM searches a conversation, it mostly does the same thing each time: pick out the names, search for them, read the results. SmartSearch does exactly that, deterministically. It weights query terms with part-of-speech and entity tags, finds candidates with plain substring matching, and expands to a second hop with rule-based entity extraction. The only learned component is the ranker that runs after retrieval.

What the paper contributes

• A multi-hop retrieval pipeline with no LLM in the loop. Query terms are weighted by part of speech and named entities instead of corpus statistics, and an index-free variant uses grep as its only retrieval tool.
• Results on LoCoMo and LongMemEval-S, plus a 27-configuration ablation on LoCoMo that traces where a 7.2-point gain comes from.
• Evidence that ranking after retrieval, not retrieval itself, is what limits accuracy on both short and long conversations.

Systems that structure memory first

EverMemOS uses an LLM to consolidate conversations into memory cells and scenes, then retrieves over several rounds with BM25, vector search and exact match. Memora builds a two-layer abstraction with LLM-written routing indices. M2A combines raw logs with LLM-generated semantic memory. TraceMem, MemSkill, BudgetMem and Locas add learned structuring or routing, which needs training data and GPUs. LongMem goes furthest and writes the conversation into model weights. SmartSearch does no processing at ingestion and uses a learned model only after retrieval.

Retrieval and multi-hop search

Substring matching is fast but brittle. BM25 is the standard keyword baseline. Dense retrievers such as DPR and ColBERT bridge vocabulary gaps but need precomputed indices. For multi-hop questions, methods like IRCoT, ITER-RETGEN, Self-Ask and ReAct have an LLM write each next query. SmartSearch keeps substring matching for recall, uses ColBERT only for the small tail where words do not match, and replaces the LLM-written next query with entity extraction over what was already retrieved.

Reranking

Cross-encoders score a question and a passage together and are usually trained on MS MARCO. Earlier work found that ranking quality matters a great deal for long dialogue. SmartSearch makes that the core of the system.

The pipeline has four stages: parse, retrieve (with an optional extra hop), rank and truncate. The first two are deterministic. Ranking is the only learned stage. A separate LLM writes the final answer from whatever survives.

How we evaluated

• LoCoMo-10: 10 conversations, 1,540 questions, about 9,000 tokens each, four non-adversarial question categories, with gold evidence marked at passage level.
• LongMemEval-S: 500 questions over histories of about 115,000 tokens, six of the seven question types (abstention excluded). Gold passages were derived from turn-level labels for 479 questions, with a median of two per question.
• Main results use gpt-4.1-mini to answer and gpt-4o-mini to judge, the same setup as EverMemOS and Memora. LoCoMo ablations use gpt-4o-mini for both; LongMemEval-S ablations use Claude Sonnet 4.6 to answer.

Published LoCoMo numbers are hard to compare: the full-context baseline alone scores anywhere from 77.1% to 91.2% depending on the evaluation framework. So LoCoMo results are reported under two protocols and never compared across them.

What an ideal search looks like

To see how much search a question really needs, retrieval was modelled as a graph of search states, where each step is a tool plus a set of terms at unit cost. Dijkstra’s algorithm then finds the cheapest path that reaches the gold evidence. The oracle knows which passages hold the evidence, not the answers. It produced traces for 1,317 of 1,536 LoCoMo questions (85.7%).

Parse and retrieve

SpaCy (en_core_web_sm) tags the question. Proper nouns get weight 3.0, nouns 2.0, verbs 1.0, and named entities a 1.0 bonus. This linguistic weighting replaces the corpus statistics BM25 relies on. Retrieval is exact substring matching over those terms, with precision left to the ranker. For about 3% of questions, names found in the retrieved passages trigger a second pass. A dense fallback covers the roughly 1% where substring matching finds nothing.

Query expansion

Two expansions are added for long histories. Entity discovery runs entity recognition on each hop’s results and searches new people, organisations, places and events at weight 2.5. Pseudo-relevance feedback takes nouns and proper nouns that appear in at least two of the top ten passages and runs one extra search at weight 0.5. Morphological expansion was tried and dropped: its noisy candidates cascaded through later hops. With both expansions, the index-free variant needs nothing but grep and one cross-encoder.

Rank

Without ranking, the first gold passage sits at an average position of 195 on LoCoMo and 47 on LongMemEval-S. Two models score candidates in parallel on CPU. The cross-encoder is mxbai-rerank-large-v1 (435M parameters, DeBERTaV3, trained on MS MARCO). The second is ColBERT answerai-small-v1 (110M), which scores token by token. Their rankings agree only loosely (Spearman 0.19 to 0.62), so they fail in different places. Reciprocal rank fusion merges them (k = 60, weights 0.7 and 0.3). Because they run side by side, a query takes about 650 ms with no GPU.

Truncate

The answer model only sees what fits the budget. A fixed budget adds passages in rank order up to 2,000 words, about 3,141 tokens on LoCoMo. The score-adaptive version first keeps the top 60 passages by fused score, then drops anything scoring below α times the best cross-encoder score, capped at 4,000 words.

Parse and retrieve

Index-free retrieval nearly matches the indexed pipeline on LoCoMo: 0.960 versus 0.967 recall within the budget, and 91.0% versus 91.9% accuracy. On LongMemEval-S the gap is wider before expansion, 78.4% versus 81.2%.

Expansion does nothing on LoCoMo and a lot on LongMemEval-S, where the two together add 9.2 points, more than the 6.0 and 1.4 they add alone. Temporal questions gain 12.8 points. Index-free with expansion beats the indexed baseline without it by 6.4 points.

Rank

From no reranker to the final system is 15.1 points. A controlled test puts 63% of the bge-base gain down to model quality and 37% to the extra context it allows. Architecture beats size: the 184M mxbai-base recovers more gold passages than the 560M bge-large. Fusing in ColBERT adds 0.5 to 1.2 points; three-way fusion and score fusion did worse than two-model rank fusion.

Truncate

At α = 0.03 the pipeline averages 2,891 tokens across both datasets and has the best worst-case recall of any single setting. Fixed budgets fill up completely on LongMemEval-S and lose variety across sessions. Pruning on cumulative score cost 2.3 points, and diversity reranking failed because the passage embeddings were almost indistinguishable (a 0.008 cosine gap).

Numbers from different protocols are not comparable. Read each table on its own.

• One setting works for both benchmarks. With α = 0.03, a 4,000-word cap and the top 60 passages, the threshold spends fewer tokens on short conversations and more on long ones.
• The compilation bottleneck shows up at both scales. On LoCoMo, 98.6% of gold evidence is retrieved but only 22.5% survives truncation without good ranking.
• Going index-free costs 0.9 points on LoCoMo and gains 0.8 on LongMemEval-S.
• Open-ended questions score 95.4% to 96.7%, more than 20 points above other systems. Raw passages keep the texture of the conversation that consolidation throws away.
• Temporal reasoning is the gap: 80.2% against 89.7% for EverMemOS on LoCoMo, and 82.7% against 89.5% for Memora on LongMemEval-S. Expansion lifts temporal questions on LongMemEval-S from 67.7% to 80.5%.

• Deduplicating passages, using the embeddings already computed during ranking.
• Compressing context. Formatting changes alone cut about 10% of tokens with no gold evidence lost. Pruning conditioned on the question is still unexplored.

Benchmark limits

• LoCoMo dialogues are generated by an LLM and may be richer in keywords than real conversations. LongMemEval-S is human-written but follows structured session formats.
• Both benchmarks use consistent character names, which flatters entity extraction.
• Histories run from about 9,000 to 115,000 tokens. Substring matching may not hold on much larger corpora.
• 97% of LoCoMo oracle traces need one hop. The hop mix on LongMemEval-S was not measured.
• Both benchmarks are English only.

Why cross-paper numbers disagree

The judge model alone can move accuracy by 5 to 10 points, and the judge prompt by 8 to 15%. Papers use different metrics, often unstated, and different question sets, from about 600 to 1,540 questions. That is why LoCoMo results here are reported under two protocols and LongMemEval-S under one shared setup.

A deterministic search loop with one learned ranking step reaches 91.9% on LoCoMo (93.5% under the aligned protocol) and 88.4% on LongMemEval-S, at about 650 ms per query on CPU. The results push back on four common assumptions:

• That retrieval needs an LLM in the loop.
• That multi-hop questions need complex machinery.
• That retrieval quality is the bottleneck.
• That a precomputed index is required.

A. Full ablation, key rows

Pipeline tweaks before reranking add at most 1.6 points and do not stack. The full 7.2-point gain comes from reranker quality.

B. Cross-encoder comparison

Combining mxbai-large-v1 with ColBERT recovers 92.2%. The DeBERTaV3 models seem to handle references between turns better. Latency measured single-threaded on CPU.

C. Offline estimates versus live runs

Simulating gains offline from gold passages overestimated fusion changes: three-way fusion was predicted at +1.8 points and delivered +0.4; score fusion was predicted at +2.7 and delivered +0.2. For a straight model swap, the estimate matched the live result (+1.2 both). Offline simulation is reliable for choosing a model, not for comparing fusion strategies.

D. Where the remaining errors come from

125 errors out of 1,540 LoCoMo questions (1,415 correct)

• Answer model got it wrong59%
• Ranking or budget dropped the evidence24%
• Search missed the evidence12%
• No evidence in the conversation5%

At the baseline, retrieval caused 67% of failures. With the final ranker, most remaining errors come from the answer model, not from search.