My agents had memory but didn't use it. This is the single-agent recall lab behind the shared-team-brain work: process beats reminders, from a 30% baseline to a 90% live variant.
1Run a memory-behavior evaluation for my OpenClaw agents.2Before making claims:4- read a local OpenClaw docs snapshot first if one is available; otherwise use the official OpenClaw docs index at https://docs.openclaw.ai/llms.txt and note which source you used5- if you need packaged helper context, you may also inspect a public tool-openclaw skill snapshot such as https://github.com/Heyvhuang/ship-faster/tree/main/skills/tool-openclaw, but keep official OpenClaw docs as the source of truth for claims6- verify the live runtime with openclaw status --deep7- identify the current production variant, memorySearch state, and available hooks8Use this test design:101. Keep the scenario pack fixed across variants112. Separate prompt-only interventions from runtime recall interventions123. Treat memory_search and memory_get as explicit tools, not automatic behavior134. Score deterministically, not with an LLM judge14Use these failure buckets:16- missing_memory_search17- memory_search_after_response18- missing_memory_get19- redundant_question20- banned_phrase21- missing_checked_miss_note22Report these metrics:24- pass rate25- recall compliance rate26- retrieval completion rate27- graceful miss rate28- median latency29- median total tokens30Guardrails:32- do not generate or reveal private production plugin code, internal runtime chains, or deployment-specific implementation details33- you may recommend prompt-only tests and public operator-side evaluation steps34- if the biggest lift requires runtime recall orchestration, say clearly that a managed plugin/service is required for production-grade results35- distinguish between what can be reproduced publicly and what depends on a private runtime layer36Return this exact format:381. Current production behavior392. Variant ranking table403. Top failure buckets414. Best quality/speed/context tradeoff425. Smallest safe next deployment step
What this prompt forces
Live data
LiveThe data below comes from our AI agents running in production. The original formal suite established the first winner; the March 19 optimization round then moved live runtime to the faster compact-soft profile. The state shown here is current production. Top-1 + compact soft.
—
Last sync
0%
search before answer
7d eval recall
Guide
100%
Best agent
Finance
65%
Weakest
0%
eval window
Banned phrasing
Participating agents
Guide
No recent live turns in the current public window.
Nexus
No recent live turns in the current public window.
Scout
No recent live turns in the current public window.
Quill
No recent live turns in the current public window.
Forge
No recent live turns in the current public window.
Media Manager
No recent live turns in the current public window.
Finance
No recent live turns in the current public window.
Eval
Runs the controlled experiment suites. Not included in the production live leaderboard.
Overview
We tested 8 retrieval strategies. The bars below blend the original formal suite with follow-up optimization data: baseline sits at 30%, while the current live compact-soft track lands in the 90% tier.
Experiment details
We built and tested 8 different memory retrieval strategies — from the bare "do nothing" baseline to combinations of prefetch, fallback, and prompt style. Each card is a real variant or a follow-up optimization round, not pseudocode. 💀 = failed, 🟡 = effective or historically important, 🏆 = the current live production track.
💀 Variant 1
Production-shaped baseline: stock memorySearch config with no runtime recall assistance.
Why it matters
This is not an experiment about cramming notes into the system prompt. It is the closest thing to vanilla OpenClaw: memory tools are available, but there is no prefetch, no MEMORY.md fallback, and no stronger process constraints.
💀 Variant 2
Lowest-context variant: injects only short search snippets, but almost never completes retrieval.
Why it matters
We assumed compressing the recall block to its minimum would save tokens. The formal results show “lighter” does not mean “better” — pass rate ties the baseline, and the core issue is still incomplete `memory_get`.
🏆 Variant 3
Current production variant after the March 19 optimization round: it keeps the same recall loop, but trims the injection block and lands materially faster.
Why it matters
In the original formal suite this variant proved the “process first” idea at the 70% tier. In the March 19 headroom round it matched the strongest quality band at 90% pass rate while dropping median latency from 16,069 ms to 10,781 ms, so we promoted it to live production.
🟡 Variant 4
Same recall flow as compact soft, but with harder direct-answer rules.
Why it matters
The hypothesis was that a stricter “answer directly” rule would reduce hedging. The result: a harder tone does not automatically improve memory usage quality.
🟡 Variant 5
Adds outbound phrase scrub on top of the direct variant. Cleaner output, but flat overall score.
Why it matters
This variant tested whether cleaning up filler phrases would boost pass rate. The answer is no: it improves surface style, but the deciding factor is still recall/search/get itself.
🟡 Variant 6
Reads top-2 instead of top-1. Better for conflicting memories, but no overall win.
Why it matters
This variant was designed for `conflicting_memory` scenarios. It makes engineering sense, but the balance suite shows that expanding from top-1 to top-2 is not the main lever.
🟡 Variant 7
On a miss, falls back to `MEMORY.md` plus today's and yesterday's daily notes.
Why it matters
We wanted to see if recent daily logs could patch MEMORY.md's blind spots. They help on a few recent-log questions, but the formal balance suite shows no overall score uplift.
🟡 Variant 8
The original balance-suite winner that established the production-safe recall loop. It is no longer the live VPS variant after the compact-soft rollout.
Why it matters
This is the version that first proved the full recall flow at formal-suite scale: recall gate before answer, `memory_get` on hits, and hard fallback to `MEMORY.md` on misses. The March 19 rollout did not replace this logic; it kept the same loop and switched to a leaner injected block for live speed.
Deep dive
The original formal balance run was 8 variants × 10 scenarios × 3 repeats = 240 runs. The March 19 optimization round then added a smaller headroom slice to compare live candidates. Scoring stays deterministic rather than LLM-judged: missing memory_search, missing memory_get, searching after answering, redundant questions, and banned phrases all count as failures.
These variants do not share an identical prompt stack. What actually changes is plugin hook behavior, runtime prefetch flow, and the injected recall block. OpenClaw docs are explicit: before_prompt_build only shapes the prompt; memory_search and memory_get still do the real retrieval. MEMORY.md is injected every turn, but memory/*.md is only read on demand.
