Redis splits its docs agent into three entities after evals expose weak retrieval

Redis has described how it rebuilt the retrieval layer of an agent that answers questions from redis.io/docs, moving from a single document entity to three linked ones. According to the company, its evaluations showed stronger retrieval overall, but at the cost of more tool calls and slower tail latency.
- Redis moved its docs agent from one RedisioDoc entity to three: RedisioChunk, RedisioPage and RedisioCommand.
- A V1 trace for "What does BF.RESERVE do?" never surfaced the BF.RESERVE command page, returning Kubernetes and REST API pages instead.
- Redis built a standalone evaluation kit that scores retrieval traces against a golden dataset using nDCG, recall and hit rate.
- According to Redis, the new design improved retrieval overall but added more tool calls and slower tail latency.
A single entity for every kind of question
Redis wrote that documentation questions come in many shapes. "What does XACK do?" is a precise command question, while "How should I think about eviction policies?" is broader, and "Why is my stream consumer falling behind?" may span several pages, commands and ideas.
For its first version, the team used Context Retriever, which lets developers model their data like an ORM. An entity defines a schema, and Context Retriever exposes retrieval tools that an agent can call. Redis started with one entity, RedisioDoc, representing a documentation page, section or chunk of text. It carried a stable ID, title, topic, section, content, vector embedding and canonical URL.
The schema generated six tools, but the application exposed four to the agent: get by ID, search by text, search by content embedding similarity, and filter by topic. The filters for subtopic and version were not exposed. Redis calls this a useful baseline, because a small toolset was easier to inspect. But it flattened several retrieval jobs into one shape, so the agent had to sift through results itself whether it needed an overview, a command reference or a precise passage.
Judging the evidence, not just the answer
Redis says it first judged the system the way a user would: did the answer sound right, did it cite relevant pages, and how long did it take. That missed a problem. A strong foundation model can turn incomplete evidence into convincing prose by using general knowledge to fill gaps, so a fluent answer can sit on weak retrieval.
According to Redis, weak retrieval forces the model to read more irrelevant text, make more calls and lean on what it already knows. That costs latency and tokens and pushes teams toward a larger model. Good retrieval, the company argues, lets a smaller, faster model produce the same answer from evidence it can cite.
The team therefore built a standalone evaluation kit that can test different retrieval agents, MCP servers, APIs or saved traces against the same documentation corpus. An agent skill helps write a small adapter for each target. The adapter calls the system, parses the response, keeps the original result order and maps source identifiers to stable page IDs. A reusable harness then converts the output into a RetrievalTrace, removes duplicate pages without changing order, and compares the ranking with a golden dataset. Each run produces a trace.jsonl and a report.md with metrics such as nDCG, recall and hit rate.
The golden dataset supplies a question, the primary sources expected to answer it, and related sources. Questions fall into four groups:
- Simple lookups, such as "What does BITCOUNT do?"
- Multi-step questions, such as creating a Bloom filter, adding items and checking membership, which expects the BF.RESERVE, BF.ADD and BF.EXISTS pages.
- Hard cases, including troubleshooting and ambiguous terminology, such as a question about XACK not reducing stream memory.
- Chat-style questions, shorter and closer to production prompts, such as "How do I monitor Redis latency?"
The BF.RESERVE trace and the three-entity design
One trace made the weakness clear. Asked "What does BF.RESERVE do?", V1 searched the full documentation collection four times. Its first results included Kubernetes and REST API pages that happened to contain words like "reserve" or "reservation." The Bloom filter guide appeared only near the bottom of the first ten results, and the BF.RESERVE command page never appeared. Redis concluded that the documentation was not missing; the retriever simply had no direct path to the command as an entity.
In the next design, the documentation is split into three entities:
- RedisioChunk holds focused passages used as answer evidence.
- RedisioPage holds page summaries used for discovery.
- RedisioCommand holds structured command references for precise lookup.
Redis also declared explicit relationships between pages and chunks, and between pages and commands, so the agent can move from discovery to evidence. The source text provided ends while describing these relationships.
What it means for agent builders
The post argues that different retrieval tasks need different kinds of information: a short passage is useful evidence but may lack context, a page summary helps discovery but may be too broad to cite, and a command has a clear name and structure that general prose search can blur. The trade-off Redis reports is that the split design retrieved better but needed more tool calls and showed slower tail latency.
For teams building documentation agents, the practical lesson in Redis's account is to measure retrieval separately from the final answer, and to model distinct retrieval jobs as distinct entities rather than one broad document type.
