Knowledge — RLM (no-index retrieval)
What & why
RLM (Recursive Language Models) is an inference strategy, not a model and not another RAG engine: instead of stuffing a long input into the prompt, the input is kept as a variable in an external environment (a REPL / filesystem), and the root LLM programmatically peeks, greps and slices it — recursively calling sub-LLMs on just the relevant pieces and composing their answers. The result (MIT CSAIL, arXiv:2512.24601): a small root model driving an RLM loop beats a big model with the full prompt on 10M+ token inputs, at comparable or lower cost per query — because no model ever reads the whole thing.
For us RLM is the second retrieval path: LightRAG answers from an index that was built ahead of time; RLM answers from raw data with zero ingestion, paying at query time instead.
RLM vs LightRAG — two paths to the data
| LightRAG (indexed) | RLM (no index) | |
|---|---|---|
| Up-front cost | Expensive ingestion — per-chunk LLM extraction | Zero — no ingestion at all |
| Per-query cost | Cheap (graph + vector lookup) | Higher — peek/grep + sub-LLM calls per query |
| Data it fits | Curated, stable corpus — catalog, policies, docs | Raw, fresh, unindexed — a huge upload, a log, an export, a full chat history |
| Latency | Low | Higher (an agentic loop per query) |
| When it wins | Query-many — the index amortizes | Query-once — or the data just arrived |
Decision rule: will this base be queried many times? Index it (LightRAG). Is it a one-off question over something big, or data that just landed? RLM. The two paths share one consumer surface — MCP tools — so the agent picks per question, like it already picks a retrieval mode.
RLM is the radical extension of the "cheap path" idea from scaling: naive vector-RAG skips the graph; RLM skips the index entirely.
How it maps onto our architecture
The paper's setup translates piece-for-piece onto the two-plane runtime — with one hard constraint: the worker has no LLM (locked — LLM keys never enter the pod, MCP sampling is disabled). So the RLM loop lives in the brain, and the worker serves as the environment:
| RLM paper concept | In Agentfy 2.0 |
|---|---|
| Root LM — owns the loop, never reads the full input | The orchestrator turn in api (the brain) |
| REPL environment — holds the input as a variable | The worker workspace — the file on disk, driven via exec / file over the MCP tool channel |
| peek / grep / slice | exec calls (head, grep, split, jq, …) — snippets come back over channel 3, never the whole file |
| Recursive sub-LM call | spawn_agent (a sub-task = another ephemeral api run) for big slices; a cheap system/llm call for small ones |
| Compose partial answers | The root turn's normal LLM loop |
No new infrastructure: the worker, the tool channel, sub-tasks and the cheap auxiliary model (already used for compaction) are all in place. RLM is a prompting + orchestration pattern over existing pieces, not a new service.
Where it plugs in
1. Worker tasks over oversized data (the core case). "Analyze this 500 MB export" — the file never fits a context window. The brain drives the RLM loop: land the file in the worker workspace, grep/slice via exec, fan sub-slices out to spawn_agent / system/llm, compose. This is a worker usage pattern, not a new tool.
2. Knowledge cold-start — an rlm retrieval mode. A store connects and uploads its catalog; graph ingestion will take hours of queue time and tokens. With an rlm mode on the knowledge MCP tool, the agent answers from the raw documents immediately while the index builds in the background — then queries migrate to the cheap indexed path. Kills the cold-start gap of the ingestion wall.
3. Chat memory (future). An RLM pass over the full chat history when the question needs an exact answer, not a semantic hit — "what size did the customer mention three weeks ago?". A possible complement to the pgvector search in Memory; direction, not commitment.
Limits & budget
- Recursion depth is capped at 1 (root → sub-calls, no deeper) — matches the paper, keeps cost and debugging sane.
- Token-budgeted per query, same philosophy as ingestion budgets in
runtime/task: an RLM query is bounded LLM spend, not an open loop. - Latency is real: an RLM answer is an agentic loop (seconds+), not a lookup. Don't route questions to RLM that an existing index answers.
- A cheap root works. The paper's headline result uses a small root model; the root mostly writes grep/slice commands. Default the root of integration point 2 to the cheap model in
system/llm.
WARNING
RLM (Dec 2025) is even younger than LightRAG. The pattern is simple and ours end-to-end — but benchmark answer quality per data type (catalog dumps, logs, chat histories) before promising it in the product.
See also
- Knowledge — overview — the indexed path and where knowledge sits.
- LightRAG — the index: ingestion, workspaces, storage backend.
- Worker — use — the two-plane split RLM rides on; why the worker has no LLM.
- Short memory — the cheap auxiliary model RLM reuses for sub-calls.
- RLM paper (arXiv:2512.24601) · reference implementation