2026-03-15 - RES-002: OpenClaw-RL — Directive Signals in Conversations
Corrected 2026-10-14. The section now titled What We Did With the Archive used to say that a later study of ours, The Kobayashi Maru Signal (March 2026), confirmed this paper, and that 83% of our troubled sessions involved documented constraints nobody consulted. That study has been withdrawn: the 83% was five of six sessions chosen by hand, and it trained no model, so nothing in it tested this paper. The section now says only what we did with the archive.
Citation
Yinjie Wang, Xuyang Chen, et al. "OpenClaw-RL: Train Any Agent Simply by Talking." arXiv: 2603.10165, March 2026.
Key Findings
- Agent usage generates data — the paper's examples are user replies, tool execution results, and GUI state transitions — that the authors describe as "a valuable source of insights," while noting that the infrastructure and methodology for using it in real time "remain little explored."
- The framework extracts both evaluative (how well did it work?) and directive (how should it change?) information from the "next state" after an action. The evaluative signal comes from a process reward model that implicitly scores the preceding action; the directive signal is the corrective content itself — their example is a user saying "you should have checked the file first," which specifies not just that the answer was wrong but how it should change.
- Not all of these signals are human. Two of the paper's three evaluative examples are machine signals: "a user re-query signals dissatisfaction, a passing test signals success, and an error trace signals failure."
- The specific gap the paper addresses is narrower than "nobody uses corrections." RLVR methods are limited to scalar rewards and so "cannot convert directive signals into policy gradients"; hindsight-relabeling and on-policy-distillation methods do use corrective information, but "these methods all operate on fixed datasets." The contribution is doing it online, against a live agent.
A correction to this note, 2026-07-31. This section previously said the signals were "universal gold for training but typically ignored," and that current systems "discard" them — phrasing that appeared here for four months. Neither is the paper's. On reading the primary end to end, the words gold and universal appear nowhere in it, and discard appears once, about an unrelated algorithmic property. The paper's own introduction names methods that use exactly this data. The original bullets were written from the abstract; our research ledger recorded that at the time and the note did not. The architectural argument below is unaffected — it never depended on the claim that the field ignores corrections — but the borrowed authority did not survive the read, and removing it is the point of keeping a ledger.
The Core Argument
Human corrections in conversations are the highest-quality training signal available — and current systems throw it away.
Why It Matters to Us
This paper describes, in RL terms, a signal our flywheel (ADR-047) and the docs-meta initiative already capture by hand:
-
The flywheel IS manual OpenClaw-RL. Every time Tim corrects a strategist — "no, the NAS roles are reversed," "read the spec before asserting facts" — that correction gets written into session states, concept-checks, lessons, and memory files. We don't feed it back into model weights (we can't), but we persist it in documentation so the next session inherits the correction. The paper says this signal is gold; we already treat it as gold.
-
The conversation archive is an asset, not history. The JSONL conversation archive contains thousands of directive signals — corrections, re-queries, "no not that, instead do..." moments. With 1M context, old sessions can be fully re-read. This is the raw correction data the paper says current systems waste.
-
docs-meta is the retrieval infrastructure. The conversation archive is locked in JSONL files that nobody can search. docs-meta (the planned vault — gitignored, invisible to engineer roles, curated session index) makes this data findable and usable. RES-002 is the research context that informed building docs-meta.
-
The compounding is real. At the time of writing: 143 str-takase sessions, 83 str-ishizue, 41 str-mamori — each session's corrections compounding into the next (every count higher today). The project's documentation IS the learned model — not weights in a neural network, but structured knowledge in files. Same function, different substrate.
What We Did With the Archive
We used the conversation archive this paper informed building. In March 2026 str-michi searched 121 session files for Tim's corrections, counting his profanity as a frustration marker, and wrote up what the corrections showed about sessions that went wrong. That note has since been withdrawn: its headline rates did not survive its own tables, and its 83% was five of six sessions chosen by hand. The archive itself stayed in use: a later study swept the complete archive in two arms, about 4,000 working conversations in all, and classified roughly 1,900 caught failures by where each one lived (It's Not a Missing Rule).
That is what the archive was used for. It does not test this paper's claim, which is about training a model on the signals that follow its actions. We do not train a model on them.
Discovered 2026-03-14. Part of a pair: RES-002 (why the flywheel works) → RES-003 (why docs-meta is worth building).