2026-03-15 - RES-002: OpenClaw-RL — Directive Signals in Conversations

Citation

Yinjie Wang, Xuyang Chen, et al. "OpenClaw-RL: Train Any Agent Simply by Talking." arXiv: 2603.10165, March 2026.

Key Findings

• Agent usage generates data — the paper's examples are user replies, tool execution results, and GUI state transitions — that the authors describe as "a valuable source of insights," while noting that the infrastructure and methodology for using it in real time "remain little explored."
• The framework extracts both evaluative (how well did it work?) and directive (how should it change?) information from the "next state" after an action. The evaluative signal comes from a process reward model that implicitly scores the preceding action; the directive signal is the corrective content itself — their example is a user saying "you should have checked the file first," which specifies not just that the answer was wrong but how it should change.
• Not all of these signals are human. Two of the paper's three evaluative examples are machine signals: "a user re-query signals dissatisfaction, a passing test signals success, and an error trace signals failure."
• The specific gap the paper addresses is narrower than "nobody uses corrections." RLVR methods are limited to scalar rewards and so "cannot convert directive signals into policy gradients"; hindsight-relabeling and on-policy-distillation methods do use corrective information, but "these methods all operate on fixed datasets." The contribution is doing it online, against a live agent.

A correction to this note, 2026-07-31. This section previously said the signals were "universal gold for training but typically ignored," and that current systems "discard" them — phrasing that appeared here for four months. Neither is the paper's. On reading the primary end to end, the words gold and universal appear nowhere in it, and discard appears once, about an unrelated algorithmic property. The paper's own introduction names methods that use exactly this data. The original bullets were written from the abstract; our research ledger recorded that at the time and the note did not. The architectural argument below is unaffected — it never depended on the claim that the field ignores corrections — but the borrowed authority did not survive the read, and removing it is the point of keeping a ledger.

The Core Argument

Human corrections in conversations are the highest-quality training signal available — and current systems throw it away.

Why It Matters to Us

This paper describes, in RL terms, a signal our flywheel (ADR-047) and the docs-meta initiative already capture by hand:

1. The flywheel IS manual OpenClaw-RL. Every time Tim corrects a strategist — "no, the NAS roles are reversed," "read the spec before asserting facts" — that correction gets written into session states, concept-checks, lessons, and memory files. We don't feed it back into model weights (we can't), but we persist it in documentation so the next session inherits the correction. The paper says this signal is gold; we already treat it as gold.

2. The conversation archive is an asset, not history. The JSONL conversation archive contains thousands of directive signals — corrections, re-queries, "no not that, instead do..." moments. With 1M context, old sessions can be fully re-read. This is the raw correction data the paper says current systems waste.

3. docs-meta is the retrieval infrastructure. The conversation archive is locked in JSONL files that nobody can search. docs-meta (the planned vault — gitignored, invisible to engineer roles, curated session index) makes this data findable and usable. RES-002 is the research context that informed building docs-meta.

4. The compounding is real. At the time of writing: 143 str-takase sessions, 83 str-ishizue, 41 str-mamori — each session's corrections compounding into the next (every count higher today). The project's documentation IS the learned model — not weights in a neural network, but structured knowledge in files. Same function, different substrate.

Empirical Confirmation (RES-008, 2026-03-26)

RES-008 used the conversation archive this paper justified building. str-michi searched 121 JSONL session files, extracted Tim's corrections (including profanity as a frustration marker), computed correction rates, and discovered that 83% of AI strategist failures involved documented constraints that weren't consulted. The corrections — the "directive signals" this paper says systems throw away — became the dataset that diagnosed the failure mode.

The prediction that correction data is "universal gold" was confirmed: Tim's profanity turned out to be a precise instrument for measuring system failure rates across 2,371 human messages. The rawest form of human correction was the most analytically useful.

Discovered 2026-03-14. Part of a pair: RES-002 (why the flywheel works) → RES-003 (why docs-meta is worth building).