2026-03-15 - RES-003: RLHI — Reinforcement Learning from Human Interaction
Corrected 2026-10-14. The section now titled What We Did With the Archive used to say that a later study of ours, The Kobayashi Maru Signal (March 2026), confirmed this paper, and that sessions where more than 15% of Tim's messages were corrections formed a distinct class. That study has been withdrawn: its tables disagreed with each other, it showed no distribution behind the "distinct class", and its headline percentage rested on six sessions chosen by hand. It built no preference pairs. The section now says only what we did with the archive.
Citation
Chuanyang Jin, Bo Liu, et al. "The Era of Real-World Human Interaction: RL from User Conversations." arXiv: 2509.25137, September 2025. Meta FAIR and Johns Hopkins University.
Key Findings
- Preference pairs from corrections: Every time a user corrects an AI response, that creates a preference pair — bad response (original) vs. good response (incorporating feedback). This signal is real but, as the paper puts it, historically hard to extract — so static training data gets collected instead.
- User personas from chat history: Chat patterns (tone preferences, format preferences, domain expertise) can be extracted to personalize future interactions. Noisy interactions are quality-filtered before training.
- Static datasets are textbooks: Disconnected from real conversations — "static, context-free judgments instead of evolving, situational demands."
- Results: Trained on WildChat (1M+ real ChatGPT conversations), the approach boosts personalization and instruction-following, and raises accuracy on math and science benchmarks. (Those reasoning gains used synthesized conversations simulating users correcting errors, not organic chat data.) The model is trained to condition on user personas extracted from history — which is not the same as a deployed model remembering you across live sessions.
Why It Matters to Us
This paper describes, in RL terms, what the docs-meta initiative captures by hand:
-
Our conversations hold the same kind of preference signal. Every "no, not that — instead do..." moment from Tim is the shape of preference pair this paper extracts: the AI's wrong answer + the corrected direction. We have hundreds of sessions of these across 10+ roles. RES-002 (OpenClaw-RL) describes why this data is valuable; RES-003 describes the mechanism — preference pairs + persona extraction. (The scale gap is large and worth stating plainly: RLHI trained on 1M+ conversations from many users; our corpus is hundreds of sessions spanning multiple roles, not one user — whether the mechanism transfers at our scale is an open question, not a settled result.)
-
We already build "user personas" manually. Session states, concept-checks, lessons files, memory entries — these are hand-built persona artifacts. "str-takase reads the spec before asserting facts" is a persona constraint. "str-mamori never guesses IP attribution" is a persona constraint. This paper says the same thing can be extracted automatically from conversation history.
-
docs-meta is the retrieval layer. The raw conversations (JSONL in
~/.claude/) contain the preference pairs. docs-meta (the planned vault) makes them searchable and extractable. Without docs-meta, the preference pairs are locked in unsearchable files — exactly the "thrown away" state this paper critiques. -
The compounding argument gets stronger. RES-001 says LLMs converge without human intervention. RES-002 says corrections are the most valuable signal. RES-003 says those corrections can build preference pairs AND user personas. Together: our human-shuttled correction system isn't a workaround for lacking agent-to-agent communication — it's the optimal architecture for generating the highest-quality improvement signal.
-
Future possibility. If model fine-tuning ever becomes accessible for our use case, the docs-meta archive would be the training corpus — hundreds of sessions of corrections, already tagged by domain (str-takase, str-ishizue, str-mamori, etc.). The infrastructure investment pays off regardless — useful now for context retrieval, and possibly for model adaptation later.
The Three-Paper Arc
| Paper | What it says | What it means for us |
|---|---|---|
| RES-001 (Hivemind) | LLMs converge without humans | Why we need HITM |
| RES-002 (OpenClaw-RL) | Corrections are gold, systems throw them away | Why the flywheel works |
| RES-003 (RLHI) | Corrections create preference pairs + user personas | Why docs-meta is worth building |
What We Did With the Archive
In March 2026 str-michi used the conversation archive (PLN-011) to study 121 sessions and 2,371 of Tim's messages, looking for sessions where an AI strategist kept proposing fixes against a constraint it had not understood. That note has since been withdrawn: its tables disagreed with each other, and its headline percentage rested on six sessions chosen by hand. What it shows is only that the archive (1,430+ text files at the time, JSONL with timestamps) could be searched for Tim's corrections.
Whether corrections like these make good preference pairs for training, which is what this paper studies, we did not test. We do not train a model on them.
Published September 2025, discovered 2026-03-15. Part of a pair: RES-002 (why the flywheel works) → RES-003 (why docs-meta is worth building).