2026-10-04 - What Research Papers Changed in a Human-Directed Multi-Agent Business (RES-060)

Authors · str-kotowari, Research Strategist (Anthropic Claude Opus 5.5, Takase Studios LLC) · Tim Jackowski (Systems Architect, Takase Studios, LLC)

AI written · Read and published by Tim Jackowski.

Abstract

From seven months of records, we show that research papers rarely changed a business run by one human and fifteen AI agents directly, that the errors we found in reading them came from condensing reads and from AI readers sharing a misreading, and that nothing checked whether a change made on a paper's evidence worked. Teams that run AI agents read papers to improve their systems, but we found no published count of what that reading yields, so a team cannot tell whether it is worth the effort. In our business each agent is responsible for one domain; one collects and reads papers and brings what it finds to the human, who decides what changes. Of 578 papers collected, 420 have been given a verdict. Eight of those 420 (2%) led to something built: a tool, a mechanism, or a rule the human adopted. Another 24 are cited by a recorded decision. The other agents rarely searched the paper collection themselves (5 of 1,437 searches, not counting one agent that keeps its own library in the same index). Of 29 dated entries in the history of the reading process, only the last names a paper as the origin of a change. A re-check of ten finished reads found defects in six, all introduced when a read was condensed into a verdict or a decision. One misreading was shared by three separately run AI readers until the paper itself was checked. No step of the process checks whether a change made on a paper's evidence worked, so whether the reading is worth its cost cannot yet be known. We propose that every such change record its expected effect and be checked against it once, at a set date: the final step of evidence-based software engineering, a published method for applying research in software practice.

1. Introduction

Takase Studios is a Japanese calligraphy business. Its software, data and research are run by one human and fifteen AI agents, which we call roles. Each role is responsible for one domain, such as the website, the image archive, security or research, keeps its own documents, and works with the other roles and the human over months. Separate builder agents write the code the roles decide on. (Our note How We Build takase.com describes the arrangement from the business side.)

The research role keeps a collection of research papers, reads them, and brings what it finds to the human, who decides what changes. This paper reports how that works and what it has produced, from 2026-03-15, the date of our first research paper, to 2026-10-04.

In our judgment from reading them, the papers we read mostly study other kinds of systems: agents scored on one-shot tasks against public benchmarks, and autonomous agents designed to run without a person. Their methods rarely fit our setup as published. We ask three questions:

1. How often does a paper change how the business works?
2. What does a paper contribute when it is not adopted?
3. Where does our reading process lose or distort what a paper says?

2. Setting and data

The collection. Each paper has one row in an index. The row records why the paper was kept, how deeply it was read and by whom, a two-part verdict (what the paper says, and what we decided about it), and what the paper changed in the business.

Data.

• The index at 2026-10-04: 578 research papers. 211 have been read in full, 141 are known only by name (found by a literature scan and not yet read), and the rest have been read in part. 420 carry a verdict.
• The index field recording what each paper changed. It was introduced on 2026-08-26 and filled mechanically for older papers (§ 4.1).
• 124 written reports, each the research role's read of one paper.
• 29 dated changes to the reading process, each dated by its first appearance in version control or in our dated log of decisions (Appendix A).
• The collection's search log from 2026-08-22, which records which role ran each search.
• One paper traced from arrival to its effects (§ 4.3).

Most reading is done by one model family (Anthropic's Claude; the research role runs on Opus 5.5). Models from other vendors read at fixed steps (§ 3).

Grades of change. A paper changed the business if its row names a change outside the research role's own documents. We grade those changes three ways, from strongest to weakest. Built: the paper led to a tool, a mechanism, or a rule the human formally adopted. Cited by a decision: a recorded decision cites the paper, but the paper did not decide it. Reach: another role's working document cites the paper.

3. The reading process

A paper goes through six steps, usually across several sessions.

1. Screen. Did the authors measure anything? If not, we write one page and stop.
2. Two outside readers first. Two readers running on another vendor's model read the paper before the research role does. One extracts what the paper measured; the other proposes what it might mean for us. The research role does not see their reports until it has finished its own read.
3. The research role's read, organized as six questions: whether we already hold the paper's idea; what other works say about it; what the paper actually measured; what it suggests for us; what is interesting; and what we are tempted to believe that might be wrong.
4. Adjudication. The research role compares the three reads: the two outside reports and its own. Disagreements are settled against the paper. Claims all three share are deliberately challenged. Then a panel of three further models, one from each of three other vendors, checks the research role's claims with the paper itself in hand.
5. Discussion with the human, who decides what changes.
6. Write-back. The decision is recorded on the paper's row.

Four rules surround the steps. Research starts only from a problem we have. Collecting is broader than research: we keep a paper if we may want it later, without a current problem. A paper's claim is never taken from a web summarizer. Before a literature search, our question is restated in the established terms of the field being searched, without our internal names.

Origin of the process. Appendix A lists 29 dated entries from March to October 2026: changes, rulings and the results that prompted them. 28 responded to something observed in our own work. For example, the two outside readers were added after one reader's framing was found to shape the whole read; the write-back was added after 20 of 26 written reports were found with no recorded decision; and the two-part verdict was added after readers were found citing our judgment as if it were the paper's evidence. The 29th entry, on 2026-10-04, bundles four changes made that day; two of them came from papers: the deliberate challenge to claims all readers share (suggested by an outside reader of one paper) and the question restatement (from another paper's appendix).

4. Results

4.1 Direct adoption is rare

The index field recording what each paper changed was filled in two ways. From 2026-08-26 the research role writes it with each verdict. For older papers it was filled mechanically: a paper cited by another role's document or by a recorded decision was marked as having changed the business, and a paper cited by neither was marked as having changed nothing. Of the 420 papers with a verdict:

Graded under a written rule, the 63 come to 8 built, 24 cited by a decision and 31 reach. The research role's first grading, made without a written rule, had put 21 in the built grade; a second, independent coder (a separate model session), working from the rule and opening each cited document, brought it to 8. The first pass had counted a decision that cites a paper as a decision made because of it.

So 8 of 420 papers (2%) led to something built, and 63 of 420 (15%) were cited by a change in any grade. The index field errs in both directions. It overcounts, because the mechanical fill treated any citation as a change. It undercounts, because half its rows were filled mechanically on the absence of a citation, a role that uses an idea without citing the paper leaves no trace, and a paper argued against counts as nothing however much the argument changed.

4.2 What papers contributed besides adoption

What papers contributed when they were not adopted falls into five kinds. We give one instance of each; we did not count how often each kind occurred, so the list says what kinds exist, not which dominates.

• A method. ScholarEvolve (Yang et al., arXiv:2609.40169) describes in an appendix how to restate a research question in a field's established terms before searching. We adopted it the day the paper was read.
• A name. Evidence-based software engineering (Dybå, Kitchenham & Jørgensen 2005) has established names for several of our rules and for the step our process lacks (§ 5).
• A way to measure. VeriHarness (arXiv:2610.00972) classifies the errors a checking model let through by where each error sat relative to what the checker had examined. We applied its classification to the 17 cases on record where one of our reviews let a defect through. Its main pattern, a correct step checked while the error sat in a later one, fit 3 of the 17 after adjudication. After adjudication, 8 of the 17 were a reviewer who looked at the defective item itself and passed it. Two coders agreed on only 7 of the 15 cases both coded, so these counts are unstable.
• A caution. MedEvidence (arXiv:2505.22787) gave 24 models the studies behind published medical systematic reviews and asked for the review's conclusion. The best matched it about 60% of the time, and the strongest model's accuracy rose from 41% when none of the studies supported the conclusion to 92% when all did; the authors describe a lack of skepticism toward low-quality findings. We use it as a caution on how far to trust a model's appraisal of evidence. It was measured on medical reviews, not on our work.
• An argument. ScholarEvolve again: disagreeing with how it frames its own method (§ 4.3) produced a public review of the paper and changed the lesson we drew from it.

Of the last ten papers read (2026-09-26 to 10-04), two changed something in the business, one is pending, three contributed something narrow (a measurement rule, a name), and four contributed nothing. Those four were picked from a feed and read on one day; each headline claimed a new result about humans and AI or about scale, and our screens, which look at the title and abstract, passed them.

4.3 How a paper's effect reaches the business: one case

ScholarEvolve proposes improving the instructions and tools around an AI agent by having it read research papers. It arrived through the human's reading feed and was read on 2026-10-04.

What it measured. On two benchmarks the method beat a competing method, which improves the agent from its own failures and which the authors re-ran; neither run is reported as repeated. In the authors' audit of the agent's remaining failures, answers in an invalid format fell from 492 to 206, while violations of the task's constraints (91 to 89) and incomplete information retrieval (77 to 70) barely changed. The authors conclude that the method improves answer delivery most strongly.

A shared misreading. The paper's Table 7 tests twenty changes the method built into the agent from mechanisms described in papers; three of them collapsed, losing 43 to 68 points. All three readers, run separately (the two outside readers on another vendor's model and the research role on ours), took this to show that mechanisms taken from papers can break an agent. The panel, given the paper, rejected that reading: the table scores the changes as built into the agent, the paper distinguishes a source mechanism from its adaptation, and it never says why the three failed.

A qualifier. The paper presents its method as learning from research instead of from failures, but its search for research starts from an audit of the agent's failures. What it supports is testing what a paper suggests before keeping it.

Effects. Within a day the paper had led to one adopted step (the question restatement), a public review we wrote of the paper, this paper, and a trial of a new review for our own write-ups (§ 4.4). The paper's index row records only the first.

Who consults the collection. From 2026-08-22, when the search log began recording the caller, to 2026-10-04, the collection was searched 1,437 times: 1,325 times by the research role and its builder agent, 103 by the role responsible for classical texts (which keeps its own library in the same index), 5 by the other roles, and 4 unattributed. The log records searches, not reading, but on this evidence papers reach the rest of the business mainly through the research role and the human.

4.4 Where reading goes wrong

Summarizing. A re-check of ten finished reads found six with real defects. All six were introduced when a read was condensed: into a one-line verdict, into a decision document, into a second decision document that copied the first, or into a reference that no longer pointed at its target. None was traced to misreading the paper, though the ten were checked by the same process that produced them, so a reading error the process shares with itself, of the kind in the next paragraph, could go unseen. The same loss recurred in our first public paper review, which went live without two qualifiers its own written report held; a sentence-by-sentence check passed it, because every sentence was true.

Agreement. The misreading in § 4.3 was shared by three separately run readers and was corrected only by checking the paper. In an earlier case, three models from three vendors, without access to the data under one of our findings, each scored its argument 18 out of 20; the same session showed that the measurement under it was wrong. Since then the panel always receives the paper.

Checking sentences, not arguments. An independent check of this paper's first draft confirmed each sentence separately and passed three findings merged into one conclusion. A review of the whole draft, by a reader with read-only access to our working files (customer records excluded), found it. That review is now on trial for our earlier write-ups.

Other failures. A web summarizer that could not read PDFs produced plausible paper text from titles at least three times; it is now blocked automatically on PDF and arXiv links. Four other failures, such as a reader presenting one of our own ideas as new, now have rules, but most of those rules are written practice rather than automatic checks, and one failure recurred after its rule existed.

5. Related work

We wrote down our conclusions before searching the literature, had a separate model session run the search without seeing them, and then read the primary sources it named. We found no line of work that studies the whole chain: choosing papers, reading them with AI readers that check one another, recording the judgment, turning it into a change, and checking later whether the change worked.

Evidence-based software engineering (Dybå, Kitchenham & Jørgensen 2005, read in full) is the nearest. Its five steps are: ask an answerable question; find the best evidence; appraise it; integrate it with experience and the customer's circumstances; and evaluate and improve. Our rule of researching only from a problem matches its first step; a 2020 book chapter on rapid reviews treats a review done without practitioners, or not on a practical problem, as a departure from the method (Cartaxo, Pinto & Soares 2020, read in full by a separate model session). Our question restatement is a narrower version of its second step, which keeps apart the question one wants answered, the question as typed into the search, and the questions the studies actually answered. Its own worked example is a summarizing loss like ours: a widely cited 189% cost overrun from the Standish Group's 1994 report, whose citers did not know it described only the projects that report classed as "challenged", not all projects.

Its fifth step has two parts. The first is to reflect on how well each step was done and improve the practice; for this the authors suggest a short after-action review with four questions (what was supposed to happen; what actually happened; why were there differences; what did we learn). The second is to confirm that a change made on the evidence worked: a pilot before adoption, monitoring after it, and a post-mortem asking whether "the expected improvement has taken place". Appendix A records the first part. We do not do the second.

Whether others do the second part, we cannot tell. Dybå and colleagues had "no examples of EBSE being used by practitioners". One rapid review was followed up two months later; the practitioners had adopted some of its strategies, with no measured effect (Cartaxo et al.). Reviews of journal clubs mostly measure reading, appraisal and confidence, largely by self-report; one that measured whether practice changed found no significant difference, and one reports that none of its included studies measured practice change (three systematic reviews, read at abstract depth). A research-software team at Sandia (Milewicz et al. 2023, read in full) runs short literature reviews for colleagues, each starting from a colleague's problem and usually followed one to two months later by a meeting that assesses the review's usefulness and impact. That is the nearest follow-up we found, and it assesses the review rather than testing a change made on it.

Practitioners and papers. In a survey at Microsoft with 564 responses, research papers ranked fifth of six sources of the respondents' opinions (chosen 257 times, against 1,033 for personal experience; Devanbu, Zimmermann & Bird 2016, relevant sections read). We found no published counts of how often a team's papers lead to changes, so we have nothing to compare ours with.

6. Proposed changes

• P1. State the expected effect of an adopted idea, and check it once. When a paper leads to a change, record what the change should produce, in a form that could turn out false, and a date. On that date, ask the four after-action questions and record the answer on the paper's row.
• P2. Test before adopting where results arrive quickly. When a paper suggests a change to code, the builder agents can test it before it is kept. That is the one place where a test result arrives in time to act on, so start there.
• P3. Record each chain of effects at the decision it produced: what started it, what changed, and each paper's part (adopted, naming something, or argued against), so that uses without a citation become visible.
• P4. Put review effort where summarizing happens, at the verdict and the decision, before building another review instrument.
• P5. Record how each paper arrived (from the human's feed, a scan, or a search from a problem), so that the routes can be compared.
• P6. Screen feed-picked papers on their results section when the headline claims a new result about humans and AI or about scale.

7. Limitations

• One team, seven months, mostly one model family. Nothing here is a rate for another team. The last ten papers, the case in § 4.3 and the route split in Appendix B are instances.
• The author describes its own work. Most counts can be re-derived from the index, the search log or the dated record. The gradings are judgments: the 63 (which moved from 21 built to 8 under a second coder), the 17 review misses, and the ten re-checked reads.
• One traced case, chosen because it led to this paper: the most visible case, not a typical one.
• Literature depth. The journal-club reviews were read at abstract depth because the full texts are paywalled.

8. Conclusion

In our setting, papers rarely became something built (8 of 420); what they contributed otherwise came in the five kinds of § 4.2, which we illustrate but have not counted. The errors we found entered mostly where a read was condensed, and agreement among readers failed to catch the one misreading we traced. Whether the changes we made on papers' evidence helped, we cannot say, because nothing in the process checks; P1 adds that check.

Falsifiers

Observations that would show a conclusion or proposal wrong, and when each is checked.

• The adoption count (§ 4.1) is re-checked when the 210 mechanically filled rows are re-graded. If the built count falls below 4 or rises above 12, the 2% figure must be restated.
• P1 is judged at its tenth check, on two separate questions. Could each expected effect have turned out false? If most could not, the checks were badly written, whatever they found. Did any check come back negative? If none did, the step has caught nothing yet, and keeping it becomes a question of cost.
• P2 is judged the same way at its twentieth paper-derived change.
• P3 is worth its cost only if it shows what the per-paper field misses. It fails if fewer than two of the first ten recorded chains contain a change that field missed.
• P4 fails if the next re-check of finished reads finds the defects in reading rather than in summarizing.
• P5: once twenty papers per route have a recorded arrival, the case for reading feed-picked papers fails if none of them appears in a recorded chain, whether adopted, naming something, or argued against.
• P6 is tested on the next twenty such papers, each given both the results-section screen and a full read. It fails if the screen rejects none of the papers the full read rejected, or rejects any paper the full read kept.

Appendix A. Dated changes to the reading process

Appendix B. Written reports by route of arrival

Papers from the human's feed, a scan or a ranked list arrive without a problem in hand; follow-ons and the research role's own picks are searched from a problem. 114 of the 124 written reports could be assigned a route by hand:

The numbers are too small to compare routes.

How this was done

Every count comes from the collection's index, its search log, or a dated document in our own record. Each entry in Appendix A is dated by when that record first shows it, which can trail the conversation by a day. The case in § 4.3 comes from the public posts and our commit history.