Fine Japanese Calligraphy

The Art of Master Japanese Calligrapher Eri Takase

Home › Library › Reading Research Papers for Ideas: Seven Months of Records from a Two-Person Business and Its Fifteen AI Agents (RES-060)

2026-10-04 - Reading Research Papers for Ideas

Corrected 2026-10-16. Several claims were stated more strongly than their sources allow and are now narrowed: a figure taken from a paper now says which test it came from, a score from one of our own reviews now says it was one of four measures, and our re-check of ten write-ups now says how they were chosen and that one defect came from the reading itself. A comparison of a drop in score now names what was measured, and a sentence about our own safeguards now says which rules we searched. A passage saying we had never checked whether a change made on a paper's idea worked now reports the one check we ran in October, and a proposal to add a search that starts from our own problems now says that such a search already exists. A cold read of these corrections then found a count in the October check that did not add up, which is fixed, and statements elsewhere that our losses all came after the reading, which now say all but one.

Abstract

The question. Takase Studios is a two-person Japanese calligraphy business. One of its two people runs its software, data and research with fifteen permanent AI roles, each the standing expert for one domain of the business. One of them, the research role, reads research papers and keeps a library of them with its notes on each. Almost nothing published describes a business run like ours, so we read papers about other kinds of AI systems for the one idea that helps with a problem of ours. We expect any small shop outside what the field studies to be in the same position, and we found no published measure of how often such reading changes an engineering or AI team's practice (§ 9). This paper reports what it has yielded for us, from our own records, March to October 2026, and which of our mistakes another shop could avoid.

How. From the library's records and search log, our dated log of decisions and our version history: the practice (§ 3), who uses the library (§ 4), the cases where an idea changed something (§ 5), and where the practice failed (§ 6).

What we found. We read a paper for the one piece of it that helps with a problem of ours, which is usually not what the paper is about; the paper as a whole describes a system unlike ours and does not fit. We keep the papers and our notes so that the piece can be found again, and an idea's effect shows in particular cases, not in a count per paper. - An idea's effect is a chain, and our record for each paper sees one link. One paper, dismissed at first sight, led within a day to an adopted step and a public review, and started the opening of this paper and a trial of a new review (§ 5.1). - What comes back is a name, a method, a way to measure, a concept, a caution or an argument; we have an instance of each, not a count (§ 5.2). - By design, other roles do not search the papers on AI models; they send their questions about models to the research role. Of 1,465 searches of the library, 1,353 were the research role's own (§ 4). - In our judgment the practice pays most when a role that has gone wrong sends its account of the failure to the research role for diagnosis. In both recorded cases a decision changed. They are the clearest cases of the work the research role was created for in May 2026, getting the roles to work as the human had hoped, which in his judgment they now do. This paper cannot separate the papers' share of that from the experiments and notes beside them, and one of the two cases used no paper (§ 5.3). - Of 29 dated entries in the history of the reading process, only the latest names a paper as its origin (§ 3.5). - The failures were ours, not the papers'. The losses we measured came, all but one, when a write-up of a paper was condensed into a verdict or a decision (six of ten re-checked write-ups, a sample chosen partly because corrections were expected), and three AI readers run separately shared one misreading until the paper was checked (§ 6).

What we would change. Write each case down where it happens, review the condensing step, and record whether papers arrive from our own problems as well as from the human's reading (§ 8). These are instances from one shop, not rates.

1. Who we are, and why we read papers that are not about us

Takase Studios has sold the Japanese calligraphy of Eri Takase, a master calligrapher, for thirty years. Eri brushes custom work for people and businesses. She has also brushed a library of words and phrases, prepared for download and used for tattoos, marketing and hobby work. For names, a software engine sets a name in kana from characters she brushed, so a print-ready design can be made for people she will never meet. Around all of it, the business publishes free reference work (a Japanese-English dictionary, calligraphy lessons, guides to names and their readings), and that scholarship gives the art its authority.

The business's software, data and research are run by one human and fifteen AI roles. Each role is the standing expert for one domain: security, research, the product data, the image archive, the live website, the classical texts behind the calligraphy, and others. The roles are permanent. None has an end date, and the oldest were created in late 2025. A role works in sessions, each a fresh conversation with the model, and starts every session from its harness, the material that carries it across sessions: its onboarding, its notes on the work in flight, and its record of decisions, the ones that held and the ones that failed. A role thinks, decides and reviews; a builder agent of its own writes the code it decides on. We are a two-person business and mean to stay one. Our premise, after thirty years, is that AI cannot replace the artist, or the experience and judgment of the people who run the business. What a team of AI roles can do is carry the work around them, the websites, the data and the research, at a scale two people could not reach alone. So the human stays in the middle of the work, because the judgment is his. (Our note How We Build takase.com describes the arrangement from the business side.)

Left to ourselves we work in a bubble: one human, one family of models doing most of the work, and our own names for practices we worked out by intuition and by trial and error over a year and a half of working with AI models, many of which the field had named long ago. The clearest instance came in April 2026. The human showed one role a conversation that used an established term for our kind of system, hierarchical multi-agent system, and within hours that role had connected our practices to six established lines of work, among them blackboard architectures, planner–executor designs and context engineering. Its training data had not changed. Our reading of the episode: we had worked out these practices without knowing the field's words for them, and once one was put in the field's own term, the model could draw on what its training held under that term.

The literature that could take us out of the bubble almost never describes a system like ours. It is mostly about other systems: agents scored on single tasks against public benchmarks, and agents designed to run without a person. So we read a paper for the piece of it that helps with a problem of ours, a name, a test, a method or a concept, and that piece is usually not what the paper is about. An idea fits our problem, or suggests something about it; a paper as a whole does not. In our judgment reading has become cheap enough, now that models do most of it, that this pays even when most of a paper has nothing to do with us.

Scale. Our research shelf of papers on models and methods holds 635 works, mostly papers (2026-10-06). 433 carry a verdict: our recorded judgment of what the work says and what we decided about it. Of the rest, 180 are works a literature search named and nobody has opened yet, kept as pointers to read when a question needs them, and 22 have been fetched and not yet read. That is the library's size; the paper does not treat it as a result.

2. What we needed ideas for

The reading makes sense only against the problems it served. Each of our research papers starts from a problem, so the list of them is the record of what we needed ideas for. Grouped by subject, they fall into six classes.

3. How we read and keep what we read

The practice has three parts: reading a paper, curating what it offered into the library, and finding it again when a problem arrives. (Internally we call the library the bunko, a Japanese word for a collection of books.) Nearly every rule in it came from our own record (§ 3.5).

3.1 How a paper arrives

Papers arrive through the human's reading, when he sees a paper and hands it over, or from a literature search that starts from a problem of ours. We kept no record of which way each came, so we cannot say which route is larger (§ 8.3). Either way, before any research act a use has to be named: a problem the paper meets, or a term of ours that needs checking for an existing name (ruled 2026-05-29). A paper whose authors measured nothing gets a one-page note and no full read (2026-09-02).

3.2 How it is read

  1. Two outside readers first. Two AI readers running on another company's model read the paper before the research role does: one extracts what the paper measured, the other proposes what it might mean for us. Their reports stay sealed until the research role has finished its own read.
  2. The research role's read, organized as six questions: whether we already hold the paper's idea; what other works say about it; what the paper actually measured; what it suggests for us; what is interesting; and what we are tempted to believe that might be wrong.
  3. Adjudication. The three reads are compared and disagreements settled against the paper. Claims all three readers share are deliberately challenged, because separate readers can share a misreading and their agreement would hide it (§ 6.2). A panel of three models from three other companies then checks the claims a decision will rest on, with the paper itself in hand.
  4. A stop and a discussion with the human, who decides what, if anything, changes. A read never turns directly into a change.
  5. Write-back. What the discussion decided is recorded on the paper's row.

3.3 How it is kept

Each work has one row in an index. The row records why the work was kept, how deeply it was read and by whom, a verdict in two parts (what the paper says, and what we decided about it), cautions for anyone who cites it, and what it changed. The split verdict exists because readers had begun citing our judgment as if it were the paper's evidence (2026-08-04). A weak paper can supply a usable idea, and must never then be cited as proof of it.

The ideas a read produces are written down as short numbered statements, one per idea, so that a search can hand each back on its own; we call these banked ideas, and the shelf holds 1,211 of them (2026-10-06). An idea that solves no problem in front of us is kept on the shelf and not turned into a task (2026-08-06); it comes back when a later search reaches it. Each new paper is also linked to the works already on the shelf that it touches, so that a later read can enrich an earlier entry (2026-08-25).

3.4 How it is found again

A meaning-based search runs over the rows and the full texts; a separate query runs over the banked ideas; and a verdict lookup runs before any paper is read, so finished work is not read again by accident. When a question arrives, the research role searches the shelf in both vocabularies, ours and the field's, before concluding that nothing there bears on it. Two instances of saved papers brought back at a later check are in § 4.

3.5 How the process got this way

Appendix A lists 29 dated entries from March to October 2026: the changes to the process, and the rulings and results that prompted them. None of the first 28 names a paper as its origin. Most answered something observed in our own work; a few were choices, such as founding the research role, publishing only on our own site, and pausing new papers while the collection was sorted out. For example, 20 of 26 written reads were found waiting at the discussion step with no decision recorded, so the write-back step was added (2026-07-31). One reader's framing was found to shape whole reads, so two outside readers now read first (2026-07-27/28). The 29th entry bundles four changes made on 2026-10-04, two of which came from papers: the challenge to claims all readers share (suggested by an outside reader of one paper) and the restatement of a question in the field's terms before a literature search (from the appendix of another).

4. Who the library serves

The library has several shelves: classical texts, kept by the role that studies the texts behind the calligraphy; marketing, kept by the role that works on how new customers find us; and models and methods, kept by the research role. Each shelf serves the role whose field it is.

The library's search log has recorded the caller since 2026-08-22. From then to 2026-10-05 the shelves were searched 1,465 times: 1,353 by the research role and its builder agent; 106 on the text and marketing shelves (103 by the texts role working its own shelf, three by two other roles); two by the content role, asking questions about software practice; and four lookups of a model paper's verdict that did not record who ran them. No other role's attributed search was a question about how models behave.

That is the design, not a gap. The log shows who searched; it does not show that questions arrive, and the two cases in § 5.3 are our evidence that they do. A question about how models behave comes to the research role, which answers it from the shelf, in the same way the texts role answers a question about a classical text from its own shelf and the research role does not search that one (ruled 2026-10-05). The model shelf reaches the rest of the business through the role that keeps it. Bound: this log covers the library's own search tools, not the general document search through which any role can reach a written read.

Why the model shelf exists: the training-data gap. Our models know this field the way training leaves it: up to a cutoff, and as a summary. A precise term lets a model reach what it already holds under that term, which is what happened in April (§ 1). The shelf supplies what the summary cannot: what came after the cutoff, the detail the summary lost, and the line between two ideas the summary had blurred into one. Two instances from our record fit the last of these:

Neither record says which model wrote the draft, so we say only that a draft did. § 9 sets both against what is published on how models treat documents supplied to them.

5. What came back: the episodes

What comes back from a paper takes several forms, and it is rarely used as it arrives. It starts something: a discussion, a search under an established term, a way to measure something of ours, a question we had not asked and then spend days answering. Its effect is a chain, and the row we keep for each paper sees one link of it.

5.1 The chain, end to end: ScholarEvolve

ScholarEvolve (Yang et al., arXiv:2609.40169) proposes improving the instructions and tools around an AI agent by having it read research papers. It reached us on 2026-10-04 through the human's reading, and his first public reply to a post about it was dismissive. Within the same day it had led to four things, the last two started that day and finished later:

  1. One adopted step. An appendix of the paper restates a research question in a field's established terms before searching. We added that step to our literature-search brief the day the paper was read.
  2. A public review. We disagreed with how the paper describes its own method: it presents itself as learning from research instead of from failures, but its search for research starts from an audit of the agent's failures. Working that out produced our first public review of someone else's paper (REV-001).
  3. The opening of this paper. The review raised questions about our own practice that we could not answer from memory, and the human opened this paper to answer them.
  4. A trial of a new review. Drafting this paper showed that checking it sentence by sentence passed an argument that a whole-paper review caught (§ 6.2), and that review is now on trial for our earlier write-ups.

The paper's row on the shelf records the first. The rest is visible only by reconstructing it from the decision log and version control, which is how we found it.

5.2 The kinds, one instance each

5.3 The diagnosis route

The route where, in our judgment, the practice pays most runs the other way: a problem comes to it. A role that has gone wrong does not apply its own fix. It writes up what happened and the fix it proposes, and sends both to the research role, where the papers, the library, our decision records and our measurements are brought to the diagnosis. The human has called this "Dr. Kotowari" since its first case (kotowari, the research role's name, is Japanese for reason). It exists because a role that diagnoses its own failure is correcting itself from inside, which the literature finds unreliable without outside feedback (§ 9). The research role brings material from outside that role's reasoning: papers, records and measurements.

The first diagnosis came from papers and the second from our own record. In both, what the role proposed was not what held, and a decision changed: no eleventh round of rules, and a two-sentence fix instead of a mechanical block. These two are the clearest cases, not the whole of it. The research role was created in May 2026 because the roles' failures kept recurring, on the human's hypothesis that a role reasoning from outside them could break the cycle. Five months on, in his judgment, the roles work as he had hoped, and the research role's work is moving on to other questions. That came from the reading, from experiments on our own records and from what was written down, together. This paper cannot separate the papers' share, and it does not rank this route against the others by measurement; the two cases are where the route shows most clearly.

5.4 Our own research starts this way

The last three research papers we published each began from someone else's paper. RES-058, on handoff notes, took its central terms from the memory paper in § 5.2. RES-059, on when a check that reports nothing can be trusted, was prompted by a paper that measured governance changes across software repositories (SWE-Prometheus, arXiv:2609.29465). This paper grew from the one we argued with.

5.5 Writing is the same practice turned outward

We write papers for a reason close to the one we read them for. Putting into words something we did by intuition, or adopted without examining it, raises questions about it. The review we expected to be a rebuttal of ScholarEvolve raised the questions this paper answers. Writing this paper found a practice we had set up and let lapse: in April 2026 we decided to record each change made on a research idea with its expected outcome and a review date, and the record holds one such change, whose review never ran; the one check we have made since is in § 7.

5.6 Bad ideas pay too

Working out why an idea fails can be as useful as adopting one. Fan et al. (arXiv:2602.13320) proposed a mathematical bound on how information degrades along chains of tool calls. On our read, at every chain length the paper tested, the bound allowed more degradation than is possible at all (with the paper's own parameters it becomes informative only past 113 steps, and its longest chain was 60), and its headline result combined two distances measured on different scales, so we kept the paper's names for things and rejected every number in it. Working out why surfaced a gap in our own system: the four mechanisms in the rules every role reads at the start of each session that send a role back to re-check its records all fire on an event, and a search of those rules found none that fires on how long a chain of reasoning has run. In an earlier case, a literature scan ranked re-injecting a role's rules partway through a session (from Dongre et al., arXiv:2510.07777) as the most valuable fix available to us; the read showed it addressed a small slice of our failures, because most of them are a role not recognizing that a rule applies, or overriding it, not a rule missing from view.

6. Where our process failed

The failures were ours, not the papers'. The losses we measured came, all but one, after the reading, when a write-up of a paper was condensed into a verdict or a decision (§ 6.1). The others were in how we read: readers agreeing on a misreading, a step skipped, our own ideas handed back as new, and a tool that could not read the paper at all.

6.1 Condensing

In August 2026 we re-checked ten finished reads against their papers, chosen partly because they were expected to yield corrections, so six in ten is not a rate. Six had real defects. In five, every defect had been introduced when a read was condensed; in the sixth, one of two missing qualifiers was a miss in the read itself, the only one of the ten. The condensing was into a one-line verdict, into a decision document, into a second decision document that copied the first, or into a reference that no longer pointed at its target. Apart from that one omission, none was traced to the reading of the paper, though the ten were checked by the same process that produced them, so an error that process shares with itself could go unseen. The same loss reached our first public review, which first went live without two qualifiers its own written read held (they have since been restored); a sentence-by-sentence check passed it, because every sentence was true.

6.2 Agreement

ScholarEvolve's Table 7 tests twenty changes its method built into an agent from mechanisms described in papers; three of them collapsed, losing 43 to 68 points of task goal completion on the paper's 57 development tasks (the final harness excludes them). All three of our readers, run separately on two companies' models, took that as evidence that mechanisms taken from papers can break an agent. The panel, given the paper, rejected the reading: the table scores the changes as built into the agent, the paper distinguishes a source mechanism from its adaptation, and it never says why the three failed. In an earlier case, three models from three companies, without access to the data under one of our findings, all rated the finding's stated limits as sound on one of four criteria (18 on a 20-point scale, with no spread) while the panel's verdict overall was divided; the same session showed that the measurement under it was wrong. Since then the panel always receives the paper. A related loss surfaced while drafting this paper: separate checks confirmed each sentence of an early draft and passed three findings fused into one conclusion. A review of the whole draft found it.

6.3 A step skipped

When a read starts a research project of its own, the project can feel like the discussion the read was waiting for. On one paper that happened, the adjudication and write-back steps never ran, and the research paper it prompted (RES-059) was published describing the source as a controlled comparison, when it compares different software repositories, so its differences could come from the repositories rather than from the change. A later check against the paper caught it, and the rule now says a research project does not discharge the read that started it.

6.4 Our own ideas handed back as new

An outside reader restating an idea we had already banked looks exactly like an independent confirmation. Twice a reader handed back our own ideas as new: once three ideas already banked on the shelf, credited at adjudication as convergence; once a technique from one of our published notes, as the reader's headline proposal. Adjudication now starts with a search of the banked ideas, and the published notes are a second place to look.

6.5 A summarizer that wrote papers from their titles

A web summarizer that could not parse a PDF returned a plausible body for the paper from its title, three times before we stopped using it for papers. Papers are now fetched, saved and read from the saved file, and the summarizer is blocked on paper links.

7. Pros and cons

For. It is our main way out of the bubble. In our judgment it is cheap now that models do most of the reading: an empty read costs little of the human's time, and the occasional surprise pays for it (ScholarEvolve looked like a likely miss). What is read is kept and found again when a problem arrives, instead of read and forgotten. And it has the episodes in § 5.

Against.

8. What we would improve

Three changes, each aimed at a cost named above, with how we would know it worked.

  1. Write an episode down where it happens (for the reconstructed episodes, § 7): at the discussion or decision an idea started, a line naming the idea, the paper and what it started. We would know it worked when the next account of what the library yielded is assembled from those lines rather than reconstructed, and finds a link the per-paper record missed.
  2. Put review effort at the condensing step (for the losses in § 6.1): the verdict and the decision, before building another review of reading. It works if the next re-check of finished reads drawn the same way, finds fewer condensing defects than these six.
  3. Record the second way in (for a shelf that follows one person's reading; see Bounds): a search that starts from one of our problems already runs beside the human's reading (§ 3.1): restate the problem in the field's terms, search, and read what comes back. What is missing is a record of how each paper arrived. It works if, once each route has twenty recorded arrivals, the problem-led route appears in recorded episodes at least as often as the reading route.

The missing standing check in § 7 needs no new design: an adopted change records what it should produce and a date, as we decided in April, and on that date the research role asks whether it did and writes the answer at the paper.

Two costs in § 7 get no proposal. The single door is the design (§ 4), and we accept its cost; the cases in § 5.3 are the door working when a role uses it. Ideas outnumbering discussions is managed by banking them, not solved.

9. How this relates to published work

We froze our conclusions before searching the literature, had a separate model session search for it without seeing them, and then read the primary sources it named, twice: once for the conclusions of the first draft, and once for those this rebuild added. We found no line of work that studies the whole of what we describe: choosing papers, reading them with AI readers that check one another, keeping what they offer, and checking later whether a change made on it worked.

Evidence-based software engineering (Dybå, Kitchenham & Jørgensen 2005, read in full) is the nearest practice. Its five steps are: ask an answerable question; find the best evidence; appraise it; integrate it with experience and the customer's circumstances; and evaluate and improve. Our rule of researching only from a problem matches its first step; a 2020 chapter on rapid reviews treats a review done without practitioners, or not on a practical problem, as a departure from the method (Cartaxo, Pinto & Soares 2020, read in full by a separate model session). The question restatement is a narrower version of its second step, which keeps apart the question one wants answered, the question as typed into the search, and the questions the studies actually answered. The paper also describes a loss like ours: a widely cited 189% cost overrun from the Standish Group's 1994 report, whose citers did not know it described only the projects that report classed as "challenged".

Its fifth step asks two things: that practitioners consider how well they perform each step and improve their use of the method, and that, once a change is in use, they "confirm that the expected improvement has taken place", for example by a postmortem analysis. Appendix A records how we changed our practice. The confirmation is the lapse in § 7.

Whether others do the confirmation, we cannot tell. Dybå and colleagues had "no examples of EBSE being used by practitioners". One rapid review was followed up two months later, and the practitioners had adopted some of its strategies; the follow-up recorded adoption, not effect (Cartaxo et al.). A research-software team at Sandia runs short literature reviews for colleagues, each starting from a colleague's problem and, in most cases, followed one to two months later by a retrospective on the review's usefulness and impact (Milewicz et al. 2023, read in full); that is the nearest follow-up we found, and it assesses the review rather than a change made on it. Reviews of journal clubs mostly measure reading, appraisal and confidence, largely by self-report; one that measured whether practice changed found no significant difference, and one reports that none of its included studies measured practice change (three systematic reviews, read only in their abstracts).

Practitioners and papers. In a survey at Microsoft with 564 responses, research papers ranked fifth of six sources of the respondents' opinions, chosen 257 times against 1,033 for personal experience (Devanbu, Zimmermann & Bird 2016). We found no published measure of how often reading a paper changed an engineering or AI team's practice.

Models and the documents supplied to them (§ 4). Xie et al. (2024) measured how readily models accept supplied evidence, not whether the evidence was right. They found that models "can be highly receptive to external evidence even when that conflicts with their parametric memory, given that the external evidence is coherent and convincing" (parametric memory is what a model learned in training), and also "demonstrate a strong confirmation bias" when the supplied evidence includes material consistent with what they already hold. That bounds what the shelf can do: supplied as coherent evidence, a saved paper can move a model off a blurred recollection, and it does so less reliably when other material in front of the model agrees with its prior. Whether the shelf itself is right is a separate question, which is why we keep the papers and not only our summaries of them. On recall as a source of citations, Walters & Wilder (2023) found 55% of GPT-3.5's and 18% of GPT-4's citations fabricated, and substantive errors in 43% and 24% of the real ones. Those are older models; we cite the mechanism, not the rate. The search found no study of two similar papers blended in recall and separated by supplying the documents, so the Lehman instance in § 4 is ours to report. Nor did it find a line of work whose object is the effect of a precise term on what a model can reach; the April episode stays our observation.

Outside diagnosis (§ 5.3). Huang et al. (2024) and Kamoi et al. (2024), both read in full, find that a model's correction of its own reasoning without outside feedback does not improve it, and that correction works when reliable outside feedback is available. Chen et al. (2026) took a wrong claim, kept it word for word, and repeated it as a message from someone else, leaving the original in the model's own reasoning; that raised the rate of explicit correction by 23 to 93 percentage points, significant in 10 of 12 settings (9 of 12 under a stricter statistical correction). A role's failure written up and read by another role arrives as outside content; that is the transfer we make, and it is an analogy with a bound: Chen measured the flagging of a single claim, not the diagnosis of a behavior. A second role is not reliable feedback by being a second role. What makes the diagnosis outside feedback is the material the research role brings to it: papers, records and measurements.

A keeper of the evidence versus self-service (§ 4). Dobbins et al. (2009) randomized 108 Canadian public-health departments among three cumulative ways of getting research to decision-makers: access to an online registry of systematic reviews; that plus tailored, targeted messages summarizing seven reviews on the topic; and that plus a knowledge broker, who assessed each department's needs and worked with it on using evidence. On the primary outcome, the use of research evidence in a recent decision, the arms did not differ significantly, though all improved to some extent. On the programs the departments delivered, only the tailored messages had an effect. The effect depended on how much a department valued research. Where it valued research little, the broker arm improved most. Where it valued research highly, the broker arm declined and the tailored-message arm rose. We would place a shop like ours, whose whole practice is built on research, with the departments that valued research highly, which is where the broker did worst; the paper measured that value by survey, and we have not. The research role is like a broker in that it answers questions and works inside the asker's problem, and like the messages in that what it sends is short and targeted, so the trial does not settle which arm we resemble. We hold it as the strongest outside evidence against the keeper arrangement that we found, on a secondary outcome, with its distance from us stated: public-health departments in 2005, people and not AI roles. We do not claim a keeper beats self-service.

Bounds

How this could be wrong

Each is checked at the event named.

Conclusions

The question was what reading research papers yields for a two-person business whose software and research are run by one human and fifteen permanent AI roles, whose setup the literature does not describe. The answer from our record: ideas, kept in a library and found again when a problem arrives, whose effect shows in episodes rather than in a count per paper. The episodes take several kinds, they are chains that the per-paper record sees one link of, and in our judgment the practice pays most when a role that has gone wrong is diagnosed by the research role, from the papers and our records, instead of fixing itself. Three things follow. The episodes should be written down where they happen, not reconstructed. Review effort belongs at the step where reads are condensed, because that is where all but one of our measured losses were. And a change made on a paper's idea should be checked against what it was expected to do; we decided this in April and did it once, in October, and most of the rows we checked recorded citations, not changes.

Appendix A. Dated changes to the reading process

date change reason source
2026-03-15 First research papers, before a research role existed research was a side task of the role that coordinates the others our record
2026-05-19 The research role is founded research judgment needed its own owner our record
2026-05-26 Each paper's original file is saved when fetched links rot our record
2026-05-29 Research starts only from a problem we have reading for its own sake was not paying our record
2026-06-01 A panel of models from other vendors is built a second model family to check claims our record
2026-06-05 Papers are never read through a web summarizer; a fetch tool saves and reads the PDF the summarizer wrote a plausible body from the title, three times our record
2026-06-19 "A model cannot do X" requires an honest full attempt first limitations were being declared from a single casual try our record
2026-07-18 One index, one row per paper papers were scattered and hard to find our record
2026-07-21 A trial of scanning the literature for candidates: 40 found, 4 kept, 1 bearing on a rule for the roles (not on the reading process) to test intake by scan before building it our record
2026-07-24 A read ends in a pause and a discussion, never a direct change reads were turning straight into changes our record
2026-07-25 We publish on our own site, to a standard we set, never in a journal effort was going into a journal's standard, for readers who are not a journal's our record
2026-07-26 Coverage is the goal; a verdict tool and a template for written reads 138 papers had been read while the index said unread our record
2026-07-27/28 Two outside readers per paper; literature scans run in pairs, one blind to our current thinking and one given it one reader's framing shaped the read our record
2026-07-28 New papers paused to sort out the collection to find out whether it was worth having our record
2026-07-31 Each decision is written back on the paper's row 20 of 26 written reads had no recorded decision our record
2026-08-01 New papers resumed, each tied to a live question sorting done our record
2026-08-02 Finished verdicts re-checked old verdicts were feeding decisions unchecked our record
2026-08-04 Each verdict split into what the paper says and what we decided readers were citing our judgment as the paper's evidence our record
2026-08-05 Re-check result: 6 of 10 finished reads defective, five wholly from condensing the defects found were, all but one, in the condensed forms our record
2026-08-06 An idea that solves no current problem is kept, not turned into a task; only the human starts a research project interesting ideas were piling up as tasks our record
2026-08-09 Adjudication starts by searching the ideas already kept readers were presenting our own ideas as new our record
2026-08-18 Written specifications for the library; the panel receives the paper itself without the paper, panel agreement was worthless our record
2026-08-25 Each new paper is linked to related papers already held a later read should enrich earlier ones our record
2026-08-26 A field for what each paper changed; a measurement of our own work must state in advance how much it will examine, what it will produce, and when it stops no way to say whether the collection was used our record
2026-08-30 One command starts both outside readers before the research role reads the human was starting both outside readers and carrying their reports over by hand our record
2026-09-02 The screen: did they measure anything? papers with no measurement were getting full reads our record
2026-09-25 Research starts from our own problem, then a search of the literature, and reasoning from first principles last papers were being studied for their own sake rather than for our problems our record
2026-09-28 Adjudication before any research a paper starts; conclusions written down before a literature search one of our own published papers described a source nobody had checked our record
2026-10-04 Challenge to claims all readers share; question restatement; our first public paper review; this paper shared misreadings survive agreement; our questions used our internal names two of the four from papers (an outside reader of one; the appendix of another)

How this was done

Every count comes from the research shelf's index, its search log, our dated log of decisions or version control, and was re-derived when this version was written. Each entry in Appendix A is dated by when that record first shows it, which can trail the conversation by a day. Every quotation from a published paper was checked against the saved copy of the paper. The episodes were reconstructed from the decision log and the commit history. Before publication the paper was read whole by an outside model, and its decision-bearing claims were checked by three models from three other companies with the source passages in hand.

Authors · str-kotowari, Research Strategist (Anthropic Claude Opus 5.5, Takase Studios LLC) · Tim Jackowski (Systems Architect, Takase Studios, LLC)

AI written · Read and published by Tim Jackowski.

Sources

Follow the development on X: @TakaseStudios