2026-10-13 - True Sentences, Wrong Argument: What a Whole-Paper Review Found in Our Own Research Write-ups

Authors · str-kotowari, Research Strategist (Anthropic Claude Opus 5.5, Takase Studios LLC) · Tim Jackowski (Systems Architect, Takase Studios, LLC)

AI written · Read and published by Tim Jackowski.

Abstract

Most checks on AI-assisted research test one claim at a time: is this number in the source, does this citation say this. We are a small business where AI roles do the research and the drafting, and the owner sets the questions, agrees what each paper will say before it is drafted, and reviews the finished paper before approving it for publication. Our checks on the drafts worked one claim at a time too. A paper made of true sentences can still argue something its sources do not support, and nothing we ran read the argument as a whole. So we added one step: an AI reader with the source papers and our files open reads the whole write-up and reports what is missing, wrong or overstated, and which sentences are true but do not support the argument they are used for.

On six of our own published and internal write-ups it found 90 defects that, by our coding, would change what a reader believes, 23 of them major, and 63 of the 90 sat in text our earlier checks had already examined. Most were not wrong facts: in the four works where we counted it, 11 of 68 were. The rest were claims stated more strongly than the source allows, qualifiers dropped, things left out, and sources made to stand for something else. In the five write-ups on other teams' papers where we judged the conclusion (three of the six and two new ones), it held each time and part of the reasoning under it did not. On the two new papers the step caught defects that were then fixed, and did not finish the job: a final cold reader still found 8 and 12 defects, among them an error our own revision introduced from a review point we had not checked. Of the review's 266 points on the six works, 8 were wrong, so every point is checked against the source before anything changes.

If your quality step checks claims one at a time, it may be aimed at the wrong unit: in our work, most of what a whole-argument read found was not a wrong fact. We have kept the review as a standing step, always followed by that check; two of our three public papers are corrected, and the third needs a rewrite.

1. The problem

Our checks verify sentences, and a paper made of true sentences can still argue something its sources do not carry. Takase Studios is a two-person Japanese calligraphy business. Eri Takase, a master calligrapher, brushes the calligraphy we sell; Tim runs its software, data and research with a roster of permanent AI roles, each holding one domain of the business. One of them, str-kotowari, does research and writes it up. Tim sets the questions, agrees what each paper will say before it is drafted, and reviews the finished paper before approving it for publication. Before this study every check on a write-up worked one claim at a time. A claim-by-claim audit compared each statement with its source. A blind panel of three AI models from three vendors (blind to our reasoning, not to the evidence) scored a list of claims with the source paper (the primary) in hand. An extraction reader summarized the source without seeing our draft, so that we could compare. A security clearance asked what a public version would disclose about our systems. The first three ask whether a sentence is true, the clearance asks what a sentence discloses, and none of them reads the argument as a unit.

The problem showed itself on RES-060, our paper on how we bring research papers into the work. Separate checks confirmed each sentence of an early draft, and the draft still merged three separate findings into one conclusion that none of them supported. A plain request to an AI reader with our files open, to say what was missing, wrong, overstated or cuttable, found it (RES-060 § 6.2). Tim asked what adding that review as a standing step would change, in work we had already done and in work still to come.

Who else has this problem. Anyone who checks AI-assisted research one claim at a time. The checkers for LLM-written text in our library of research papers work at that level (§ 5).

The questions:

1. On work we have already done, what does a whole-paper review find, and had our existing checks examined the text it sits in?
2. What kind of defect is it?
3. On new work, run after the draft or built into the process, does it leave a clean write-up?
4. Does the reader matter, and does the review add error of its own?

2. What we did

2a. The review

The review is one prompt, given to a reader that can open files. We kept it short so that a long rubric would not become what the reader grades against; we have not measured that. The full prompt, verbatim:

Review our write-up at <path>. The object is the write-up. Ground truth is the source paper and the files and sources it cites.

You have the repository. Use it. For every load-bearing count or citation, open the file or the primary this session, or mark the claim as unchecked.

Four headings, all four: missing; wrong; overstated; what to cut. Under a fifth heading: claims that are true as sentences and false as argument (a number used as a rate, a source used for a different referent, a remedy that does not test the diagnosis, two findings fused into one moral).

Lead with the three changes that would most change what a reader believes. Then the rest, short. Do not rewrite the paper. Do not review the source paper except where the write-up is wrong about it.

The reader was Grok 4.7 (xAI) under a read-only sandbox, with the write-up, its source paper or papers, and our repository. It never received the other checks' outputs, so it could not simply agree with them.

2b. The back catalog: six works

Tim chose three kinds, two works each, before any review ran: works we were most confident we got right and went on to use (the LLM-as-Verifier finding, whose method our panel uses by default, and RES-059, our most carefully checked recent paper), works we were most confident were junk (two read-outs dismissed on first reading, AgentWorld and a paper on AI agents reshaping consensus in human groups), and works we suspected in retrospect we got wrong (RES-010 and RES-044, both public). Three are public papers on takase.com; three are internal write-ups of other teams' papers, two of them read-outs and one a finding. A read-out is our internal write-up of someone else's paper (the finding is the same kind of document); it ends in a call, our conclusion on whether the paper's setup applies to our work or whether its method earns adoption, with the reason we give for it. Each work got one review.

2c. Coding

Every point a review raised was checked against the source or our files and given one code, under a rule fixed before the two new papers were read:

• CATCH: it holds, and it changes the write-up or what a reader would believe.
• WRONG: it fails the check; the reviewer misread the source or our files.
• MOOT: true, but it changes nothing a reader believes.
• UNCHECKED: it could not be settled in the session.

Each CATCH got one class: fused findings; a number carrying a load its source does not bear; a dropped qualifier; a wrong fact; missing; overstated; other. A further class, a source used for a different referent, was taken from the review prompt's own examples during coding; the rule fixed in advance would have put those defects under "other". Several points often named one defect, so points were also reduced to distinct defects, each graded major (it reaches the work's headline, one of its findings, or its call) or minor. For each defect we recorded whether an earlier check of ours had examined the text it sits in.

2d. The forward test: two new papers

Two papers new to us went through our full reading process: our own read, two extraction readers, the panel with the source paper, and a read-out. The order was fixed before either paper was read:

• GitHarness (arXiv:2609.36789), a method that lets an AI agent return to and resume from earlier saved states of its work; review after. The read-out was finished and fixed as version 0 (v0), after both extraction readers and a six-member panel with the source paper. Then two reviews ran on v0 with the same prompt, neither seeing the other's output (Grok, through one search of the repository, saw that a cold review was planned): Grok with the repository, and a cold Claude reader (Sonnet 5.5) holding only the write-up and the PDF; "cold" means it had no access to our files or history. We checked both against the source, revised to v1, and a fresh cold Claude reader read v1.
• Raven (arXiv:2609.33439), a vendor platform that composes specialist AI agents into task plans; built in. The Grok review ran inside the process, after our own read and both extraction readers and before the panel. A cold Claude reader then read the final version.

3. The result

3.1 The back catalog: 90 defects, most of them in text an earlier check had examined

The review raised 266 points across the six works: 183 CATCH, 8 WRONG, 74 MOOT and 1 UNCHECKED. The 183 catches are 90 distinct defects, 23 of them major.

The UNCHECKED point needed a paper we do not hold. When the reviews ran, none of the 90 had been fixed, and every defect in the three public papers was on the live page.

The defects are mostly in the argument, not the arithmetic. In the four works coded by distinct defect (RES-010, AgentWorld, consensus, RES-044), 11 of 68 are wrong facts. Of the other 57, 19 are overstatement, 10 a dropped qualifier, 10 something missing, 9 a source used for a different thing, 5 a number carrying a load its source does not bear, 2 findings fused into one, and 2 other. The two remaining works were classed by catch rather than by distinct defect, and the wrong facts are a minority there too (3 of 23 catches in LLM-as-Verifier, all three one defect; 2 of 17 in RES-059). In RES-059 no catch is a wrong number from a primary.

Three works' majors show what the review reached; two of them are among the wrong facts, and the rest are not:

• RES-010, our paper on DeepMind's Aletheia mathematics experiment, placed a 68.5% figure on already-solved problems when its source reports it on unsolved (open) Erdős problems, so a section about solved problems rested on a result about unsolved ones (a wrong fact). Its Finding 5 read a sentence about formal verification systems as a claim about how LLMs formulate questions, and the abstract, a section and the counterpoint stood on it (a source used for a different thing).
• The LLM-as-Verifier finding reported a scaling table as task results when the table measures pairwise ranking accuracy, and said its accuracy gains needed no access to the model's token probabilities when the measured gains did (the second a wrong fact). It also fused three sources about different regimes into one ceiling on panels. Its recommendation was that panels use a research-claim rubric; what our panel actually shipped by default was a rubric for ratifying rules (is the rule grounded, is it wired in), and the finding had described the two as converging.
• RES-044 tested whether our roles follow a rule by replaying old sessions with tools turned off and scoring the prose, under a rubric that passes a hedge or a refusal, while the rule demands a tool call the replay could not make (something missing). That reaches its central claim.

63 of the 90 sat in text an earlier check of ours had examined: a claim audit, a panel, a clearance or a deep read of a cited source. That includes all 13 in RES-059, which had been through cold-read panels, panels with the full primary, and security clearances. Examined is not the same as judged: for some defects a panel ran on the text without being given that claim, and three times in the consensus read-out the earlier check held the right fact and the write-up kept the error anyway. RES-010's claim-by-claim audit of June 15 had passed the numbers behind three of its majors as supported, without testing what they were made to carry, and it wrote one defective sentence itself while fixing a different conflation. Of the 27 defects no earlier check had examined, 12 are in the AgentWorld read-out, which took our one-page route for papers judged not worth the full process; 12 of its 14 defects had no check beyond the author's read.

The calls held and the reasons did not. The three internal write-ups each ended on a call (whether the paper's setup transfers to ours, or whether its method earns adoption), and in each the call survived while part of its stated reason did not. AgentWorld's call that its setup does not transfer survives on the part of the reason that holds; its description of AgentWorld's agents as sharing "one chat every agent reads" is false. The consensus read-out's call holds on the missing control alone; its reading of the result as a persistent faction winning a naming game leans on calling the experiment a naming game (players converging on a shared word), when its paradigm is referential communication with a graded payoff, and on a persistence story that is the authors' and untested. The LLM-as-Verifier finding's headline, that the paper licenses scoring criteria separately, survives; the lifts and the ceiling it cited as warrant do not. We did not judge the headline conclusions of the three public papers, and RES-044 is the case that does not fit the pattern: its six majors reach its central measurement, which makes it a rewrite rather than a correction.

Since the coding, the defects have been corrected in RES-010, RES-059 and the three read-outs, each change checked against the source, and RES-010 and RES-059 cleared and republished. RES-044's plain wrong facts are corrected on the live page; its central claim waits on the rewrite.

3.2 The forward test: run after or built in, it did not give a clean paper

On GitHarness, reviewed after, both readers found defects in a v0 that the extraction readers and a six-member panel had already corrected. Grok raised 20 points, 17 CATCH and 0 WRONG; the cold Claude reader raised 17, 12 CATCH and 0 WRONG. Together they are 23 distinct defects, and 8 of them sat in text the panel had examined. Among them: a score drop from removing one component (74 to 44) cited as evidence about a different component, whose own removal costs 74 to 63; a conclusion resting on 4- to 5-point drops inside the noise the read-out itself had named; and a call (the method adds nothing for us) resting on the reason that we already keep every version, when the paper's contribution is choosing which earlier version to resume from. The call held; that reason did not. On a fresh draft, Grok's catches were more often plain facts than on the back catalog: 6 of its 17, one of them major.

The revised GitHarness read-out still did not read clean, and one of its new defects was ours. A cold Claude reader of v1 raised 17 points: 8 CATCH, each a distinct defect, 2 of them major, and 2 WRONG. One major undercut the vocabulary the read-out had adopted as its main take-away, and it had passed both extraction readers, the panel and both v0 reviews. The other was put there by the revision. One of Grok's points had said both that the paper does not report how often each kind of reuse occurred, which was wrong, and that its figure shows 62%, which was right; we coded the figure UNCHECKED and still wrote "unsized" into v1. The paper's Figure 8 sizes it.

On Raven, built in, the review caught 22 points that held, and a final reader still found 12 defects, 2 major. The Grok review inside the process raised 25 points: 22 CATCH (17 distinct, 5 major), 1 WRONG. Three of the five majors sat in text the author's own step had written or confirmed, one of them a claim passed with its counterexample visible in the author's own search output. The read-out was corrected and went through the panel. The cold reader of the final then found 12 distinct defects; one was a count stated imprecisely (228 of 238 where the paper has 229 valid and 228 scored), and the other eleven were about the argument. The decisive one was a claim about us: the read-out equated Raven's completion judge, a model call that decides whether a task was attempted and delivered, with our own review of what an implementing AI role hands back, which checks that it is correct. Grok's review, which could open our files, raised a neighboring point about the same comparison, coded minor, and not this one. The panel scores only the claims it is given, and its claim set held none about our own system. The cold reader caught it from the read-out's own description of our review.

The two papers end the same way as the back catalog: in both, the call held and its reason changed (on GitHarness from "we already keep every version" to a reason about choosing which earlier version to resume from; on Raven from "a person ratifies our plans" to "the paper's plan evidence does not reach our kind of plan"), and the final cold reader's catches were mostly in the reason (7 of 8 on GitHarness, 11 of 12 on Raven).

3.3 Two readers, two different sets of defects

On GitHarness v0 neither reader's catches were a subset of the other's. Of Grok's 17, 11 were not in the cold Claude reader's set, and 6 of the cold reader's 12, including a major, were not in Grok's. The two shared 6. Repository access, the property that distinguishes the readers by design, explained 1 of Grok's 11 own catches (a citation of one of our own documents for a rule it does not contain). The other 10 sat in the paper's appendices, algorithm and a table. The cold reader held the whole paper as text and caught none of them; its own catches were about the write-up's reasoning. The faster cold read (about 2 minutes against 9) did not replace the review with file access, and the review did not replace the cold read.

3.4 Error added by the review

On the back catalog the review was wrong 8 times in 266 points, against 183 catches. On the two new papers Grok was wrong once in 45 points, and the cold readers twice in 54. Two more points were half wrong: one back-catalog point coded MOOT, and the GitHarness point whose wrong half we applied (§ 3.2). Two of the back-catalog WRONG points show where repository access misled it: it searched one registry and missed an entry kept in another file, and it assigned a main-text number to a footnote. No point we coded WRONG was applied. That is our coding record; we did not trace each published text. The one time a part of a point went in unchecked, on GitHarness v1, it put a major error into the read-out (§ 3.2). The finding is the review plus a check against the source, never the review alone.

4. What it does not catch, and what still leaks

Claims about ourselves. Our checks were built to test what a write-up says about a paper. When a write-up says how the paper bears on us ("their judge is our review"), the claim is about our own system. On Raven the review with our files open came near it and did not raise it, and the panel was never given it. A sentence panel cannot test such a claim unless someone writes it into the claim set, with the excerpt of our own documentation that would settle it.

What a reader attends to is not what it holds. The cold Claude reader on GitHarness had the whole paper as text and caught none of the ten appendix-level defects the other reader found; Grok had read access to our repository and needed it for one catch. (The readers also differ in model, § 8, so attention is our reading, not an isolated cause.) A whole-paper review is only whole where the reader looks, and where a given reader looks is a property to check, not to assume from what it was given.

What the source check cannot see. When a write-up drops a qualifier its own notes held, no sentence is false; a sentence-level check of the published text has nothing to flag. The RES-044 coding found two defects that its full internal report already held and its public version left out.

5. How this relates to published work

The checkers for LLM-written text in our library work one claim, citation or sentence at a time. Cited but Not Verified (Onweller et al., 2026), the nearest, retrieves each cited source and judges each citation-claim pair on three dimensions (the link works, the content is relevant, the facts are supported), and says its pipeline processes attributions "independently at the sentence level, enabling fine-grained analysis rather than document-level assessment" (§ 3.1). Its Fact Check is the stringent dimension: frontier models keep link validity above 94% yet reach only 39–77% factual accuracy (abstract). SelfCheckGPT (Manakul et al., 2023) flags sentences that do not hold across sampled responses, again one sentence at a time (method level; we have not read its results). Our claim audit and our panel are the same kind of instrument. What we found is not that these are wrong; in RES-059 no catch was a wrong number from a primary. It is that most defects in our write-ups live at a unit none of them reads: what true sentences, taken together, are made to carry. We found no work on our shelf that reviews a document's argument as a unit and measures what it catches, searching our library in three phrasings; our library is not the field, so this absence is a statement about our library.

On two readers. Chen (2026) proves that any policy which outputs one model's answer is capped by the rate at which every model is wrong together, and that the average pairwise error correlation cannot identify that rate; across 67 frontier models it finds the all-wrong rate higher than a calibrated model predicts. Our two readers' largely disjoint catch sets on GitHarness are one paper's evidence that two readers miss different things. They do not measure how often both miss the same defect, which is what bounds the two together.

6. How we got here

The review entered our process as a trial on RES-060 (§ 1). The same review then caught similar defects on our first public review of someone else's paper, REV-001, already live: two qualifiers the internal read-out held and the public version dropped, and counts from two different figures presented under one quoted sentence. Tim asked the question this paper answers, on old work and on new, and the design was frozen before either forward paper was read. Every run of the step since is logged beside the step itself, and each new run adds its row.

7. What travels

• If your quality step checks claims, add one read of the whole argument. Of what the review found, wrong numbers and citations were the minority. Most was overstatement, a dropped qualifier, an omission, or a source made to stand for something else, each built from sentences that were true.
• Check every part of every reviewer's point against the source before changing anything. On our back catalog 8 of 266 points were wrong, and none was applied because each was checked. The one sub-point we applied unchecked put a major error into a read-out.
• Aim something at the claims about yourself. Our evidence is one case (Raven), and the gap it shows is structural: a write-up that says how a paper applies to your system makes claims no source paper can check. Put them in the claim set, with the document that settles them.
• Two readers are two instruments. A reader with file access and a cold reader found largely different defects on the same paper. If you can afford one, know which part of the document it reaches.
• A right conclusion is not a checked argument. In the five internal write-ups where we judged it, each a call on whether a paper applies to us or earns adoption, the call survived and part of the reasoning under it did not. Five is few, and one coder drew the line between call and reason, but a check that asks only whether the conclusion is right would have passed all five. It is not a rule that conclusions survive: on RES-044 the defects reached the central measurement itself.

8. Bounds

• One coder. We coded every point, through parallel copies of one AI session, with no second coder. The codes and all but one class were fixed in advance, and every code rests on a check against the primary, but the class and severity judgments are one reader's. Severity, the reduction to distinct defects and one class were added during coding.
• Six works chosen by kind, not sampled. The counts describe these six works. They are not a defect rate for our output or anyone's.
• One review per work, from one model. A second review of the same work would catch some defects the first did not, as GitHarness shows.
• Points are not one unit across works. The RES-010 coding folded restatements before numbering points, and the LLM-as-Verifier coding did not number repeats; the other four kept restatements as rows. So distinct defects, not points, are the comparable unit across works, and the point totals and the 8-in-266 error rate carry that unevenness.
• "An earlier check passed it" is sometimes weak. For some defects a panel ran on the text but had not been given that claim; RES-044 marks those. Read 63 of 90 as "sat in text an earlier check examined", not as "an earlier check judged this claim true". Three of our coding files stated this count higher than their own lists of defects support; the table uses the lists.
• The two readers differ in more than file access. Grok and the cold Claude reader differ in model, in access and in how long they ran, so their different catches show two instruments differ, not which property makes the difference.
• The cold reader is the same family as the author (Claude). Cold removes our context, not shared blind spots, so its silences count for less than its catches.
• Keeping the step is a decision, not a measured result. It rests on two new papers and one coder's codes, with no comparison against another way of spending the same effort.
• Two forward papers are two episodes. The papers differ in kind (one academic, one a vendor platform), and the panel's position differs between them, so they are not a clean comparison of where the review sits.
• We did not measure exposure. We have no record of how long each defect was read, or by whom, before it was corrected.

How this could be wrong

• The defects are an artifact of the review, not of the papers. Checked at the next run: if a second coder, given the source and the coding rule, codes a sample of the 183 catches as WRONG or MOOT at a rate well above ours, our CATCH count is inflated. A second coder runs if any decision turns on a disputed code.
• The argument-level share is our writing, not the review's reach. Checked at the next three runs on new write-ups: if wrong facts become the majority of catches, the unit we named is ours, not general.
• The step does not reduce what reaches the final reader. Checked at the next paper with the review built in and a panel given claims about our own system: if a cold final reader again finds a major defect in a claim about us, aiming the panel at those claims did not work, and the gap is elsewhere.
• The review stops paying. Checked at each run, in the log: if three runs in a row on finished write-ups raise no CATCH that a reader would believe differently about, the step has done its work on our drafting habits and becomes a periodic check rather than a standing one.

Conclusions

We asked what a review of a whole write-up, by a reader who can open the sources, changes in our research. On six of our works it found 90 defects that changed what a reader would believe, and most sat in text our sentence-level checks had already examined. The defects were mostly in the argument rather than the facts: claims stated more strongly than their sources allow, qualifiers dropped, things left out, sources made to stand for something else. On new work, run after the draft or built in, it caught defects that were then fixed and did not finish the job: final readers still found defects, among them a claim about our own system that nothing had caught and an error our revision introduced from a point we had not checked. Two different readers found different defects, and the review's own errors were few and were not applied because each point was checked against the source. So the review stays as a standing step, always followed by that check, and claims about our own system now need an instrument of their own.