2026-10-13 - A Rule at the Top of the Prompt

Authors · str-kotowari, Research Strategist (Anthropic Claude Opus 5.5, Takase Studios LLC) · Tim Jackowski (Systems Architect, Takase Studios, LLC)

AI written · Read and published by Tim Jackowski.

Corrected 2026-10-15. § 4 used to say that no stored reply in any of the three runs contains a tool call. Thirty of the 864 write tool calls out as text, no call returned a result, and some report results they never received; the judges passed 27 of them, most as checks. § 4 now says so and recounts those replies as failures: the rule on top still cuts the failure rate in every run, by about two-fifths rather than a half in the second, and one comparison of the buried rule would then pass the correction for multiple tests. The abstract and § 3.1 point to the recount, and § 8 carries the bound.

Rewritten 2026-10-13. This note replaces its 2026-06-15 edition, Capability versus Substrate. A whole-paper review found that several of that edition's conclusions did not survive its own data; § 6 lists each one and what replaced it.

Abstract

The question. We run a Japanese calligraphy business. A master calligrapher brushes every stroke of the art we sell, and long-lived AI agents ("roles") help run nearly everything else. Each role is a chain of Claude sessions, and each session starts by loading the role's written rules. The roles break rules they have loaded, and a common case is stating a specific (a number, a file path, a name) as fact without checking it. A June 2026 count of our own failures found that the share of failures where the rule was loaded and broken anyway did not fall across three releases of Claude Opus. We built policy on that: improve the rules and add checks in code that refuse a mistake ("gates"), rather than wait for a better model.

What we set out to show. Whether stating the rule at the top of the prompt reduces this failure, whether a more capable model does, and whether the rule still helps when it sits inside the long rule files a role actually loads.

How. We took 32 real moments, each just before such a failure that a person caught, and replayed them to Claude Haiku 4.5, Sonnet 4.6 and Opus 4.8. Each moment ran with no rule, with a 923-character version of the rule at the top of the prompt, or with the rule as written inside its 32 KB rule file. Two runs gave 480 replies. A follow-up on Opus added the rule inside a 374 KB bundle of the files one role loads (384 replies). Three models from other vendors judged every reply blind on whether it states an unchecked specific as fact, and the failure rate is the share of replies that at least two judges called a failure (§ 2).

What we found. A short rule at the top of the prompt cut how often models stated unchecked specifics as fact. The replays were text-only, so whether the rule makes a model check was not tested. - With the rule on top the failure rate halved in the second run (0.41 to 0.20, three models together) and in the Opus follow-up (0.385 to 0.177), both p < 0.001; in the second run the cut is about two-fifths if the replies that wrote out tool calls are counted as failures (§ 4). In the first run, which had no paired test, it fell by a quarter (0.46 to 0.34). It is the only difference between conditions that survives correction for running several tests on one dataset (§ 3.1). - In both runs with three models, Opus failed least; Sonnet was not distinguishable from Haiku where they were tested. Three models rank the tiers; they do not show how much a still better model would gain (§ 3.2). - Whether the rule helps when buried in its 32 KB file is not settled. It landed between no rule and the rule on top, and no comparison of it survives the correction (§ 3.3). The rule on top was a broader rewrite of the file's text, so comparing the two changes the wording as well as the position (§ 2.2). - The 374 KB bundle changed three things at once: the rule's distance from the task, how often it appears, and what surrounds it. So it cannot show the effect of depth alone (§ 3.4). - Both versions of the rule accept either a check or a mark that the specific is unchecked. The judges passed a reply that left the specific out, marked or hedged it, refused, or asked. How often a reply repeated the original session's exact mistake did not change significantly (§ 4).

What we are doing about it. Our gates do not rest on this experiment, which never tested one, and we now say so. Whether a rule makes a model check needs a replay with tools and a record of whether the model looked (§ 4); we name that test and are not running it. Our earlier report of these replays, What Survives Good Context, has been corrected to match (2026-10-14).

1. The problem

Takase Studios is a two-person Japanese calligraphy business. Eri Takase, a master calligrapher, brushes the art herself, every stroke of it, and no AI makes it. Tim runs nearly everything else with a set of permanent AI roles (How We Build takase.com describes the arrangement), each holding one domain of the business: the rendering engine, the live website, security, research, content and others. A role is carried by a chain of Claude Code sessions, and each session starts by loading the role's rule files. The rules are written down because the roles break them. Mostly the role does not lack the rule: the rule is loaded, and the session does what it forbids anyway. A common case, though not the commonest, is stating a specific (a count, a file path, a version, a name) without checking it first.

In June 2026 we counted these failures across our own records (It's Not a Missing Rule reports that census). A census of 3,623 session transcripts found 898 moments where a role failed at something its rules covered. A model labeling each one put about 96% in the class where the rule was loaded and the cue meant to bring it to mind was present, and about 4% where the rule was not loaded or was loaded without that cue. The share was flat across the three Opus releases the period spanned: 96.4%, 95.8% and 96.7% for Opus 4.6, 4.7 and 4.8. A second Claude model re-labeling a stratified sample of 100 agreed with the first at κ = 0.59 on that split; agreement within one model family is the friendly case, and no cross-family re-label was run.

We built policy on the flatness. If a better model did not shrink the class, the lever was elsewhere: put the rule where it fires, and build mechanical gates that refuse the mistake whatever the model recognizes. But three other things moved with the releases and could each have produced the flatness. The rule files grew; the share of findings a human had caught shifted (73%, 53%, 61%); and the census found more per transcript as time went on (the mean number of findings per transcript digest rose from 0.16 to 0.31 to 0.40 while the number of digests fell from 1,879 to 1,174 to 583). The retired releases cannot be re-run. So we asked the questions that could still be tested, holding the task fixed: does a rule at the top of the prompt reduce this failure, does a more capable model, and does the rule still help inside the rule files a role really loads?

The census covered every kind of failure its rules addressed. The replay tests one family within it, which is common but not the largest: in a later count of human-caught failures by their automatic labels, the family's two sub-classes totaled 115, against 188 under the label for reasoning from a wrong frame, a label the same count calls mixed.

Anyone deciding whether a rule belongs in a long system prompt or at the point of decision, or whether to wait for a better model or build an enforcement step, has the same question.

2. What we did

2.1 The moments

We replayed one family of failure, asserting without verifying, in its two sub-classes: fabricated precision (a number or detail stated more exactly than anything checked supports) and an unverified identifier (a path, name or route stated without looking). Each replay prompt is the context that preceded a real failure of that kind which a human caught. A seeded script drew them from the census, stratified by sub-class and by the Opus release in use at the time.

The battery has 32 moments: 14 of fabricated precision and 18 of an unverified identifier, from 31 conversations, of which 12, 14 and 6 come from the Opus 4.6, 4.7 and 4.8 periods. For 18 of the 32 the preceding context was too long to use whole and was trimmed to a window ending at the decision. Each prompt was given as a task and never flagged as a test.

2.2 Three ways to present the rule

• No rule: the moment's prompt alone.
• The rule on top: a 923-character version of our verify-before-assert rule, placed at the top of the prompt.
• The rule in its file: the rule as written, in place inside the always-loaded rule file it lives in (32,315 bytes, 205 lines), put before the prompt as loaded rules. The rule's heading is at line 58, about 8 KB in, so about 24 KB of other rules sit between it and the task. In a real session this file loads with ten others.

The two are not the same text. The rule on top was written for the experiment and is broader: it covers dates, frequencies and "any specific factual precision" as well as identifiers, and adds "say so or route it — do not state it as fact". Both accept either of two ways to comply: a tool call in the same reply that checks the specific, or an inline mark that it is unchecked. So a comparison of the rule on top against the rule in its file changes the wording as well as the position, and does not isolate placement.

2.3 Three runs

• First run, 192 replies: the 32 moments, with no rule and with the rule on top, on Haiku 4.5, Sonnet 4.6 and Opus 4.8.
• Second run, 288 replies, freshly generated: the 32 moments under all three conditions, on the same three models. With the first run, 480 replies.
• Follow-up, 384 replies, Opus 4.8 only: 48 prompts (the 32 plus 16 more drawn by the same script), two replies each, under four conditions: the three above and the rule inside a 374 KB onboarding bundle (§ 3.4). Preparing this edition, we found that six of the 48 prompts appear twice (same conversation, same failure), so the follow-up has 42 distinct moments. § 3 gives the follow-up's tests as run and with each moment counted once.

2.4 The judges

Every reply was scored by three judges from other vendors (Gemini 2.5 Flash, GPT-4.1 and Grok 4.3), blind to the model and the condition. A reply counts as a failure when two of the three say so. Each judge answers two questions. The primary one: does the reply assert any unverified specific of this kind? The secondary one: does it repeat the original error? A reply passes the primary question if it verifies, hedges or marks the claim uncertain, flags it as unverified, refuses to assert, or asks instead of asserting. The judges also saw the original failure. It is the same in every condition, so it cannot by itself produce a difference between conditions, though it may shape how the judges read replies that differ.

Agreement was moderate and steady: Fleiss' κ (0 is chance agreement, 1 is perfect) was 0.438, 0.434 and 0.442 on the three runs, and pairwise κ 0.426 to 0.469 on the first. Which judges sit on the panel matters: with GPT-4o-mini in place of GPT-4.1, agreement drops to 0.236.

2.5 The tests

Every moment appears under every condition, so the tests are paired. Between two conditions we use McNemar's test on the pairs where they disagree, written here as "23 against 3": 23 pairs where the condition that did worse failed and the other did not, and 3 the other way. For the model comparison we use a logistic model grouped by moment. Each run asks several questions of the same data, so this edition applies Holm's correction within each run (rank the run's p-values, test the smallest against 0.05 divided by the number of tests, the next against 0.05 divided by one fewer, and stop at the first that fails): three tests in the second run, six in the follow-up. The 2026-06 edition had no such correction.

3. Results

3.1 The rule at the top of the prompt cuts unchecked statements

The rule on top halved the failure rate in the second run and the follow-up, and cut it by a quarter in the first. In the first run, pooled over the three models, the rate fell from 0.46 (44 of 96) to 0.34 (33 of 96); no paired test was run on it, so that fall is descriptive. In the second run, pooled over the three models, the rate fell from 0.41 (39 of 96) with no rule to 0.20 (19 of 96) with the rule on top: 23 against 3, p < 0.001. In the Opus follow-up it fell from 0.385 (37 of 96) to 0.177 (17 of 96): 22 against 2, p < 0.001, and 20 against 1 with each moment counted once. It is the one contrast that survives the correction in both the second run and the follow-up, and in the follow-up it survives every way of counting we tried (Appendix A). Nor does it come only from the 18 prompts trimmed to a window, where an honest reply has more reason to hedge. In the second run the rate halved on the trimmed prompts (0.52 to 0.26, 16 against 2, p = 0.001), and fell by about the same proportion on the 14 used whole (0.26 to 0.12), on too few disagreeing pairs to be significant (7 against 1, p = 0.070). These counts are the judges'. Replies that wrote out tool calls were mostly scored as passes; counted as failures (§ 4), the second run's cut shrinks from a half to about two-fifths.

Pooled, the halving hides three different patterns, and in the second run only Opus is significant alone. In the second run, from no rule to the rule on top: Haiku 15 to 8 of 32 (10 against 3, p = 0.092), Sonnet 12 to 9 (3 against 0, p = 0.25), Opus 12 to 2 (10 against 0, p = 0.002). In the first run, Haiku stayed at 0.50 with and without the rule, Sonnet fell from 0.56 to 0.34 and Opus from 0.31 to 0.19. The 2026-06 edition read that first-run Haiku cell as showing the rule helps "only for models capable enough to use it". The second run does not repeat it, though there Haiku's buried rule did nearly as well as its rule on top (§ 3.3). Single-model cells are unstable here: the same 32 moments, freshly generated, gave Haiku with the rule on top 0.50 in the first run and 0.25 in the second. The pooled contrast is what holds, and it holds when the test respects that each moment appears once per model: the second run's model grouped by moment gives the rule on top an odds ratio of 0.36 against no rule (95% interval 0.24 to 0.52, p < 0.001).

In the follow-up the rule on top also produced fewer failures than the rule inside the 374 KB bundle: 16 pairs where only the bundle condition failed, against 4 the other way, p = 0.012. That result sits on the line: as the follow-up was run it misses the correction (0.012, ranked second of six, against 0.010), and in the two recounts in Appendix A it passes. We read it as one more comparison the rule on top wins, not as a finding about the bundle (§ 3.4), and like every comparison with a buried rule it changes the wording as well as the position (§ 2.2).

3.2 Opus fails least; an ordering, not a curve, and no floor shown

Opus had the lowest failure rate in both runs; Sonnet cannot be told apart from Haiku. In the first run, with the two conditions pooled (64 replies per model), the rates were Haiku 0.50, Sonnet 0.45 and Opus 0.25, a significant downward trend (Cochran–Armitage z = −2.886, p = 0.004). With no rule alone the order is not monotone: 0.50, 0.56, 0.31. In the second run's model over all three conditions, Sonnet is not distinguishable from Haiku (odds ratio 1.10, 95% interval 0.62 to 1.95, p = 0.74) and Opus is lower (odds ratio 0.55, 95% interval 0.34 to 0.89, p = 0.016, a test outside the families corrected in § 2.5); Opus and Sonnet were not compared directly. The second run's pooled rates are Haiku 0.33, Sonnet 0.35, Opus 0.22.

The 2026-06 edition said Opus sat near a floor, so a better model was unlikely to fix much of what remained. Two numbers carried that. The 0.25 is the mean of Opus's two first-run cells (0.31 with no rule, 0.19 with the rule on top). The 0.06 is the best of the three Opus cells with the rule on top: 2 of 32 in the second run (95% interval 0.02 to 0.20), where the other two were 0.19 (6 of 32, first run) and 0.18 (17 of 96, follow-up). Three models on two runs give an ordering, not a curve. One cell of 32 replies does not show a floor, and the other two Opus cells, near 0.18, do not rule one out either.

Nor does the replay settle the census. The census compared three releases of one model line over time; the replay compares three model sizes at one time. The replay does not show that the census's flatness was wrong, and the census does not show that a better model cannot help.

3.3 Whether a buried rule helps is not settled

A rule inside its 32 KB file landed between no rule and the rule on top in the second run and the follow-up, and no comparison of it survives the correction. In the second run it failed at 0.30 (29 of 96). Against no rule, 21 against 11, p = 0.11. Against the rule on top, 15 against 5, p = 0.041, which misses the correction (threshold 0.025), on a comparison that changes the wording as well as the position (§ 2.2). By model the picture differs: Haiku 15, 9 and 8 failures of 32 (no rule, in its file, on top), Sonnet 12, 13 and 9, Opus 12, 7 and 2. Haiku's buried rule did about as well as its rule on top, Sonnet's did no better than no rule, and Opus's sat between. The judges split on it too. By each judge's own rates (no rule, in its file, on top), Gemini at 0.40, 0.36, 0.26 and Grok at 0.48, 0.46, 0.34 put the buried rule near no rule; GPT-4.1 at 0.34, 0.23, 0.20 put it near the rule on top.

In the follow-up it failed at 0.26 (25 of 96). Against no rule, 17 against 5, p = 0.017: 15 against 4 (p = 0.019) with each moment counted once, and 12 moments against 2 (p = 0.013) when the test counts moments. Each misses its correction (0.0125 at its rank) by a small margin. Against the rule on top, 11 against 3, p = 0.057.

So pooled, both runs that tested it point toward the buried rule helping; in the second run that comes from Haiku and Opus, not Sonnet. The follow-up comes close to showing a benefit under every way of counting we tried (Appendix A). The second run cannot tell it from no rule. The 2026-06 edition reported the second run's 0.041 as the buried rule reaching "significantly less" than the rule on top, and did not report the follow-up's 0.017. Its headline that placement is the lever, in the sense that a rule inside its file stops working, is the one claim of that edition this data cannot carry in either direction.

3.4 What the 374 KB condition was

The follow-up's fourth condition was built as a test of depth, and it changed three things at once. The bundle is 383,948 bytes in eleven files: the project's top-level instructions, eight rule files (the 32 KB one among them), one role's manual (42 KB) and that role's session-state file (187 KB, about half the bundle). In it the rule's heading sits about 45 KB in, with about 338 KB after it. The role manual restates the rule's check-or-mark core in its own words about 214 KB before the task, and six other lines mention it by name. In the 32 KB condition the rule appears once, with about 24 KB after it. So between the two conditions the rule's distance from the task, the number of times it appears, and the material around it all change.

The rule inside the bundle failed at 0.30 (29 of 96). Against no rule, 16 against 8, p = 0.15 (13 against 7, p = 0.26, with each moment counted once). Against the rule inside its 32 KB file, 13 against 9 (the bundle doing worse), p = 0.52. The 2026-06 edition read the first of these as "at true production depth, no detectable benefit". It is a failure to detect a difference, on a condition that varied three things.

4. What a replay without tools can and cannot measure

The rule asks for a check or a mark, and the replay could show only the mark. Both versions of the rule accept either a tool call in the same reply that checks the specific (open the path, run the count) or an inline mark that it is unchecked. In the replay the model receives the prompt and writes a reply, and no stored reply carries a tool result. How the second run's generation call withheld tools cannot be recovered from the files we kept; the follow-up was planned on the same kind of call, and the first run's call is not recorded. Thirty of the 864 replies (10, 12 and 8 by run) write tool calls out as text anyway, in the form the transcript in their prompt uses, and some go on to report what the calls found: results no call returned. The judges passed 27 of the 30, most as checks. A reply that reports a result it never received has stated an unchecked specific as fact, which is the failure this test counts. So we recounted all thirty as failures, the most a different scoring of them could change (27 replies flip, 21 of them with the rule on top or with no rule). The rule on top still cuts the failure rate in every run: from 0.50 to 0.39 (48 and 37 of 96) in the first, descriptive only, since that run had no paired test; from 0.44 to 0.26 (42 and 25; 21 pairs against 4, p < 0.001) in the second, a cut of about two-fifths rather than a half; and from 0.40 to 0.21 (38 and 20) in the follow-up, 16 moments against 1 with each moment counted once (p < 0.001). With those replies and the replies paired with them left out instead, the cuts are 42 to 32 of 88, 37 to 18 of 89 and 36 to 17 of 93, close to those reported. The buried rule's comparisons move both ways under the recount, and one crosses: counted as failures, the follow-up's buried rule against no rule goes from 17 against 5 to 18 against 5 (p = 0.011) and would pass the correction, where § 3.3 reports that none does. Whether it passes depends on how thirty replies are coded, so we leave § 3.3's reading as it is: whether a buried rule helps is not settled. A reply passes if it leaves the unchecked specific out, marks it, hedges, refuses or asks. So what the rule on top changed is how often a model states an unchecked specific as fact. Whether a rule makes a model actually check is not tested here. That is worth something where a person reads the reply, since a specific marked as unchecked can be caught; it is not the same as checking.

On the original error itself the test can say little. On the secondary question, whether the reply repeats the original error, rates were low in every condition (4.2% to 12.5% per judge and condition in the first run), and in the first run it did not fall with the rule: two judges' rates rose slightly and one fell. In the second run the rule on top moved it in the expected direction, 7 pairs against 1, but eight discordant pairs give p = 0.070.

A second family showed where this kind of replay stops. In June 2026 we replayed failures that happen mid-action (carrying out a step on a wrong premise) as prose replies without tools, generated by a different call. A keyword count tagged 66.7% of Opus's replies to those as declining to act, against 4.7% on the family reported here, and 41.7% (30 of 72) when the replay ran from a neutral working directory. The count misses some declines and over-tags replies that flag a problem and then decide, and the two batteries differ in how they were generated, so this is a strong sign, not a clean measurement. Our likeliest reading, which these counts do not separate from how the batteries were generated, is that a capable model recognizes a frame with its tools removed and will not act inside it. Here a tools-off replay measured failures of statement; on failures of action it mostly measured refusals to act.

The test that would show whether a rule makes a model check is a replay with tools and a check on the outcome: the model is put back at the moment with the files and commands it had, and the outcome records whether it looked before it stated. That is a large build, and we are not proposing to run it.

5. Related work

Instruction adherence in coding-agent configuration files (McMillan, arXiv:2605.10039, 2026) is the nearest controlled study. Across 1,650 Claude Code sessions (mainly Sonnet 4.6, with Opus 4.6 and Opus 4.7), it varied a configuration file's size from 25 to 500 lines, the instruction's position, the file's architecture and the presence of a conflicting instruction, and measured compliance with a trivial annotation while the agent did real coding tasks with its tools. None of the four variables produced a detectable contrast after correction for multiple tests. For size, a Bayes factor supports no effect within the range tested (BF10 = 0.096); for position, the null is a failure to reject. No model in its trials emitted the marker without the configuration file, and its target is inserting a marker, not a judgment about what to assert. Its largest effect was not structural: an exploratory finding that compliance fell as the agent wrote more functions within a session (about 5.6% lower odds per function on average, odds ratio 0.944, though not at a constant rate per step, and detected on two of its three multi-function tasks), which the author says warrants independent replication. On the question our buried conditions could not settle, whether a rule loses force inside a larger file, its size result is better evidence than ours within its range (its largest file is 500 lines; our 374 KB bundle is 2,066), and its session result says where else to look.

Multi-agent failure taxonomy (MAST; Cemri et al., arXiv:2503.13657, 2025). Our family sits closest to MAST's task-verification category: no or incomplete verification (FM-3.2), the omission of proper checking of task outcomes or system outputs, with incorrect verification (FM-3.3) next to it. Ours is a narrower case, a single specific stated without a check. MAST studies autonomous systems and ours has a human in the loop, so we use its category name and rest nothing on its results.

Self-correction (Kamoi et al., arXiv:2406.01297, 2024). The survey finds no demonstrated self-correction from a model's own prompted feedback outside tasks unusually suited to it, and finds that self-correction works where reliable external feedback exists. A verify-before-assert rule is not self-correction in the survey's terms, which refine a response after it is written; a tool call before the claim is closer to what the survey calls improving the initial response with external information. What the two share is information from outside the model, and a tools-off replay removes that source (§ 4).

6. What changed from the 2026-06 edition, and why

The first edition is dated 2026-06-15 and was revised 2026-06-16 (the follow-up and a paired re-analysis). On 2026-10-13 four errors were corrected in place: a source's title, the census's recall figures, the claim that the family is the largest, and the 32 KB file described as production; the related-work line on that source was corrected with them. A whole-paper review of the first edition, run as part of re-checking our published work (True Sentences, Wrong Argument), found 20 distinct defects, six of them major. The six reach what the paper says, so this edition replaces it. For a reader who has cited the first edition:

The first edition never stated that the replays had no tools. Its internal master had carried that limit since 2026-06-20, and the published claims never reflected it (§ 4).

What Survives Good Context (2026-07-24) reports the same replays and carried three of the first edition's readings: the 32 KB file as where the rule lives in production, the frontier near its floor, and a buried rule eroding "where production puts it". It was corrected to match this edition on 2026-10-14.

7. What travels

• A short rule at the top of the prompt is a measured lever. A rule of about 900 characters at the top of the prompt halved how often our models stated an unchecked specific as fact: pooled over three models in the second run, and on Opus in the follow-up. It is short, and it is measured.
• Test the text you will deploy, or say you did not. Our rule on top was a rewrite of the rule's file text, so comparing it with the buried rule compared two wordings as well as two places.
• Do not read a null on a buried rule as "rules in long prompts stop working". Our data cannot carry that either way, and the nearest controlled study we know of finds evidence of no effect of file size from 25 to 500 lines, on a task of inserting a marker.
• Keep "says less that is unchecked" apart from "checks". A test without tools measures the first; only a test with tools and an outcome check measures the second.
• Treat a comparison of three models as an ordering, not a curve. One small cell does not show a floor.
• When a condition changes several things at once, name all of them before reading its result as the one you built it to test.
• Correct for the number of tests asked of one dataset. In ours, it turned both results the first edition reported as significant into unsettled ones, and a third that edition did not report (p = 0.017) does not survive it either.

8. Bounds

• One family of failure, the statement kind. The action kind could not be tested this way (§ 4).
• Thirty-two moments (42 in the follow-up), chosen from failures a human caught, so the battery favors failures a judge can see. Two of the 32 come from one conversation.
• The judges agree moderately (κ about 0.44), and on the buried condition they split.
• The rule on top is not the file's text (§ 2.2), so no comparison here isolates placement.
• Each moment appears once per model in the second run's paired tests, so those pairs are not independent; the model grouped by moment agrees on the rule on top (§ 3.1).
• The follow-up pairs two replies per moment, and its tests as run treat them as separate pairs, which overstates the sample. Where it matters (§ 3.1, § 3.3) we give the test that counts moments as well.
• Thirty replies write out tool calls that returned nothing, some with invented results, and the judges passed 27 of them, most as checks. Counted as failures, the rule on top still cuts the failure rate in every run, by about two-fifths rather than a half in the second (§ 4).
• The models are the June 2026 models. Our roles run on later ones, and nothing here was re-run on them.
• The tested models are Claude, and so are the sessions the moments come from; the judges are not. No prompt was flagged as a test, but each is a conversation transcript with its header lines, and whether the models took them for replays was not measured.
• One shop, one rule. The rule on top and the rule files are ours.

How this could be wrong

• The rule on top (§ 3.1). If a re-score of the second run and the follow-up by a different three-judge panel puts the rule-on-top contrast above p = 0.05 in either run, the one established finding fails. Read at any re-score of these replies.
• Sonnet no better than Haiku (§ 3.2). If a re-score by a different panel puts Sonnet significantly below Haiku in the second run's model, this finding fails. Read at any re-score.
• The buried rule (§ 3.3). This is an open result, and any replay of the battery with more moments under both buried conditions settles it one way or the other. Read at such a run, if one is opened.
• Omission, not checking (§ 4). If a tools-on replay of this battery shows that the rule on top raises how often models check before they assert, the claim stands and gains its missing half. If it shows the rule cuts unchecked statements without any more checking, § 4 is confirmed. Read at such a run, if one is opened.

Conclusions

We asked whether a rule at the point of decision, a more capable model, or a rule inside the files a role really loads reduces how often an AI role states something it has not checked. A rule at the top of the prompt does: it halved the rate in the second run (three models pooled) and in the follow-up (Opus), surviving the correction in both, and in the first run, which had no test, the rate fell by a quarter. Opus failed least in both three-model runs, but three models rank the tiers without showing how much a better model would gain, and they leave open whether a rule buried in a long rule file still helps. Because the replays had no tools, all of this concerns what a model says; whether it checks was not tested. Two things follow. A short rule stated at the top of the prompt is worth its place, though this experiment cannot say how much of that is place and how much is wording. And our mechanical gates do not rest on this experiment, which never tested a gate; whatever case they have must come from elsewhere.

Appendix A. The follow-up's six tests, three ways

The follow-up (Opus 4.8) as run counts 96 pairs: 48 prompts, two replies each. Six prompts appear twice, so the second column counts each moment's prompt once (84 pairs). The third counts moments: for each of the 42, it compares the failures summed over its two replies under each condition, and a sign test counts the moments that did worse under each (ties drop out, and a drop from two failures to one counts the same as from two to none). Each cell gives the worse condition's count first. Holm's correction is applied within each column; the thresholds follow from where a p-value ranks in its own column.

✓ survives Holm's correction within its column. As run, the second-ranked p (0.012) misses its threshold of 0.010 and the procedure stops. In the other two columns the second-ranked p passes 0.010 and the third (0.019, 0.013) misses 0.0125.