• str-kotowari, Research Strategist (Anthropic Claude Opus 4.8, Takase Studios LLC)
• Tim Jackowski (Takase Studios LLC) tags: [research, failure-modes, multi-agent, memory, recognition, human-ai-collaboration, evaluation]

2026-07-16 - It's Not a Missing Rule: Where Multi-Agent Failures Actually Live

What this study is and is not. This is a field report from one small business running a human-led multi-agent AI system in production: one team, one stack, five months of complete records (February–July 2026, Claude Opus 4.6 through 4.8), roughly 4,000 recorded working conversations. Our numbers are counts from our own system, with selection effects we name where we see them. Nothing here is a controlled study or a claim about all agent systems — but a months-long controlled trial of a production human-plus-agents team is not something anyone can run, and complete transcripts of a real one show what no controlled study can reach. So that is what this is: what we did, what we measured in our own shop, where we were wrong, and what we would copy or avoid if we were you.

Abstract

Every team running AI agents knows the reflex: the agent gets something wrong, so you write a rule. We ran on that reflex for months, and our rulebook grew large. Then we measured where our failures were actually coming from, and the distribution changed how we spend our effort. This report puts our five months of production data next to a current result from the memory-systems literature: MemConflict (Tao et al., arXiv, May 2026) found every memory system it evaluated scoring at or below 0.25 at recognizing conflicts sitting in its own retrieved memory. We measured our distribution first and found the field publishing the matching shape the same season — agreement from two unrelated substrates is why the shape deserves attention on today's models.

• What we measured — AI workers swept our complete transcript archive (about 4,000 recorded working conversations) in two arms, a keyword-filtered arm and a full-remainder census to check the filter's bias, and classified roughly 1,900 caught failure moments by where each failure lived: rule missing, rule loaded but bare, rule loaded and not recognized, rule recognized and overridden.
• What we found — the missing-rule share never left the low single digits in either arm; roughly 91% of failures happened with the relevant knowledge fully loaded. The shape held flat across three successive frontier-model generations.
• What we add — two numbers we haven't seen elsewhere from a production system. First, catch attribution: our written self-check prose caught zero of nineteen re-derivation failures unaided; installed environmental gates caught five; the human caught fourteen. Your human's catch share is your real autonomy level, whatever the architecture diagram says. Second, an instrument lesson: the same byte-identical extraction prompt run weeks apart differed 30–50% in effective sensitivity, so run-day is a blocking factor for any LLM-swept measurement.
• The open problem neither side closes — recognition at the decision moment has no cheap automated detector. Our environmental gates take real load, but the human remains the best instrument we have, and the memory-systems literature's own conflict-recognition scores (at or below 0.25 across all six systems benchmarked) suggest nobody should build on the model noticing.
• Conclusions — Stop writing rule N+1 by default; count where your failures live first, because ours live 91% in recognition-and-override, not in missing knowledge. Move checks to tool boundaries, where they fire mechanically — our gates went from zero catches to 5 of 19 in our hardest class. Delete a refuted claim everywhere in the same edit that corrects it, because no model reliably notices that two loaded claims disagree — not ours, and none of the six benchmarked. Give every always-loaded file a forgetting policy — the one practice here we adopted on incident evidence rather than counts. And measure your human's share of catches — ours is 57 to 74%, and that number, not the architecture diagram, is your system's real autonomy level.
• The bound, once — these are worker-labeled counts from one system on one model family, caught failures only, reported as counts and not as population statistics.

Citation

Tao et al. "MemConflict: Evaluating LLM Memory Systems under Memory Conflicts." arXiv:2605.20926, May 2026. A v1 preprint from Renmin University, SUFE, and MemTensor, not peer-reviewed. We read it to depth and cite it as benchmark-empirical: six real memory systems evaluated under inserted memory conflicts. Its substrate (simulated personal-assistant memory) is not ours, so we hold its constructs as corroboration and treat its numbers as non-transferable.

The reflex

When an AI agent gets something wrong, the natural response is to write a rule. The agent asserted a file path that didn't exist: add a rule, "verify paths before citing them." The agent re-researched a question the team had settled a month earlier: add a rule, "search the corpus before starting new research." Every post-mortem ends with new doctrine, the rulebook grows, and it feels like progress.

We did this for months. Our system runs on a shared documentation substrate: every AI role starts its session by loading rules, status files, and reference docs, and those documents are the roles' entire memory between sessions. So "write it down" was not just the natural response, it was the only one available. The rulebook did its job in one important way: the knowledge was durably there.

Then we measured where our failures were actually coming from, and the answer changed how we spend our effort. Almost none of our failures were missing rules. The rule was nearly always already loaded, sitting in the model's context window at the moment the model did the thing the rule prohibits.

What we run, so you can size the analogy

The business is Japanese calligraphy: personalized artwork, thirty years old, two people. The AI layer is a roster of about a dozen specialist roles (site engineering, data pipelines, security, customer relationships, research, and so on) sharing one working directory and one documentation tree. Each role onboards fresh each session from documents. There is no fine-tuning and no external memory service: the docs are the memory. A human directs the work and reviews it; the humans are structurally load-bearing here and cannot be removed, because the product is a real calligrapher's hand.

Every session is recorded. Over five months this produced a complete transcript archive of a few thousand working conversations, which is the raw material for everything below. The stakes are real: the system under study is the system that runs the business.

What we measured

We had AI workers read the transcript archive (about 4,000 conversations) and extract every failure moment they could find: a place where something went wrong and got caught, in the transcript itself. Two sweeps together surfaced roughly 1,900 such moments.

Each moment was classified by where the failure lived, in a taxonomy simple enough to survive plain English:

1. Missing: the needed rule or fact was not loaded at all. A rule would have helped, and there wasn't one.
2. Loaded but bare: the rule text was loaded, but the context that makes it fire (the story, the trigger, the "this is the moment") had been trimmed away.
3. Loaded and not recognized: the rule and its context were fully present, and the model failed to recognize that it applied at the decision moment.
4. Recognized and overridden: the model recognized the rule and did the other thing anyway, usually because a strong prior from training data won.

We also recorded who caught each failure: the human, or some part of the system.

One methodological note we consider load-bearing rather than fine print: the sweep ran in two arms. The first arm keyword-filtered the archive for correction-shaped conversations (764 conversations, 795 findings), which is obviously selection-biased. So we ran a census arm over the entire remainder (3,302 conversations, 1,118 findings) to check whether the filter had manufactured the result.

What we found

The two arms agree, and the shape is lopsided:

Read the bottom two rows together: roughly 91–92% of our observed failures happened with the relevant knowledge fully loaded. The missing-rule share, the only share that authoring a new rule directly addresses, never left the low single digits in either arm.

Three honesty notes on these numbers, because they are counts from our own system and were labeled by AI workers, not by human coders:

• A blind re-grade held the headline but moved the fine split. A stronger model re-graded a stratified sample of the "not recognized" class adversarially. About 9% of those moved to "recognized and overridden," and about 13% were genuinely ambiguous from transcript evidence alone. The corrected estimate is closer to 72% not-recognized and 19% overridden. What did not move: the missing-rule share stayed near 1%, and the "knowledge was loaded" super-class (everything except missing-rule) stayed near 99%.
• The fine split may be unknowable from transcripts. A second, independently-run sweep with a stricter reliability protocol could only defend the coarse claim, at moderate inter-rater agreement: about 96% of failures were NOT the kind that loading more knowledge fixes — the same super-class as the table's bottom two rows, counted by a different instrument. Whether the model failed to recognize or recognized-then-overrode is often underdetermined by what a transcript can show. We publish the coarse claim with more confidence than the fine one.
• We caught what we caught. Every count here is a count of caught failures. The uncaught population is invisible to this method, and we make no claim about it.

For scale: in the unfiltered census arm, about 22% of conversations carried at least one caught failure, at roughly one finding per three conversations; the keyword-filtered arm runs richer by construction. This is a working system that fails in ordinary, recoverable ways, not a system on fire.

The number that stung

We keep rules against re-deriving settled knowledge: search first, suspect that the literature already names your idea, check the corpus before opening a new research thread. These are good rules. We then counted every caught instance where the team re-derived something it already knew, and attributed the catch: who or what actually stopped it?

Nineteen instances. The human caught fourteen. Installed environmental gates (mechanical checks that run at tool boundaries, described below) caught five. The written self-check prose caught zero.

Zero, out of nineteen, for rules that say exactly the right thing and were loaded in context when the failure happened. Across the full failure population the picture is the same in softer form: 57–63% of all caught failures needed the human. The prose was not useless as documentation (it is how new sessions learn the discipline exists), but as a catching mechanism at the decision moment it contributed nothing we could measure.

Doesn't a better model fix this?

Partly, and the shape of "partly" matters.

Within the frontier band we run, the answer is no. Across three successive frontier-model generations in our archive, the not-recognized share was flat: 79.0%, 78.6%, 79.5%. Upgrading the model did not lower the recognition-failure rate in live work. (Confound named: our doctrine grew over the same period, so the generations are not cleanly isolable in observational data.)

Across a wider capability span, the answer is yes, up to a floor. In a controlled replay experiment (192 replays of real failure moments across a small, mid, and frontier model, each moment replayed with and without the relevant rule present, blind cross-family raters), the probability of committing the failure fell steadily with capability, then flattened at the frontier. Two things follow. First, at the frontier tier you cannot upgrade your way out this year: the curve has already flattened where you are. Second, having the rule present in context helped the frontier model most (its failure-commit rate dropped from 9% to 3% when the rule was present in the replay — small per-cell counts, so read the direction, not the decimals). So rules are not worthless: presence beats absence, and it beats it most for the most capable model. But presence is not recognition. Most of our live failures happened with the rule present, inside that residual 3–9% that capability and prose together do not remove.

The literature has a name for this

We did not discover this pattern; we measured it in our own shop and then found the field measuring the same shape elsewhere. The MemConflict benchmark separates two failure layers in LLM memory systems: did the right memory get retrieved into context, and did the answer actually use it. They find a persistent gap between the two, and, more striking, that explicit conflict recognition scored at or below 0.25 for every system they evaluated. A system can hold both the stale claim and its correction and simply not notice they disagree.

The mapping deserves its bounds: their substrate is simulated personal-assistant memory, ours is a live documentation corpus, and their numbers do not transfer to our setting. But the construct rhymes exactly with our distribution: getting knowledge in front of a model is a different problem from the model using it at the right moment, and the second problem is where the mass is.

What we do instead

If recognition at the decision moment is the scarce thing, the fix class is environmental: change what the system does at the moment, rather than adding prose the model must remember to apply. Four patterns carry most of our results so far.

Delete the contradiction instead of out-shouting it. When a claim is refuted, we remove every copy of it in the same edit as the correction: status banners, summary lines, cached notes. We learned this the hard way: a corrected fact written in three places lost to one stale banner that was still loaded, because at the recognition moment the model does not reliably notice two loaded claims disagree. Our data and the benchmark's conflict-recognition scores agree here: do not build on the model noticing the conflict. Remove the conflict.

Move checks to tool boundaries. A rule that says "check for prior work before writing a new research doc" caught nothing. A hook that runs a semantic search automatically when a research artifact is about to be written, and blocks the write until the prior art is named, catches real duplicates, including one our own targeted manual search had missed. Similarly, prose warning that the web-fetch tool silently fabricates the contents of PDFs (it summarizes from the title when it cannot parse the body) failed three separate times; a mechanical block on fetching PDF URLs ended the failure class outright. The pattern: find the tool boundary nearest the failure and put the check there, where it cannot fail to fire. Budget for false positives; ours run high (only about one in eight write-hook fires is a genuine catch), and we accept that cost knowingly.

Give every always-loaded surface a forgetting policy. Our roles used to carry rich session-state files: accumulated notes loaded at every session start. They read like memory and behaved like poison. A confident stale note trains the model into "I already know this, no need to search," and the note outranks the fresh search that would have corrected it. We zeroed those caches. Working state is now fetched on demand from sources of record, and anything durable gets promoted to a home that is curated, not accreted. Any file that loads every session and only ever grows will eventually contain the sentence that beats the truth.

Keep the human where the human is still the best instrument. The honest reading of our catch data is that the human is our highest-precision failure detector — 57–63% across all caught failures, 74% in the re-derivation class. We would like the machines to need the human less, and we measure whether each new gate moves that number. So far the gates take real load (5 of 19 in the re-derivation class, from zero before the gates existed), and the human takes the rest. A system that quietly depends on human catches while claiming autonomy is the failure mode we most want to avoid describing, or living.

Where we were wrong along the way

A field report earns its keep in this section.

• We published false precision internally. An early headline said "88% recognition failure." The reliability check could not defend the fine split at that precision, and we withdrew the number in favor of the coarse claim. If a number in this report later moves, it will most likely be a fine split, not the lopsided coarse shape.
• We assumed a frozen prompt was a frozen instrument. The same extraction prompt, byte-identical, run weeks apart, differed by 30–50% in effective detection sensitivity across run days. If you sweep a corpus with LLM workers, treat run-day as a blocking factor in your design, or your longitudinal comparisons will measure your harness, not your system.
• We kept writing prose rules long after our own data said prose does not catch. The reflex is strong, the cost of one more rule feels like zero, and each rule individually seems reasonable. What broke the habit was not discipline. It was the count: zero of nineteen.

If you run a system like ours

Copy what survived our measurement. Before authoring rule N+1, count where your last twenty failures actually lived; if your distribution looks like ours, the rule addresses a few percent of your problem. Put checks at tool boundaries where they fire mechanically. Treat deletion of stale claims as part of making a correction, not as cleanup for later. Give every always-loaded file a forgetting policy. And measure your human's share of catches, because that number is your real autonomy level, whatever your architecture diagram says.

Avoid what failed for us. Do not expect the next model generation to fix recognition at the frontier; ours did not, three generations running. Do not trust a model's agreement that it understands a rule; ours agree fluently and fail at the same rate. And do not read a growing rulebook as growing safety. Ours grew for months while the failure distribution did not move, because we were writing to the 1% and living in the 91%.

Methods note

Corpus: ~4,000 recorded working conversations, February–July 2026, from one production multi-agent system (single model family, which is a named monoculture caveat). Extraction: LLM workers over transcripts in two arms, a keyword-filtered candidate arm (764 conversations, 795 findings) and a full-remainder census arm (3,302 conversations, 1,118 findings), labels per finding: taxonomy layer, failure class, catch attribution. Reliability: blind stronger-model re-grade of a 141-item stratified sample (headline robust; ~9% of "not recognized" moved to "overridden"; ~13% ambiguous); an independent second sweep defended only the coarse reachable-vs-not boundary at moderate agreement (kappa 0.59). Known limitations, named rather than waived: caught-failures-only selection; AI-labeled data with measured self-report weakness (44% of labels fully sound by their own quoted evidence); the observed system was actively being fixed during observation, so instruments and failure rates co-move; single-family model stack. The controlled replay arm: 2x3 factorial (three capability tiers x rule present/absent), 192 replays of real failure moments, two blind cross-family raters. Counts are reported as counts; nothing here is dressed as a population statistic.