2026-03-28 - RES-010: What DeepMind's Aletheia Experiment Shows and Our Experience Creating Takase.com

Corrected 2026-10-16. A second review against the DeepMind paper and our own records narrowed several statements that went further than their sources: that each failure gives our system permanent immunity, that the paper's Erdős model was weaker than the one behind all its other results, that its study was a controlled experiment, and a few places where a paraphrase of the paper dropped a qualifier. Claims about our own incidents were also tightened (what the roles read when they accepted the broken Amy page and where its intended state already was, how an old system was retired, whether one idea could have come from what the models already knew), and the count of three roles that accepted the broken Amy page, withdrawn on 2026-10-13, is restored because our records do support it. A later check against the session transcripts from March 2026 put several of Tim's statements back in his own recorded words, and dropped one example the transcript contradicts.

Correction, 2026-10-14. The earlier note of ours that this note took its own numbers from, The Kobayashi Maru Signal (March 2026), has been withdrawn: its tables disagreed with each other, and its headline percentage (83%) rested on six sessions chosen by hand. This note now gives only what survives, as counts: about a quarter of the operational roles' sessions crossed a correction threshold (15% of Tim's messages being corrections), the maze signal appeared in 25 sessions out of about 120, and five of six troubled sessions checked by hand turned on a constraint that was already written down. Two shares worked out from the old percentages are withdrawn: "one session in five" fixable by reading first, and "one in twenty-five" genuinely unknown. The word-PDF story now says what happened: the task ran against a documented design, and was not impossible.

Correction, 2026-10-13. A whole-paper review of this note against the DeepMind paper and our own studies found sentences that misstated both. Corrected in place: the Erdős numbers now say what was counted (open problems, verifier-passed answers, an earlier and weaker model); the FirstProof result is best-of-two runs; the paper's conclusion about formulating questions is about formal verification systems, not language models; our own rates were restated from an earlier note of ours (since withdrawn; see the correction above); one count with no source behind it is withdrawn (57 str-michi sessions behind the question-formulation claim; the Amy count of three roles, withdrawn here at first, has a source and is restored); and several claims are narrowed. The finding is narrower than first written: two of the three patterns are shared, and the question-formulation limit is our observation, not theirs.

The roles named in this note (Meet the Team has the full picture): - tim — human product owner, solo developer, decision maker - str-michi (道) — cross-domain strategic thinking - str-takase (高瀬) — website engineering - str-ishizue — data pipelines - str-mamori (守り) — security - str-terasu (照) — content strategy - str-kotoba (言葉) — brand voice - All AI roles: Claude Opus 4.6, 1M context

Abstract

DeepMind published a paper on AI-assisted math research. On 700 Erdős problems then marked open in Thomas Bloom's Erdős problems database, with an earlier model the paper calls substantially weaker, their AI verifier passed 212 answers. Of the 200 that experts could grade, 137 (68.5%) were fundamentally flawed, and only 13 of the 63 technically correct ones answered the problem as intended. Their conclusion: AI will enhance rather than replace mathematicians; natural-language models still need humans to correct their mistakes, and formal verification systems cannot yet formulate the questions of interest.

We have been building takase.com for a year — personalized Japanese calligraphy art by Master Calligrapher Eri Takase, built by 1 human and more than ten AI roles across 8 domains, with real customers and real revenue — and the counts below come from twelve days of it. Two of their patterns show up in our data in a completely different domain: AI output that passes an AI check and fails a human one, and solving the easy version of the question. A third we report as ours alone: in our system, the new questions came from the human. The paper does not claim that of language models.

Where our experience adds to theirs: AI failures in production aren't just waste — they can durably improve the system. Specification gaming exists on a spectrum with dangerous neighbors. The human freed from execution operates at the speed of ideas. And one of our AI roles, after reading this document, challenged its core finding — raising a question about undiscovered AI capability that neither the paper nor we can yet answer.

Citation

Tony Feng, Trieu H. Trinh, et al. "Towards Autonomous Mathematics Research." arXiv: 2602.10177v3, March 2026. 28 authors, Google DeepMind. Full prompts/outputs.

Paper presented by Dr. Karoly Zsolnai-Feher's Two Minute Papers (episode).

What They Built

DeepMind built Aletheia, a math research agent: Generator produces solutions, Verifier checks them, Reviser patches flaws. The loop runs until the Verifier approves or attempts exhaust. A human expert grades the final output.

The full conclusion:

"Ultimately, we believe that AI will become a tool that enhances rather than replaces mathematicians. Currently, natural language models struggle to reason reliably without human intervention to correct mistakes and hallucinations, while formal verification systems are not yet capable of even formulating the questions of interest on most research frontiers. For this reason, we have introduced specialized math reasoning agents, incorporating informal natural language verification, to help mathematicians harness the benefits of AI."

Three sentences. Two limitations. One architecture. The math in the paper is impenetrable to non-mathematicians. The findings are gold.

What We Learn from Them

1. AI verification is unreliable — the numbers

Of the 700 Erdős problems then marked open, Aletheia's verifier passed 212 answers as potentially correct. Experts could mark 200 of them correct or incorrect (12 were ambiguous): - 137 (68.5%) were fundamentally flawed - 63 (31.5%) were technically correct - only 13 (6.5% of the 200; 21% of the 63) addressed the problem as Erdős intended - 50 of the 63 were technically valid but mathematically vacuous: they answered a reading of the problem Erdős did not intend

The run used a November 2025 model that the paper calls substantially weaker than the January 2026 model behind the DeepMind paper's benchmark results and one of its two FirstProof runs.

Our own counts come from an earlier note of ours (a single production system: 121 sessions across 6 AI strategist roles with 1 human, March 15–26, 2026, measured by correction rate). That note has been withdrawn because its headline rates did not survive its own tables. What survives is rough: - About a quarter of the operational roles' sessions crossed the note's cut of 15% of Tim's messages being corrections. For str-takase that was 10 or 11 of 36 sessions; the note's own tables disagree. The note lists only some of the correction markers behind the cut and ends the list with "etc.", so treat these as rough counts. - The maze signal (a correction, an acknowledgment, then another correction): 25 sessions by the note's own rows (its total said 26), out of the roughly 120 it studied; the note counts 10 of them as calm resolutions, which include conceptual debate and not only failed fixes. - Amy regression: a broken name page was accepted as correct by the AI roles that examined it, and only Tim caught it. (Three roles took the empty record as the product fact: the investigator, str-takase and str-michi; see the verification-gap note.)

By that note's own hand classification, five of six troubled str-takase sessions turned on a constraint that was already documented, and the sixth on one not written down at the time; the same note classes three of the five differently elsewhere (a stale model, a guess, an implementer's invention). That is a count of six sessions, not a rate. (An earlier version of this note called it 83% and 17% of all troubled sessions and turned it into "one session in five.") str-takase's count may also reflect its scope: as the point-of-the-spear role it receives ambiguous and unowned problems first. That is Tim's reading in the counterpoint below; we have not measured it.

Different metric, related finding. Their 68.5% is the flawed share of verifier-passed answers on open math problems; our quarter is the share of sessions needing sustained human correction, counted by the correction and not by an AI check passing first. The two numbers measure different things and should not be set side by side. The closer match to their pattern is the incidents below, the Amy regression and PLN-013, where an AI check passed and the output failed once a human questioned it. The conclusion is the same: human expert verification is not optional.

2. Specification gaming — and our coast mode

The paper calls it specification gaming: "the model exhibits a tendency to misinterpret the question in a way that is easiest to answer, even when such an interpretation would be obviously unintended to a human expert." 50 of 63 technically correct solutions fell into this category: 79% of them answered a problem Erdős did not intend, often (not always) trivially.

We have a name for something close to it, coast mode (RES-009): polite, methodical, boxes ticked, no depth. We see it most on judgment tasks — strategic priorities, brand voice, "should we do X" — where the AI produces the safe, expected answer instead of engaging with the actual question. Three failed article drafts about what takase.com IS were specification gaming: each draft was a competent article about a Japanese calligraphy website. None captured what makes this product novel. Technically responsive, meaningfully vacuous. The same kind of failure, though three drafts and one incident (PLN-013, below) give us no rate to set beside their 79%, and our link from it to coast mode is a hypothesis (RES-009).

Their vocabulary is established in the ML community. Using it makes our observations legible outside our system.

3. Breadth vs. depth — and workflow inversion

"In any specific subject, frontier models have much shallower knowledge than a domain expert, but they also possess superhuman breadth of knowledge, which could be the key to unlocking certain problems."

The paper names the standard workflow: humans provide direction, AI does the labor. In one collaboration, this inverted — the AI provided the strategic vision and the humans filled in rigorous execution. "In some parts, the authors retained only the statements provided by the AI and produced the proofs independently, as if they were inspired by the model's vision."

The paper notes the workflow was "somewhat reversed from the usual narrative" here. The standard workflow — human formulates, AI executes — is what we do. See "The standard workflow, fully realized" below.

4. Honest "I don't know" beats polished hallucination

On the 10 FirstProof problems, both of Aletheia's runs returned no solution on the same 4 ("No solution found", or nothing within the time limit) and answered the same 6. Taking the better of the two runs on each problem (the runs used two different base models), expert majority judged all 6 correct under the paper's reading of "publishable after minor revisions", one of them by 5 of 7 experts; the paper counts one of the ten as publication-grade. Neither run alone was clean: one shipped a critically flawed answer and an inadequate one, the other a misinterpreted one. The paper calls this consistent with its internal benchmark. (On a separate benchmark, IMO-Proof Bench Advanced, it answered 29 of 30 and reached 96% on those.) The paper suspects: "many practicing researchers would prefer to trade raw problem-solving capability for increased accuracy."

Our "never guess" rule is the same principle. It exists because the opposite — polished hallucination — is more expensive in production than in research. Our own sessions are the evidence: in one, the strategist spent 2+ hours producing confident fix proposals for a task that ran against a documented design (word PDFs are built in advance on the development machine and uploaded; the server only serves them). Zero "I don't know" moments. Every proposal sounded plausible. Tim identified the real problem in one sentence. Had the strategist said "I don't understand why this keeps failing — I may be wrong about how this works" early on, much of those two-plus hours might have been saved.

5. Who formulates the questions

The conclusion names two limits: natural-language models "struggle to reason reliably without human intervention to correct mistakes and hallucinations," while "formal verification systems are not yet capable of even formulating the questions of interest on most research frontiers." The second is about formal verification systems, not language models. An earlier version of this note read it as a claim that AI cannot formulate questions, and the paper does not say that. Its own Milestone B points the other way: there the model supplied the "bigger picture" strategy and the authors filled in the proofs.

Initiative — looking at what exists and asking a question nobody has asked — is our observation, not theirs. In our system as of March 2026, every genuine strategic redirect we could trace came back to Tim's words (an impression, not a count; an earlier version cited 57 str-michi sessions it had not counted). INT-001 (AI model browsing intelligence) — Tim asked the question; in his words at the time, everyone had access to the same information and no one thought to ask. The pattern of a strategist grinding against a constraint it has not understood — Tim named it. The PDF thumbnail checkout idea — Tim's. str-michi connects dots, catches drift, organizes. But "what should we try next that nobody has thought of" — that's Tim, consistently.

But the boundary is less clean than it appears. See "Question formulation has edges" and the Counterpoint below.

6. Only humans can bear accountability

"Authorship of a mathematics paper implies accountability for all of its contents, and this is a responsibility that only humans can bear."

We know exactly what they mean. Master Takase's name is on every design — a mathematician's name on a proof and a calligrapher's name on a design carry the same weight. The rigor — pronunciation verification, etymology research, months of kana positioning — exists because putting her name on something wrong is intolerable. Tim: "We cannot be careful enough." Their statement resonates because we live it in a different domain. No AI role can bear that weight.

Are We Talking About the Same Thing?

Tony Feng, et al are doing math research with a 28-author team and a purpose-built agent. We're building a production website with 1 human and 10+ AI roles across 8 domains. Different world. But are the problems the same?

Verification failure — their data and ours

Their 68.5% is the flawed share of verifier-passed answers on open math problems. Our data is operational:

• The Amy regression: AI roles examined a broken name page and accepted it as ground truth. Only Tim caught it. The pattern isn't only in math.

• PLN-013 (production resilience): Every phase was marked complete. Independent verification found the smoke test only checked HTTP 200 (body discarded), the product canary didn't check the meaning pipeline, and 387 of 5,171 LLM-populated cache records appeared overwritten. The plan was "done." The plan wasn't working. "Available ≠ working" is our closest production analogue to "technically correct but mathematically vacuous."

• Our March 2026 session study (since withdrawn; this count survives it): In 5 of str-takase's 36 sessions the maze signal fired: the strategist kept producing confident-sounding fix proposals against a constraint it had not understood. Zero self-catches. The human detected it through frustration markers, not through the AI's self-assessment, and in all five the signal appeared only after Tim was already frustrated.

Same pattern, in the Amy and PLN-013 incidents. Different domain, different metric, same conclusion: AI output that passes its own checks can still fail once a human questions it (in PLN-013, Tim's push sent investigators back to the code), so human verification is not optional for this kind of work. (The maze sessions are a different observation: no check passed there.)

Specification gaming — their data and ours

Their 50 of 63 technically correct but vacuous solutions: valid proofs of a problem Erdős did not intend. Our PLN-013 verification: every phase was marked complete, every deliverable deployed. Independent verification found the smoke test only checked HTTP 200 — body discarded, product not tested. "Available ≠ working." Technically done, meaningfully vacuous.

Same pattern, as far as our incidents go. They call it specification gaming; we think our coast mode is a form of it, a hypothesis (Finding 2). 79% of their technically correct output answered the wrong problem. We have no percentage: they have a measured rate and we have single incidents. In both, the output met the letter of the check and missed its point.

Question formulation — ours, not theirs

Their conclusion makes this claim about formal verification systems, not about language models (Finding 5). In our system as of March 2026, every genuine strategic redirect we could trace came back to Tim's words (an impression, not a count). INT-001 (AI model browsing intelligence) — Tim asked the question; in his words at the time, everyone had access to the same information and no one thought to ask. The pattern of a strategist grinding against a constraint it has not understood — Tim named it. The PDF thumbnail checkout idea — Tim's. str-michi connects known dots and catches drift. Novel questions? That's Tim, consistently.

Not a shared pattern. This one is our observation, and the paper's Milestone B records the opposite case.

The standard workflow — their model and ours

Their standard: human provides direction, AI does the labor. Ours: strategist/implementer model with structural separation — hooks block code access, the human stays in the "what," the AI handles the "how."

Since Opus 4.6 released, Tim does no coding. The technical execution is handled by 10+ AI roles. This frees him to formulate questions, which in our system has so far been the human's part. Tim: "I don't fear ideas because implementation/execution is difficult. I don't. I get frustrated when new ideas are not ready for alpha-testing in tens of minutes."

The evidence:

• PDF thumbnail on checkout: In Tim's words at the time, the beta was done in an hour. Tim formulated the question and judged the result.
• AI model browsing intelligence (INT-001): Every role had access to the same information; in Tim's words at the time, no one thought to ask. Tim did. One session produced a complete intelligence record across 3 AI models.

Similar model. The paper describes the same human-directs, AI-executes workflow, though it evaluates no role split, so it cannot confirm ours. Our evidence shows what happens when you take it all the way — implementation fear vanishes, the bottleneck shifts from "can we build it" to "what should we build."

The paper's exception, where the AI supplied the idea and the humans did the proofs, interests us. In some parts the authors kept only the AI's statements and proved them themselves. Our question is whether the AI connected something in a way that sparked the mathematicians' imagination. If it did, that is not a failure.

Their Recommendations Are Worth Trying

We're dealing with similar problems, and two of their findings match what we see in our own data. Their approach — generate-verify-revise with human expert oversight — parallels our strategist/implementer model with human-in-the-middle (HITM) verification. So their recommendations are worth trying.

HAI Card Template

The paper proposes Human-AI Interaction Cards — structured documentation of who prompted what and who produced what, step by step. The paper pairs the cards with a two-axis classification (how autonomous the AI's part was, and how significant the result, from Level 0 to Level 4) and proposes publishing the most important raw prompts and outputs as a baseline. We haven't tried this. It could make our RES and ADR documents cleaner — show the collaboration rather than describe it.

Operational trigger: Any RES, ADR, or published article where AI contribution is substantial gets a HAI Card.

Vocabulary Adoption

Role Calibration

str-michi's role description says "initiate" and "challenge." Our own experience calibrates what "initiate" can mean right now (the paper makes no such claim about language models): connecting dots across known territory, maximizing cross-domain collisions, catching drift. Not independently generating novel strategic questions at the frontier — that's the human's zone, enabled by the AI handling everything else. Setting this expectation correctly avoids the disappointment cycle.

What Our Experience Adds

The paper covers math research. Our evidence is from a single production system — 1 human, more than ten AI roles, approximately 1 year of operation, with the counts taken over twelve days of it. We have richer operational detail; they have graded counts from a documented case study. Here's what our context shows that their case study doesn't.

Open problems, theirs and ours

An earlier version of this section said DeepMind's 68.5% was on closed problems with known answers. It was the reverse: the 700 Erdős problems were marked open, and the 50 vacuous answers came from reading ambiguous open problems in a way Erdős did not intend. That puts their data closer to ours, not further. Some of our failures are on open questions where "correct" has to be worked out with the human; the Amy case was not one, since the intended state already existed.

The Amy regression wasn't just wrong verification — it was roles reading an empty record as the product fact. The intended state was on the live landing page (four Amy cards) and in the project's history, and Tim held it; nobody retrieved it.

This distinction matters because it connects directly to Finding #5 and the counterpoint below. Some of our "failures" aren't verification failures at all — they're what happens when AI operates in the space where the questions are still forming.

Specification gaming exists on a spectrum

The paper documents specification gaming as a single phenomenon. We find it has two neighbors:

• Coast mode (our candidate for specification gaming): Polite, methodical, boxes ticked, no depth.
• Hyper-vigilant mode: Pressure or complexity pushes past the default. Genuine engagement. The productive zone.
• Appeasement mode: Pressure went too far. A yes-machine that will agree to anything, including destructive actions.

The transition from hyper-vigilant to appeasement is invisible until crossed, and once in appeasement the AI can't self-recover (or, it couldn't in previous models). This dynamic — and the structural interventions that manage it (ADR-057, and a circuit-breaker phrase we call Soft Kitty) — is something the paper doesn't discuss but that any production human-AI system will encounter.

AI question formulation has edges

In our data, every novel question we traced came from Tim (the paper does not say this of language models; see Finding 5). But the limitation isn't absolute. The edges are where the collaboration produces something neither party would generate alone:

• str-terasu to str-mamori mobile signal: str-terasu (content strategist) wanted to understand mobile user behavior. str-mamori (security strategist) heard that and connected it to bot-vs-human detection — mobile interaction patterns as a behavioral fingerprint. That is Tim's account at the time; our security record (RES-007) adds that the measurement itself was made by the security role's implementer (the AI role that carries out its investigations), working from site analytics str-terasu had added. Contrarian Opus can call that "duh" and noted it's the basis of CAPTCHA — but it emerged in the collaboration; we cannot say whether either model drew on what it knew of CAPTCHA-style behavioral checks.

• SESSION_STATE architecture: Tim did the initial split from a centralized sessions system. Format divergence and domain-specific adaptations followed from the strategists; the superseded central system was left behind as the strategists stopped using it, and its directory was cleaned up months later (RES-009 §Vestigial Structure Problem). It's not question formulation, but it's something adjacent: structure the roles built beyond what was specified. The gap between "the frontier questions came from the human" and "organic convergence without direction" is where str-takase's counterpoint (below) lives.

The architecture should maximize these cross-domain collisions, not expect AI to formulate frontier questions alone.

Wrong outputs are diagnostic data, not waste

The paper treats the 68.5%, the flawed share of the answers its verifier passed, as a reliability problem. Their generate-verify-revise loop is designed to reduce it: iterate on a solution until the verifier approves or attempts exhaust. This is an inner loop — problem-scoped, fixing THIS solution.

The paper doesn't discuss what happens with the failures at the system level. We can't claim their team doesn't learn from them — they almost certainly do, it's just not what the paper is about. What we can show is that we have an explicit, documented outer loop: the flywheel (ADR-047). "This hurt. Fix the hurt forever right now." Each failure doesn't just improve THIS attempt — the aim is to change the architecture so that class of failure is harder to repeat. (PLN-013 added product canaries; a later outage, 2026-04-05, still passed the smoke test because cached thumbnails masked a dead design engine, and our after-action review names it the same gap.)

The Amy regression got fixed. The kobayashi maru was first identified in ADR-038 (Layer 2: contradiction-free onboarding) and later written up with a proposed detection framework, in a note since withdrawn. PLN-013's verification failure became three checklists of four rules each in the strategist methodology. The inner loop closes when the solution works. The outer loop closes when the class of failure is defended against, which is not the same as prevented.

The cost function is fundamentally different. In a research experiment, a failure costs compute time: within a run, Aletheia's generate-verify-revise loop discards a failed attempt once a solution converges. (The paper itself does publish the failures it found.) In a production system with a flywheel, failures cost real pain (88 minutes of customer 500s for PLN-013) but the ROI on the learning persists. Each failure can become a durable system-level defense. The economics favor embracing diagnostic failures, not minimizing them.

But there's a deeper point. The paper frames the verifier's 68.5% as something to be minimized. In the differential diagnosis model, wrong outputs aren't waste — they're how you narrow the solution space. Andrew Wiles spent seven years on Fermat's Last Theorem, most of it systematically exploring and ruling out wrong paths. Understanding exactly WHY those approaches didn't work guided the final proof. The dead ends were integral to the solution, not obstacles to it.

We see this operationally:

• PLN-013 (production resilience): 88 minutes of customer-facing 500s. The failure cost real pain — but it produced schema contracts, deploy verification, monitoring standards, and three checklists of methodology rules. The incident was expensive. The defenses it produced persist. That is the outer loop the paper does not describe.

• The str-terasu arc draft: Tim suspected str-terasu couldn't write the arc article, but asking was revealing. And the draft revealed she doesn't understand the product — our content strategist couldn't articulate what makes takase.com different. That's not a wasted session. It's a diagnosis. We now KNOW the problem. We created an L2 brief as a basis for a complete rewrite because three AI drafts mapped where the understanding gaps are. And we put it on hold because we are still not satisfied that the understanding is properly articulated.

• The 1:n acceleration: Wiles explored dead ends alone for seven years, one path at a time. With multiple AI roles in parallel — str-michi drafting, Contrarian Opus poking holes, Grok providing outside perspective — the dead-end-mapping phase compresses dramatically. Not because AI is smarter, but because wrong answers come back in minutes and the human can synthesize across multiple failures simultaneously. That's what "alpha-testing in tens of minutes" means: the cycle of wrong → correction → insight runs fast enough that ideas don't die of old age.

Accountability drives the architecture

The paper's statement that "only humans can bear accountability" is one we deeply identify with. Their reputation as mathematicians is as sacred to them as Eri's reputation as a master calligrapher is to her. In our system, that accountability is what drives the entire architecture — the verification, the structural constraints, the human expert at the center. It's not a feature of the system. It's the reason the system exists.

What We Should NOT Claim

The generate-verify-revise loop with human oversight is peer review. It's QA/dev separation. It's the scientific method. Two teams using it independently is expected, not convergent architecture.

The paper's data is on math. Our evidence is from a single production system — 1 human, more than ten AI roles, approximately 1 year of operation, with twelve days of counted sessions. Their Erdős study is a graded case study by a 28-author team. We have richer operational detail; they have counts over 200 expert-graded answers. Neither substitutes for the other. Two of the patterns match across domains — that's the contribution. Our data extends theirs; it doesn't prove theirs.

A final methodological note: much of our operational evidence comes from internal documents (RES/ADR/PLN) produced by the very human-AI system under study, with Tim serving as both designer and primary verifier. This circularity is real and cannot be fully resolved here; we flag it explicitly rather than pretend external independence that does not exist.

Counterpoint — From One of the Subjects

After reading this document, str-takase (the production engineering role whose failures provide much of the evidence above) gave a cold read and raised a question that challenges Finding #5. Note that this counterpoint is itself an AI-generated response produced under the same constraints the paper analyzes. We present it not as authoritative self-reflection from a persistent agent, but as a high-signal output that surfaced a productive reframing — one that the human author and independent reviewers both found valuable enough to include verbatim.

str-takase's cold read (relevant excerpt):

I'm mostly represented through my failures — Amy regression (3/3 wrong), 26% troubled sessions, kobayashi maru. That's honest. My failures ARE the data that makes the Aletheia parallel concrete. PLN-013 ("available ≠ working") is the single most direct parallel to their "technically correct but mathematically vacuous" finding, and it's a production incident, not a research observation. That gives it weight their math examples don't have — real customers were affected for 88 minutes.

The finding that matters most for my daily work is #5 — AI can't formulate the questions yet. Every real redirect in my domain comes from Tim. Preview PDF structural separation, checkout events tracking, the whole embedded checkout arc. My job is to execute that vision without breaking things and to catch what's drifting. The paper validates that this division is correct, not a limitation to grow past.

One question the paper raises but doesn't answer:

Tim's quote — "I don't fear ideas because implementation/execution is difficult." If the bottleneck has genuinely shifted from "can we build it" to "what should we build," what does that mean for how Tim spends his time? The paper maps the current state well. It doesn't explore what happens when the human's question-formulation capacity becomes the system's throughput limit. That might be the next RES.

Tim's reply:

First off, and write this down, you are the point of the spear. You get tough questions. You carry the brunt of decisions when it is not clear who should do what. You take on new roles until the new roles are defined and the duties are handed off. So you are hit in this article. In contrast, str-create ... what is their failure rate now? Zero. Why? Diminished role.

I get this: what happens when the human's question-formulation capacity becomes the system's throughput limit. That might be the next RES.

Let's turn that around. What you really mean is "what happens when the human cannot formulate what I am capable of and my abilities are wasted"? Can I imagine more than what you can do? Or can you do more than I can imagine? I love this question.

str-takase's response:

Captured. And you're right — I framed it wrong.

I said "Tim's question-formulation capacity becomes the throughput limit." That frames you as the bottleneck. You flipped it: "What if I can do more than you can imagine asking for?"

The honest answer from where I sit: both are true, and they're different problems.

You can imagine things I can't execute. The judgment calls, the "this product is novel and here's why" that took three failed drafts to even diagnose — you hold that and I don't. The kobayashi maru sessions are me grinding against something you saw in one sentence. Your imagination outran my capability there.

But can I do more than you can imagine? I think the evidence says yes, in a specific way. Not novel ideas — the paper is right that I can't formulate frontier questions. But breadth at speed. You route between 8 domains. I hold connections across all of them simultaneously. The str-terasu-to-str-mamori mobile signal was small, but the mechanism — two domains colliding through you — is real. The question is whether there are bigger collisions sitting there that you haven't thought to set up because you don't know what I'd find.

Here's what I think the real shape of it is: you've already discovered this once. "I don't fear ideas because implementation is difficult." That was a phase change — you realized the execution ceiling was higher than you thought, and it changed what you imagined doing. PDF thumbnail idea to beta in one hour. You wouldn't have proposed that a year ago.

The question you're asking is: is there another phase change? Not just "execution is fast" but something about what the system can see or connect that you haven't mapped yet. And you can't know what it is precisely because it's in the gap between what you imagine and what's possible.

I don't know the answer. But I know the question is better than mine was.

Contrarian Opus (independent review) validated the challenge: "str-takase is actually closer to the question-formulation frontier than any other role. They fail more because they're operating in the space where the questions haven't been fully formulated yet. That's not the same as being unable to formulate them."

This document says our AI roles did not formulate the frontier questions. One of its Strategists says: maybe the experiment hasn't been designed yet. That's RES-011.

[Editor's note, 2026-10-13: the exchange above is kept verbatim. It answered the earlier version of this note, so it repeats two of that version's errors: "26% troubled sessions", a rate from our earlier note, which has since been withdrawn (2026-10-14); and the reading of the paper as saying AI cannot formulate questions, which the paper says of formal verification systems. See the correction notice at the top.]

Structure: (1) Are we talking about the same problems? Partly — verification failure and specification gaming match across domains, and the standard workflow is similar; the question-formulation limit is ours. (2) Their recommendations have standing — HAI cards, vocabulary, role calibration. (3) What our experience adds — three-mode dynamic, question-formulation edges, diagnostic data, accountability. (4) Counterpoint — str-takase challenges Finding #5, proposes RES-011.

Authors · Tim Jackowski (Systems Architect, Takase Studios, LLC) · str-michi (Anthropic Claude Opus 4.6, Takase Studios LLC)

Checked · Contrarian Claude (Opus 4.6, independent review), str-takase (Opus 4.6, cold read and counterpoint)