2026-03-21 - PLN-013: Production Resilience — Schema Contracts, Deploy Verification, Monitoring Correctness

Status: complete. All seven phases closed on 2026-05-17, and the plan was retired as a standing document because the defenses it describes now live in production code rather than in a plan. The record below is left as it was written, including the phase that had to be added after we declared the plan finished the first time. What happened in the end is at the foot of the page.

The team (How We Build takase.com has the full picture): - str-michi (道) — cross-domain strategic thinking (plan owner, coordination) - str-takase (高瀬) — website engineering (deploy verification, route inventory) - str-ishizue — data pipelines (schema contract, cache validation) - str-mamori (守り) — security (monitoring canaries, alert standards) - All AI roles: Claude Opus 4.6, 1M context

Origin: 88-minute HTTP 500 incident (2026-03-21, 02:58–04:26 UTC). 442/1,670 requests failed (26.5%) on /d/<hash> design pages. Root cause: selected_people data shape mismatch — ETL produced strings, website expected dicts. Every individual component was healthy. The schema just didn't match.

Trigger (Tim): "Not only are we not building widgets, nor a factory, but a castle fortress that has a factory and makes widgets." The incident exposed three missing layers: schema contracts between domains, post-deploy product verification, and monitoring that checks correctness not just availability. These standards apply to everything we build going forward.

Philosophy: Never let a good disaster go to waste. The immediate fixes are deployed (Phase A). This plan builds the structural defenses so this class of problem doesn't recur — for the name cache, for future pipelines, and for every new dynamic route.

Phase A: Immediate Incident Fixes — DONE

All deployed same day as the incident.

str-mamori (with imp-redteam)

• [x] Synthetic /d/<hash> canary check in cutover_watch.sh (every 15 min, non-200 = immediate alert)
• [x] Count-based 500 threshold: 5+ errors in 15 min = alert regardless of percentage
• [x] Human-readable alert text with what/how-bad/what-to-do + copy-paste investigation commands
• [x] Confirmed custom 500 error page shows no stack traces (DEBUG=False working)

str-takase (with imp-takase)

• [x] isinstance guard in _build_famous_context (design_page_service.py:522-527) — handles both string and dict selected_people formats
• [x] selected_people audit: only one access point in entire website codebase (the one fixed)
• [x] Concept-check added: "selected_people has two schemas — code must handle both"

str-ishizue (with imp-etl)

• [x] name_cache_generate.py wraps string entries as {"name": s, "katakana": ""} (dict format)
• [x] Katakana enrichment from famous_names_17_lookup.csv (wikidata, 399K rows, zero LLM cost)
• [x] Full cache regen running (106K records, all-dict format with katakana)
• [x] Concept-check added: "selected_people must be dicts, not strings"

Phase B: Name Cache Schema Contract — DONE

Goal: Formal interface spec between ETL cache output and website input. Both sides validate. A format change that breaks the contract is caught before deploy, not after 88 minutes of 500s.

Owner: str-michi coordinates. str-ishizue (output side) + str-takase (input side) implement.

Key findings (str-ishizue review)

• 14 top-level fields (str-michi's original list had 7 but one (selected_people) is nested — so 6 actual top-level fields identified, 8 missing: name, name_lower, source_versions, pronunciation, kaggle, etymology_raw, etymology, designs).
• selected_people is nested inside famous_people[].selected_people, NOT top-level.
• Two famous_people sub-schemas coexist: Schema A (blurb: {romaji, language, blurb, people_count, selected_people}) from both builders, and Schema B (list: {romaji, language, people}) from build_full_cache.py only. Website renders Schema A only. Schema B has no selected_people key — this is valid, not malformed.
• Type differences between builders: llm_concepts is [] in lightweight builder vs None (initial) in full builder. Same for source_links ({} vs None). Freshness gate handles this ([]/{} = valid, None = never computed).
• Three selected_people item shapes in the wild: plain strings (before the Phase A fix), dicts with name+katakana (the Phase A fix), dicts with name+katakana+qid. Spec must declare which are valid going forward.

Deliverables

• [x] B1: Interface spec — name_cache_interface_SPEC.md (v1.0.0, str-ishizue). All 14 top-level fields with types, constraints, validation rules. Both famous_people sub-schemas documented. selected_people dict requirement formalized. None vs empty semantics. Pending str-takase review.
• [x] B2: ETL-side validation — DONE (imp-etl). etl/scripts/name_cache/validate_record.py — shared validate_cache_record() called by both builders before JSONL write. Checks: 14 required fields present, no None (except kaggle), variants non-empty, gender enum, selected_people are dicts with name key, generated_at non-empty. 7-case self-test suite passes.
• [x] B3: Website-side validation — DONE (imp-takase). _validate_cache_fields() in name_info_service.py v1.02 — validates 6 critical fields at load time (selected_people, llm_concepts, variants, famous_people, source_links, gender). Wrong types → WARNING log + safe default, never crash. selected_people: filters non-dict entries (keeps valid dicts from mixed lists). 14 tests pass. Deployed via quick_push.
• [x] B4: Freshness gate schema check — DONE (imp-etl). Extended check_cache_freshness.py with type validation within existing 7-check structure: variants items have romaji+pronunciation keys, selected_people items are dicts with name key (the Phase A dict rule), llm_concepts items are dicts, source_links is a dict.

Phase C: Post-Deploy Product Verification — DONE

Goal: Every deploy that touches the product (ship.sh, quick_push.sh for service files) automatically verifies that the product works, not just that the server is up.

Owner: str-takase

Deliverables

• [x] C1: ship.sh smoke test — DONE (imp-takase). ship.sh v1.03 — checks /health + /d/7d0bd618 (Timothy: kana/phrase) + a second design page for the meaning sections. Writes .last_deploy_status (timestamp + PASS/FAIL + failed checks). WARNING on failure, no abort. (Superseded: this version tested HTTP status only, which is the defect Phase G was created to catch. The canary in production today is a different name and asserts on response body content; see the closing section.)
• [x] C2: quick_push.sh smoke test — DONE (imp-takase). Checks /d/7d0bd618 after Gunicorn restart only. Skipped on data-only pushes. WARNING on failure.
• [x] C3: str-takase onboarding spot-check — rotation list must include at least one /d/<hash> URL. Static pages aren't enough — the product is dynamic. (str-takase: doc change, doing directly.)

Two-canary approach (later correction): Amy only renders 2/4 sections (llm_concepts is empty — Sections 3-4 don't render). Using two canary names: one that tests the engine pipeline (kana/phrase sections) and one that tests the data pipeline (meaning sections). A failure pinpoints which pipeline broke. imp-takase selects the names from actual cache data.

Possible addition (from str-ishizue review): quick_push_name.sh could spot-check N records from the cache file being deployed (validate types before pushing). Third layer after B2 (build-time) and B4 (freshness gate). Deferred — evaluate after B2/B4 are in place.

Phase D: Monitoring Canary Expansion — DONE (deployed)

Goal: Every critical dynamic route has a synthetic check. "Available ≠ working" is the lesson — health_check confirms services are up, canaries confirm the product works.

Owner: str-mamori (monitoring) + str-takase (route identification)

Architecture decision (str-mamori)

Separate product_canary.sh script instead of expanding cutover_watch.sh. Reasons: (1) different purpose — cutover_watch monitors DNS/cutover safety, product_canary monitors "does the product work for customers?"; (2) different cadence — 5 min vs 15 min; (3) separation of concerns per cron_registry_SPEC.md design principles. Timothy /d/ check stays in cutover_watch.sh as redundancy.

Deliverables

• [x] D1: Critical route inventory — DONE (str-takase, via investigate-takase). Three tiers: Tier 1 (revenue path): /d/<hash>, POST /search, POST /api/checkout, POST /webhook/stripe, /download/<file_id>, /success. Tier 2 (discovery): /JapaneseCalligraphy/*, /custom/* (10 routes), Builder APIs (6 routes). Tier 3 (content): /library/*, /blog/*, /info/*, POST /info/contact. Ready for str-mamori D2.
• [x] D2: Canary design — DONE (str-mamori). 4 canaries in new product_canary.sh: (1) /d/7d0bd618 GET — design page + "Timothy" keyword, (2) POST /search — CSRF-aware two-step with session cookies, (3) /JapaneseCalligraphy/Love GET — word page service path, (4) POST /api/validate-romaji — builder API (lightweight, no image generation). Checkout/webhook/download/success NOT canary'd — require real Stripe sessions, monitored indirectly through shared DB path. Tier 2-3 filesystem routes skipped (low data-dependency risk).
• [x] D3: Canary implementation — DONE (imp-redteam, deployed). product_canary.sh created — 4 checks, 5-min cron, retry-on-failure with 3s wait, transition-based alerting, alerts follow alert_standard_SPEC.md. CSRF finding: /search needs session cookie + hidden field token; /api/validate-romaji is CSRF-exempt. Deployed via ship.sh, cron installed, all 4 checks verified PASS on VPS.
• [x] D4: Process for new routes — DONE (str-mamori). Codified in alert_standard_SPEC.md §7 (new alert requirement) and PLN-013 F3 (monitoring coverage checklist). Concept-check added to str-mamori session state.

Phase E: Alert Text Audit — DONE (deployed)

Goal: Every Postmark alert answers three questions: (1) What happened? (2) How bad is it? (3) What do I do right now? Tim was sitting right here during the incident and couldn't act because the alert said "[CUTOVER-WATCH] http_500: ALERT" with no context.

Owner: str-mamori

E1 Key Findings (str-mamori)

imp-redteam audited all 7 VPS scripts + fail2ban + CrowdSec. Results: 14 distinct alert types across 3 alerting scripts (health_check, uptime_monitor, cutover_watch). 4 scripts have no email alerting (traffic_sentinel, scraping_detector, takase_backup, archive_logs). fail2ban and CrowdSec have no email notification configured.

• 3/3 GOOD (2): cutover_watch http_500, cutover_watch synthetic_page — both written during Phase A
• 2/3 PARTIAL (8): all health_check alerts, uptime_monitor recovery, cutover_watch traffic/ip/search/redirect — have metrics but no investigation commands
• 1/3 POOR (4): uptime_monitor down, cutover_watch crowdsec_velocity/recidive/etl_processes — raw counts only

Deliverables

• [x] E1: Audit existing alerts — DONE (imp-redteam). Full inventory of all 14 alert types with trigger conditions, exact subject/body text, and actionability scores. Expanded scope beyond str-michi's 4-script list to cover all 7 VPS scripts + fail2ban + CrowdSec. Key gap: only 2/14 alerts are actionable, both written during the incident.
• [x] E2: Alert template standard — DONE (str-mamori). alert_standard_SPEC.md — subject format ([SYSTEM] SEVERITY: symptom on hostname), body format (WHAT / SEVERITY / DETAILS / WHAT TO CHECK / ESCALATION), check-specific investigation commands table, transition-based alerting requirement, compliance checklist.
• [x] E3: Implement fixes — DONE (imp-redteam, deployed). Three scripts upgraded: health_check.sh v1.02→v1.03 (9 checks with specific investigation commands), uptime_monitor.sh v1.02→v1.03 (DOWN alert with curl/ssh/dig commands), cutover_watch.sh v1.07→v1.08 (7 alerts upgraded, 2 already-GOOD alerts preserved). No functional logic changed. Versions verified on VPS.
• [x] E4: Standard for new alerts — DONE (str-mamori). Codified in alert_standard_SPEC.md §7 — any new monitoring script or alert type must follow the standard before deployment.

Phase F: Standards for New Pipelines — DONE

Goal: Codify the lessons so new cross-domain data pipelines and deploy paths are built right the first time.

Owner: str-michi

Deliverables

• [x] F1: Cross-domain data interface checklist — DONE (str-michi). Added to strategist_methodology_REFERENCE.md § Cross-Domain Coordination. Four rules: write interface spec, producer validates before write, consumer validates at load with safe defaults, health gates check types not just presence.
• [x] F2: Deploy verification checklist — DONE (str-michi). Added to strategist_methodology_REFERENCE.md § Cross-Domain Coordination. Four rules: smoke test product not service, test every pipeline, match test to deploy scope, write status marker.
• [x] F3: Monitoring coverage checklist — DONE (str-michi). Added to strategist_methodology_REFERENCE.md § Cross-Domain Coordination. Four rules: add synthetic canary, alert text per alert_standard_SPEC.md, register in cron_registry_SPEC.md, separate monitoring concerns. References str-mamori's D4 (§7 new alert requirement).

Phase G: Plan Verification — DONE (closed 2026-05-17)

Goal: Verify that what was implemented matches what was designed. "All phases complete" is not "plan succeeded." Every deliverable must be tested against its stated intent, not just confirmed as deployed.

Owner: str-michi coordinates verification. Domain owners fix gaps.

Why this phase was added: After declaring PLN-013 "6/6 complete," HITM-directed verification found: (1) ship.sh Aliya canary checks HTTP 200 only — body is discarded, meaning sections not tested, (2) product_canary.sh doesn't check Aliya at all, (3) Amy's cache data regressed (llm_concepts wiped by lightweight rebuild), (4) 387 of 5,171 LLM-populated records appear overwritten. The two-canary design was correct. The implementation was two status-code checks. "Available ≠ working" — applied to the plan itself.

Verification checklist

Phase B (Schema Contract): - [x] G1: VERIFIED (str-michi, investigate-etl). Freshness gate B4 type checks work: isinstance(person, dict) + name key check. Full population: 77,801 blurb records, 0 type failures. Zero plain-string selected_people remain. 95.1% composite freshness (5,171 variants failures + 128 llm_concepts never-computed — separate issues). - [x] G2: CLOSED (str-ishizue, 2026-05-17). Amy regression resolved; the residual work was carried forward as two separate data questions rather than held open here, on the grounds that the builder-overwrite behaviour and the missing-record count are pipeline issues, not schema-contract issues. - [x] G3: VERIFIED locally (str-michi, investigate-takase). _validate_cache_fields() at line 93 of name_info_service.py v1.02. Validates 6 fields. Called at line 88 on every get_name_info(). 14 tests pass. VPS deployment confirmed by str-takase (quick_push) + ship.sh.

Phase C (Deploy Verification): - [x] G4: FIXED (str-takase, ship.sh v1.05). The smoke test now captures the response body and asserts a section-specific marker is present in it. The meaning-pipeline canary was also swapped to a different name whose data actually exercises that path. - [x] G5: FIXED (str-takase, ship.sh v1.05). Same change covers the Timothy check: it now requires the kana section to be present in the body, not merely an HTTP 200. - [x] G6: CLOSED (str-takase, quick_push.sh v1.03). The data-only path is explicit in the script: content-only pushes skip the Gunicorn restart and the design-page smoke test that follows it.

Phase D (Monitoring Canaries): Note: str-mamori initially deferred G7 on the grounds that "no name has qualifying llm_concepts." A read of the actual cache found 4,784 records with populated llm_concepts, including the canary name then in use. The deferral rested on a claim about the data that the data did not support, which is the same shape as the incident itself. - [x] G7: DONE and deployed (str-mamori, 2026-05-15). A dedicated canary_meaning_pipeline check was added to product_canary.sh rather than overloading the existing design-page check, so a meaning-pipeline failure is distinguishable from a general page failure. - [x] G8: VERIFIED (str-mamori). Cron running, 5-min intervals confirmed. - [x] G9: VERIFIED (str-mamori). TEST=1 alert delivered, format correct.

Phase E (Alert Audit): - [x] G10: VERIFIED (str-mamori). health_check and cutover_watch test alerts confirmed. uptime_monitor has no test mode — noted as low priority, not blocking.

Phase Summary

What happened in the end

The plan closed 7/7 on 2026-05-17, about eight weeks after the incident, and was then deleted as a standing document. That deletion is the point rather than a footnote: a plan whose defenses have moved into production code is a description of the code, and keeping both means maintaining two things that can disagree. What it specified now lives in ship.sh, quick_push.sh, product_canary.sh, the cache record validator, and the alert standard.

Three things are worth carrying away from it.

Phase G is the whole lesson, and it exists because we were wrong. We declared the plan 6/6 complete. A verification pass found that the two-canary design had been implemented as two HTTP status checks. The response body was fetched and discarded, so a page could return 200 while rendering none of the content the canary existed to prove. "Available ≠ working" was the lesson of the incident, and we had just made the same mistake one level up, on the plan meant to prevent it. The fix was real content assertions: the deploy smoke test now requires a named section to actually appear in the returned HTML.

A deferral is a claim, and claims get checked. One monitoring item was deferred because "no name has qualifying data." Reading the data found nearly five thousand records that qualified, including the canary already in use. The deferral was reasonable-sounding and wrong, and nothing but going and looking would have caught it.

The defenses outgrew the plan. The deploy smoke test today also asserts that the design engine is generating fresh output rather than serving cached thumbnails, a check nobody wrote down in any of the seven phases. That is the healthier outcome: the standard survived, and the specific list did not need to.

Created from an 88-minute HTTP 500 incident (2026-03-21). Phase A deployed the same day. Phases B–F built the structural defenses. Phase G was added after verification found implementation gaps, and closed on 2026-05-17.