The instrument can change the answer.
The historical judge penalised valid USER context and disabled tools. Later recovery repaired incomplete judging material, not the responses.
Core & judge · 6 studiesText · control · judge · source
What changes when the instructions do.
Thirty-four studies of SOUL, initiative, memory—and the instruments used to judge them.
Three findings
Control can outweigh the candidate.
Every initiative v2 arm reached 14/14 verified positive outcomes. The shared ceiling—and a shared pause failure—blocked a SOUL ranking.
Initiative · 5 studiesRetrieval is not answer generation.
Memory screens completed 12 evidence requests and returned 45 hit slots, with zero generation or judge calls. A no-memory control remains unexercised.
Memory & sources · 3 studiesV4 / CURRENT
V4 failed the replacement gate. It was installed later by choice.
Two blind advisers judged 16 fresh pairs each. Historical tarot controls stayed separate.
| Blind adviser | V4 | CURRENT | Ties | Frozen gate |
|---|---|---|---|---|
| GPT-6 Sol | 10/16 | 6/16 | 0 | Fail: CURRENT > 3 |
| Kimi K3 | 8/16 | 4/16 | 4 | Fail: V4 < 10 and CURRENT > 3 |
Gate protocol and subsequent choice
Measured. 32 fresh finals plus 4 historical tarot finals; 36/36 finals from 36 candidate calls and 36 source-complete judgments. Each adviser required V4 ≥10 and CURRENT ≤3 wins on fresh pairs. The threshold was not softened after the result.
Afterwards. The exact tested English V4 was installed through a new, direct user choice with rollback and native-loader verification. This deployment decision did not retroactively pass the benchmark. Follow-up recorded persistence as verified.
Chronology & research streamsFour distinct namespaces
Four ways to ask one question.
Historical v3.7, initiative v4/v5 and English V4 are different namespaces. Their scores and protocols do not form one ranking.
The core and its judge
The v3.3 reduction preserved parity; v3.5 showed that one line can support a different register. An audit then found the judge penalising emoji, valid USER context and disabled tools. The measuring instrument became part of the object of study.
6 core and calibration studiesInitiative and limits of action
Five series moved from an invalid conflation of pausing with inaction to a control ceiling, real compacted history and wrapper ablation. Shared control failures mattered as much as candidate differences.
5 initiative studiesV4/reference: from text to apparatus
Sixteen catalogue studies tested language, runtime, overlays, compute, context and sources. Later reports added an effort screen and final blind gate. V4 failed the replacement gate and was installed afterwards through a separate user choice.
20 catalogue and later-state studiesMemory as access to evidence
Preparation, nine primary retrieval queries and three post-result diagnostics separated source availability, freshness and project identity. Twelve successful API responses are not twelve generated answers: generation and judge calls were zero.
3 memory and source studies
Host audit & late-tail diagnosticsSeparate observations
External observation · not a generated-answer experiment
The host audit exposed a different seam.
A read-only Quoth audit observed automatic context delivery. It was not a repeat of the local retrieval screen, a confirmed repair or an independent audit of the host ledger.
46/214 logged turns were slow or busy: 44 slow plus 2 busy, or 21.50%. Busy is not the same cause as deadline lateness; 46 is not the number of turns with useful recoverable evidence.
Same-note is not necessarily duplicate. At least two hits from one note occurred in 36/154 eligible turns (23.38%). Some tasks genuinely need complementary fragments from one source.
One ledger is not the whole context. Zero accepted-current claims in Quoth’s own ledger does not imply an absence of persistent context, always-loaded material or vault evidence in Sera.
A fixture needs the source text itself. Future no-live-access answer tests need exact oracle excerpts with provenance and temporal basis. Locators and a desired conclusion are insufficient.
A new slice: separate diagnostics, not a rewrite of the 214-turn audit.
Mixed census
217 records, 8 version labels: 49 slow and 2 busy. In the stratum labelled with working health 0.14.0: 1 slow/74 (1.4%). This is neither a causal version comparison nor an estimate of the current configuration from a pooled cohort.
Fresh dry-prefetch
9/9 familiar queries were fast and non-empty; vault median 11.33 ms, max 63.38 ms. No warmup or retries, with shared uncontrolled cache/load. Generation and judge calls: 0; hit quality was not assessed.
Natural tail: unmeasured
Same-natural-job late completion and the post-gate would-inject result remain UNMEASURED. A repeated fast query cannot recover an old event. Production policy is unchanged; a future observation window does not block this exhibition.
The minimal proposal is record-only instrumentation for late completion and would-inject evidence. This observation does not imply a budget increase, diversity change, claims approval or rollout.
Methods, caveats & open questionsWhat the numbers cannot mean
The judge is inside the experiment too.
Blind does not mean infallible. Historical judge v1 understated v3.6 by about 1.5 points in the audited sample, confused persona texture with violation and missed USER context. Disputed probes were rerun, specs repaired and original verdicts retained.
Source fidelity is checked mechanically and semantically. The wrong-treatment source pilot stopped because it used AI synopses; the corrected batch used exact primary fragments. Context-package recovery restored complete history to judges and checked real line anchors without regenerating answers.
Denominators do not mix. Dialogues, API responses, model stages, strategy finals, source reads and judge calls remain different quantities. Historical 0–10 scores do not combine with newer pairwise preferences. Unknown cost is not recorded as zero.
Not tested. A generic no-memory control, autonomous integration of retrieved evidence into a deeper answer, full live-native equivalence and a durable cross-protocol ranking. The retrieval screen is not an answer-depth evaluation.
This page contains curated fields only. Protected source-to-record mappings and SHA-256 values live in a separate nonserved coverage ledger. The standalone HTML loads no fonts, data, scripts or raw research files.
The next question became more precise.
Not “more memory?” but “which source is available?”
First establish corpus availability, revision and project identity. Then separately measure how a frozen source packet integrates into an answer.
Not “XHIGH is better” but “where does the win repeat?”
The positive effort signal failed its preregistered repeatability gate. That invites a sharper question, not a universal improvement claim.
Not “V4 won” but “what did the author choose?”
The scientific gate and later installation remain two events. Strong explanation passages and local voice errors coexist.
Explore all studies34 records · searchable
Evidence atlas
34 studies
No studies match these filters. Reset them or broaden the search.
The study list could not be loaded in this browser.