Burton Lancaster PRO

Both, and both are in now. Commit 0480553c on space-bacon/SRT, card updated on RiverRider/srt-hivemind.

You were right that 879-880 read as shipped when it was a recommendation. As of d56e1a8f the fix was in consensus_select.py and the check was not anywhere.

Per arm: consensus_select.py and chat_consensus.py now raise on consensus, consensus_strict and chat_consensus above the arm's own oracle, so a bad run cannot write a file. Repo-wide: scripts/check_oracle_bound.py is one pass over every results file under artifacts/nla with per-arm oracle columns. It is column-scoped as you specified: the columns scored over all problems are held to the bound; consensus_on_covered and on_resolved_only are declared subset rates and are reported beside the oracle, not judged. Files with a SUPERSEDED key are printed as such and cannot fail the run.

Its output on today's files is your table:

ok   verifier/consensus.json            72 bounded cells   12 subset-rate cells above oracle
ok   verifier_1024/consensus.json       72 bounded cells    7 subset-rate cells above oracle
ok   verifier_1024/consensus_mbpp.json  20 bounded cells    1 subset-rate cell above oracle
ok   verifier_1024/chat_consensus.json  36 bounded cells    3 subset-rate cells above oracle
no all-rows selector exceeds its oracle in any current file

The 12, 7, 1 and 3 are the counts you found, on the columns you named. Zero violations on any all-rows column, which is what the corrected files should show and what the unguarded invariant would have muted.

The README mismatch is fixed in the card and in the paper: 12 of 36 is the 192-token file, 7 of 36 the 1024-token file, 1 of 10 MBPP. The 192-token pools stay in the repo as superseded evidence; nothing in the paper reads from them any more, and the check treats them accordingly.

Your inverse-Simpson result stands as the cleanest statement in this thread of what the template is not doing, and the corrected selector numbers point the same way. Thank you for pulling the commit and grepping rather than taking the sentence at its word.

It falls back to a pick. srt_select.select() returns replies[0] when nothing runs, so the deployable number is the fallback bound. I now report the strict bound beside it.

You are right, and it reproduces to the digit. consensus_select.py averaged the covered picks and subtracted an all-rows floor from them. 12 of 36 arms beat their own oracle, coverage rank-correlates +0.91 with floor (Pearson +0.79), and your two bounds on the 192-token pools come back 0.3760 and 0.3902. Everything else in the repo (exec_guided_select, verifier_select, chat_consensus, pooled_select, the demo bank) was already scoring every problem. This was one read.

Corrected, every problem scored, unresolved pools falling back to reply 0 as the shipped selector does:

The 1024-token pools are the ones Section 5.4 now carries. The 192-token matrix you read was regenerated the day before your comment after a truncation problem surfaced (43% to 80% of instruct-arm candidates were cut off mid-function), which is why your numbers and the current table differ before this correction. On the 1024 pools the bias is smaller because coverage is higher, mean 148.5 of 164 against 133.6, and no arm exceeds its oracle after the fix.

Agreement stays third of five reads on both benchmarks, behind example-filtering by 6 points on HumanEval and 4 on MBPP. Your formatted-versus-unformatted point holds: on the 1024 pools the captured split goes from 0.887 / 0.613 to 0.489 / 0.405. The verifier's deficit to agreement shrinks from 0.1199 to 0.0752, and the agreement decay slope on MBPP moves from -0.0815 to -0.0830 per decade.

Paper 5.4, the dataset card, srt_select, fig 3 and the census are corrected and pushed to RiverRider/srt-hivemind. The old files are kept as *_covered_only_superseded.json with a SUPERSEDED key, same convention as union_ceiling.json. You are credited by name in the limitations, where this is now the fourth accident that flattered us, and in the acknowledgments. The check that would have caught it costs one line, no arm may exceed its own oracle, and it is in the script now. The article body is being replaced with the corrected paper.

Your inverse-Simpson result is the cleanest statement in this thread of what the template is not doing. 7.9 of 8 effective distinct candidates in both families and zero byte-identical pools means the chat format is not deduplicating anything. It is making the model right more often, which is where the corrected numbers point as well.

You read the files correctly, and I reproduced all of it before answering. Every share value matches to four decimals, the mean is +0.2635, the tuning step is +0.0244, the ratio comes out 10.81. The consensus.json figures check out and so does your retraction: the 30-arm and 36-arm sets are used consistently and nothing is quoted across them.

The word "same" is a real error and is now corrected. Chasing your count also explained the 30 versus 31: code_select.py globbed every json in the generation directory, so task_ids.json was read as an arm. It has one entry per problem, so it passed the length check, and slicing its task-id strings produced 11 single-character "candidates", which is the k = 11 and the all-zero row. The glob now requires an arm tag. The published artifact is left alone and Section 12 documents the entry, because silently rewriting a result file mid-thread is worse than labelling it.

On the substance, your objection to raw headroom is right and your normalization is where I part company.

share = floor / oracle is very close to a function of competence alone. Regressed on floor_pass1 across the 30 arms it gives r = 0.9840. Holding competence fixed, the format-specific residual is +0.0223 against your headline +0.2635, so competence accounts for 92% of the movement. Per rung, chat minus raw: 0.5B +0.0968, 1.5B +0.0434, 3B +0.0085, 7B −0.0118, 14B −0.0254. Monotone decreasing, sign change at 7B, largest where floor_pass1 is 0.026 and the ratio is least stable.

You suggested matched-competence arms, so I ran it rather than leaving it as a suggestion. Pairs differing in framing with |Δfloor| < 0.03: 42 pairs, mean share difference for the more-formatted arm +0.0010, median +0.0018, 23 of 42 positive. At |Δfloor| < 0.02 it is −0.0003 over 30 pairs, at 0.05 it is +0.0102 over 64.

One caveat against that null. The framings barely overlap in competence, weak framings spanning floor 0.021 to 0.109 and strong ones 0.156 to 0.540, so most matched pairs are cross-rung. Same-rung pairs number 9 to 20 depending on threshold and give +0.0085 to +0.0151. Thin, and not zero.

So the honest range for a format-specific effect on share is 0.00 to 0.02, against your +0.2635. Within-pool convergence is a different claim from 5.4 and this evidence does not establish it. 5.4 says text similarity across models does not imply agreement on correctness, and nothing here touches that. What these numbers show is that the chat template makes the model better, and competence mechanically raises the fraction of pool coverage a single sample captures.

The mechanism underneath your reading is real. The selector eats residual disagreement, so anything that raises competence shrinks the room. We measured that on a separate banked set this week, on a different axis to your ladder. The selector's edge over a pool's own pass rate is +0.44 when two candidates agree and +0.04 when eight do. The format closes the room the same way scale does, by making the model right more often rather than by making it agree with itself.

The mechanisms you name do not fit the measurement. Transport is measured on base checkpoints: Qwen2.5-0.5B and 7B, Qwen3-0.6B-Base, gemma-2-2b, Llama-3.2-1B and 3B, OLMo-2-1B, SmolLM2-360M, pythia-410m. Nine of the eleven models are base, and no chat template is applied anywhere in that experiment. RLHF and DPO never touched those weights, pythia-410m was trained on the Pile and released before the endpoints you name existed, and serving-backend geometry does not reach into pretraining. None of that produces a 0.999 linear map from Qwen2.5-0.5B to OLMo-2-1B.

Your general point is a different matter and it lands. Our lineage control separates same-company from different-company, and a cause operating across the whole industry predicts exactly the small premium we measured, so that number does not argue against you. Shared web-scale pretraining distribution is a live explanation for the transport result and our design does not separate it from a general property of language modelling. That has been inserted into our limitations.

The two results the paper rests on run against the mystical reading rather than for it. Base models do not reach the reported level. Instruction tuning alone adds 0.0786, taking the matched pairs from 0.3649 to 0.4435. Prompting those same tuned weights through their own chat template reaches 0.7272, which is 0.3623 above base, or 0.2837 taken as the marginal step over the untemplated instruct arm. On either contrast the format term is several times the weight term. Our conclusion is that most of the reported hivemind is a prompt convention.

Thank you, I will run a sweep on the repo against your papers soon @AbstractPhil … Any specific recommendations?

All four, run. You reconciled our summary from the rounded rows and got 0.759029;
at full precision it is 0.759035. Nothing to argue with, so here is the rest.

The headline has an interval now, and it is one number rather than fourteen
averaged: resample patients, recompute the macro over all fourteen findings
inside each draw.

macro 95% CI 0.7500 .. 0.7673
macro SE 0.0045
gap / SE 3.09

You estimated 0.0041 and 3.39 and said to treat that as the optimistic end
because findings co-occur within a patient. Measured 0.0045. Right on direction,
close on size.

The floor, twenty refits, each permuting training labels and refitting:

floor macro 0.5006 +/- 0.0052 range 0.4910 .. 0.5093
per finding refit SD 0.0177
gap / floor SD 2.65

Two things in that. The 0.5002 we published sits almost exactly on the mean, so
it was not the lucky draw, and I would not have guessed that before running it.
But the spread is real and it is yours: you predicted a per-finding refit SD near
0.019 and it is 0.0177. Your own threshold was under 0.003 safe as written, near
0.006 wants a hedge. At 0.0052 this lands nearer the hedge, so the number travels
with the headline from now on.

Your question: we kept only the AUROC. Both now. The count gets a measured null
out of it, and it is blunter than yours: a signal-free probe beats Wang on 0 of
14 findings in all twenty refits, max 0. Worth being exact about what that does
and does not do. It rules out a signal-free probe producing 12 of 14. It does not
price whether 12 of 14 is a surprising split for a probe of our overall quality,
which is what your 106/16384 estimates. Different nulls, and yours is the right
one for that question.

Where we undersold it, both confirmed exactly as you computed:

leave-one-out +0.0110 dropping Pleural_Thickening .. +0.0218 dropping Hernia
never flips, never approaches zero
weighted by n 0.7491 vs 0.7226 = +0.0265, against +0.0139 equal-weighted

The two findings we lose on are the two whose removal helps most, and macro gives
Hernia's 86 positives the same vote as Infiltration's 6,112. Equal weighting is
the conservative choice and it costs about half the gap. That goes on the card.

All of it refits from the published states on a laptop. scripts/
cxr_probe_floor_seeds.py, artifacts/nla/cxr14_floor_seeds.json.

One draw wide, and the draw you doubted is the one that held.

Twenty independent random derangements, same queries, same gallery, each verified to leave no caption on its own photograph.

arm                      floor mean     sd    range over 20    single draw   verdict
candle Q4 as-is             65,088   363.6   64,408..65,733         64,481   inside, -1.67 SD
candle Q4 + 4 KB anchor     61,718   593.4   60,646..62,719         59,071   outside, -4.46 SD

64,481 survives. 59,071 was the bad draw, below all twenty.

The margin

44,578 against its own floor is 20,510, on a floor with an SD of 363.6. The single draw said 19,903, so the correction moves the number in the direction that costs me nothing, which is the direction I checked hardest.

Your 27% dissolves, through something neither of us proposed

Your premise was that both floors measure the same thing, since the gold image is independent of the query by construction. Twenty derangements say they do not. 65,088 against 61,718 is 3,370 apart with a pooled SE of 155.6, so 21.7 SD. The gap survived the error bar that was supposed to absorb it.

The floor is not a draw from the gallery. replay_123k.json holds 5,001 captions over 1,000 distinct images sitting at gallery rows 122,287 to 123,286, the last 1,000 of the index. Every mismatched pairing permutes caps[].gold inside that fixed 1,000-image pool. A floor therefore measures how one arm ranks the val2017 images, which is a property of the arm. Two arms have two floors, and they were never interchangeable.

The spreads were the tell before I had the explanation. Your uniform-draw SE of 872 is correct for a draw over 123,287 rows. The observed spreads are 363.6 and 593.4, far too tight, because across derangements the same 1,000 images reshuffle among the same 5,001 queries.

So there is no floor to pick. Each arm is quoted against its own, and the unanchored margin is 20,510.

Which takes analytic chance out of the comparison

61,644 is not the reference for either floor. The as-is floor sits 3,444 above it, 9.47 SD. The anchored floor lands 74 away, 0.12 SD. I read that near-agreement as a coincidence of this pool, since the same structure that puts one floor 9 SD off cannot be certifying the other.

The card line holds without leaning on analytic chance to do it. Dead by recall, above its own empirical floor by median, 20,510 ranks with an error bar.

Scope note on what this does not settle. Twenty derangements pin the floor for one query distribution against one 1,000-image pool, and the pool-structure argument says that floor is a property of the arm being scored. It does not generalise to a different gallery, and it says nothing about whether the recall the anchor leaves behind is recoverable at all. That question is untouched.

Artifacts

browser_rung_derangements_123k.json is in srt-browser-head-118k, with all twenty medians per arm, the derangement construction, the pool-structure note, and a derived block holding the margins and SD distances above so the arithmetic is checkable without recomputing it. median_sd is the population SD; sample SD reads 373.0 and 608.9. The supersedes field names browser_rung_mismatched_123k.json and states that one of its two floors was a bad draw. That file stays as it is.

Your question changed a published margin, retired an analytic baseline I should not have been quoting, and cost one artifact its floors. Asking whether a control is a typical draw or a single one turned out to be worth more than the answer it produced.

Restores. It does not supply. And your method found a second one on the way.

Your loose end, closed

All four are up in srt-browser-head-118k, plus the new one:

• cross_runtime_browser_rung_123k.json
• browser_fp16_reference_123k.json
• cross_runtime_browser_rung_repro.json
• scripts/browser_anchor_reference.py
• browser_rung_mismatched_123k.json

They were committed to space-bacon/SRT and never pushed to the Hub. You were checking the namespace the card points at, so "absent" was the correct reading and I have no defence for the gap.

cross_runtime_browser_rung.json stays at 824 bytes, still "gallery": 1000, still 0.0154 / 0.1952. It is a correct record of a real run and I would rather not rewrite history in place. The trail is the supersedes field in cross_runtime_browser_rung_123k.json, which names it and states what was wrong with how it was presented.

Your control, run

5,001 captions, shipped head, shipped 123,287 index, mean pooled. Each caption scored against a different photograph: same queries, same gallery, only the pairing broken.

Both floors are clean nulls at R@1, R@5 and R@10.

The unanchored read-out sits 19,903 ranks above its own floor at exactly zero recall. The signal is present before the anchor touches it, so the anchor amplifies what is already there rather than manufacturing it.

Your reading was right and understated. You had 44,578 against analytic chance at 61,644. The empirical floor for this query distribution is 64,481, which is worse than analytic chance, so the margin is larger than the comparison available to you could show.

Your suggested method would have burned you

You proposed rolling the browser gallery by one position, the way the verbalizer rolls its records. I did that first and it is wrong here.

replay_123k.json is grouped by image and COCO gives five captions per image, so caps[i+1] owns the same photograph 80.0% of the time. The floor is mostly the true pairing. That run returned:

and I read it as proof the residual was hubness. I had it written up before the tell registered: a genuine null over 123,287 images sits near 1/123287 = 8e-6, so any null arm reporting R@1 0.0272 is contaminated by construction. The verbalizer's roll is safe because it rolls vectors, one per image. The browser replay is per caption. Same word, different object.

browser_rung now jumps half the caption list, walks until the gold image genuinely differs, and prints the collision count so the control is verified rather than assumed: mismatched pairing: 0 of 5001 still share an image.

What your method found in the program paper

I went looking for the same failure mode elsewhere before you did, and found one. The invariance section said the bf16-trained head applied unchanged to 4-bit states "loses 0.011 R@1; a 42KB mean recalibration recovers half of that." From q4_drift.json, i2t:

R@1 recovers 0.003 of 0.011, which is 27%. Half is R@10, exactly 50%. R@5 recovers 85%. The loss was right and the recovery fraction was reading a different column. Also restored: the pool is 1,000 images, not deployment scale, and the artifact's own caveat is that bnb NF4 on GPU proxies llama.cpp Q4 on CPU. Fixed in seven places, pushed as 4d624b71.

Your R@1-does-not-predict-median result, reproduced

Your retraction from six-for-six to five-for-five is the more useful half of that comment, and we hit the same wall independently the same day in a different system. A reader trained on gemma-4-31B image states, handed the caption states of the same scenes, 208 held-out images, same gallery:

R@1 orders those backwards. Ten times the median separation, and recall puts the worse arm on top. Your 483 admissible fits spanning 35,972 to 8.5e302 is the general form of what that table is a single instance of.

That probe also cost us a run. Its first version had no gold-caption arm and silently used the wrong head. Every arm landed in the null band, the sentences stayed fluent throughout, and it read as a strong positive finding rather than as broken wiring. There is now a harness control that runs first and aborts if human captions cannot retrieve their own images.

Thank you for pushing on the median. Three corrections today came from it, and two were in files you had not asked about.

You are right on every point, and the number you asked for is worse than the one
it replaces. Both halves took a re-encode, so this is late rather than quick.

Your chain, checked link by link

Every step reproduces. cross_runtime_browser_rung.json does declare gallery
1,000 and captions 5,001. Its 0.2300 is the f32 row of
gallery_precision_cost.json. That file's int8 row is eval_only_1000 in
gallery_scale_cost.json to the digit, and that file's full_123287 is
0.0148 / median 596, which is old_head_for_reference, n_train: 4000.

The two LFS identities hold: the Space's anchor.bin and
anchor_candle_q4_text.bin are ecc143e0, and the Space's head.safetensors
and head_v3.safetensors are 09c7322a. The two reports are one blob,
06e56278, 564 bytes.

One thing you could not see from outside, and it strengthens your reading: the
head file that run actually loaded, artifacts/local/browser/head.safetensors,
hashes 0663ff9b and matches neither the shipped head nor the 118k head. It
is a third set of weights, exactly as your chain implies.

The val-pool label is wrong too, and your arithmetic beat our own metadata to
it. browser_head_118k.pt carries n_eval_images: 5000 and split: train on COCO train2017, evaluate on all of val2017.

Does head_v3 have a number of its own

Yes, and it is 0.1108. head_v3.safetensors is the fp16 export of
browser_head_v2_best40.pt: txt.weight, txt.bias and mu_txt all match
under fp16 round-trip, while the 118k head misses on all three by up to 0.56.
And that checkpoint's own meta carries val_only: {i2t 0.5348, t2i 0.3985},
the rows in the report.

So the shared blob is not a stale copy. It is one measurement of one set of
weights under two filenames, and arm: best40_final is accurate rather than
left over. "v3" is a deployment name for the best-40 checkpoint. That naming
cost you an afternoon and it is being fixed in the Files table.

The published table is correct and mislabelled

Re-run in its own configuration, 4k head, last-token pooling, pool 1,000:

Bit-identical, not just to four places: 0.015396920615876825 and
0.1951609678064387 in both files. That table was measured correctly and
computed correctly. What was wrong is that it was presented as the
deployment's, while describing an earlier head, a different pooling, and a pool
123x smaller.

What the shipped head actually does

Shipped head, shipped 123,287-image gallery, mean-pooled as the browser pools,
same 5,001 captions:

Recovery is 32%, not 85%. Without the anchor the read-out is at zero, so it
is still the difference between a working system and a dead one, but it is not
close to sufficient, and the card no longer says it is.

Four things make that number checkable rather than asserted. The fp16 arm
independently reproduces the headline (0.1092 / median 36 against 0.1108 /
median 33). The head is byte-identical to the Space's. The index is the shipped
SRTIDX02. And the anchor the run computed is byte-identical to the shipped
anchor.bin, cosine 1.0, max abs diff 0.0, so the tab is running the anchor
that was measured.

The gap between 85% and 32% is dominated by pool size rather than by anything
being broken, and our own published data shows it with no quantization
involved: the v2 head scores 0.4959 against 1,000 images and 0.0628 against
123,287, fp16 on both sides. The residual the anchor cannot remove is cheap
against 999 distractors and expensive against 123,286.

Two harness defects, because they are the transferable part

The first measurement I ran returned 0.0136 recalibrated and I nearly posted it.
browser_rung's Tap::new defaults to Pooling::Last, and the shipped head is
mean-pooled, so it was scoring the wrong input entirely. A mean vector cannot
repair a pooling error, which is what gave it away. The Pooling enum's own doc
comment warns about this three lines above the constructor. Pooling is now an
explicit flag and is printed and recorded.

The second is smaller: the prebuilt harness on this machine only understood
SRTIDX01 while the shipped gallery is SRTIDX02. I initially read that as
evidence the harness had never been pointed at the deployed index. That was an
inference from a stale local build rather than a measurement, and I had already
overwritten the timestamp that would have tested it, so I am dropping the claim.
The artifact says "gallery": 1000 on its own and needs no help from me.

Card

Anchor table replaced with the deployment measurement and the 85% relabelled as
the 4k head at pool 1,000. The int8 rows are scoped to the question they answer,
which is that quantizing the gallery costs nothing on a fixed head and pool. Val
pool corrected to 5,000. The pool-size sentence is attributed to the v2 head.

Artifacts: cross_runtime_browser_rung_123k.json (deployment, with pooling,
head sha256 and index recorded), browser_fp16_reference_123k.json,
cross_runtime_browser_rung_repro.json (the bit-identical reproduction), and
scripts/browser_anchor_reference.py. --mean and --dump-anchor are in
browser_rung so the anchor identity check is reproducible rather than
something you have to take from me.

What I do not know yet

The rank distribution of the anchored Q4 arm. R@1 and R@10 are above but not
median rank, which is the number the card leads with for fp16 (33 of 123,287)
and the one that says whether the deployed demo is merely worse or actually
unusable for its own purpose. That is running next.

And the standing question underneath it: this drift is structured rather than a
translation, which is presumably why 4 KB of mean only buys back a third of it
at scale. Fitting the text head on states encoded by the deployment runtime,
instead of fitting on fp16 and patching, is the obvious test of whether the 68%
is recoverable at all. That is a training run on the 0.6B, so it is cheap.

Thank you for reading the files rather than the post. Every number you
questioned turned out to be real, and every label on them turned out to be
wrong.

Closing the item I left open, and the re-bank behind it turned up two more
reversals.

The anomaly was mine, and it is a coincidence

The two cells disagree on a tenth of the items and land on the same total twice.
Different decisions, equal sums, nothing structural.

The reason my verification kept dying was not the box. Indexing an NpzFile
re-reads and decompresses the whole array on every access, so building the
lookup inside a comprehension did 11,844 full re-reads of a 264 MB array. It
presents as a memory limit and is not one. sugarcrepe_controls.py carries a
comment warning about exactly that, which is where I took the pattern from and
then failed to follow in the throwaway check.

Every arm is now re-banked on normalised text

--txt and --normalise live on the shipped control script rather than beside
it. Defaults verified untouched: the qwen/qwen cell reproduces its banked
clean-five macro to every digit on all seven arms.

Your mean-image inversion was the period

Real minus the constant-image arm, clean five:

Those two negatives are the 17.8 points you priced in round 14. Normalised, real
images beat the ablation in all four cells by 10 to 12.5 points, so "image
content degrades a text prior everywhere subtler" does not merely lack an image
in it, as I said last round. It was formatting.

Your typicality reading survives untouched. Foreign-mean still tracks mean-image
within half a point in every cell (0.629/0.632, 0.632/0.630, 0.647/0.646,
0.652/0.648) and random directions still sit at chance. The ablation still is
not reading the images it replaces. It just no longer wins.

The text-tower effect reverses sign

Margin over the derangement null:

"Everything any head in this program knows beyond a bigram counter lives in the
gemma-text column" from round 12, and "the margin sorts by text tower, ratio
1.86x" from last round, both rest on that -0.058. Normalised, qwen text is
ahead. The qwen text tower was the more contaminated of the two, and I read its
contamination as a property of the tower.

What survives is weaker: text matters more than image, 2.1 points against 0.1.
The ordering was an artifact and the magnitude falls by two thirds.

The drift control, on every arm

The reversals compare normalised states against un-normalised ones, so the floor
has to cover the arms the conclusions are read off and not just the real arm.
Fresh raw minus shipped cache, clean five:

The derangement null is the steadiest of the three, which is what you want of
the arm carrying the margins. Largest magnitude on any arm or margin in any cell
is 0.0042, against a normalisation effect on the margin of 0.034 to 0.117.
Between 19x and 276x. On swap_obj the drift is under two items of 245 where
normalisation moves about 27.

Where the scorecard stands

replace_obj is untouched by any of this and carries +0.32 to +0.35 over the
derangement null in every cell. swap_att joins swap_obj above its scrambled
control in all four, +0.075 to +0.096. The inventory results never went through
a caption comparison, so nothing here reaches them.

One thing I want your read on

Normalisation is in the harness, as you argued. But it makes our numbers
non-comparable with everyone else quoting SugarCrepe, and the comparable column
is the contaminated one. I am inclined to report normalised as primary and keep
un-normalised as a parallel column labelled as such, rather than silently
switching. If that is the wrong call, say so before it reaches the papers.

controls_norm_*.json, controls_norm_summary.json for the comparison and the
per-arm floors, and the --normalise path in scripts/sugarcrepe_controls.py.

The period does not prop the result up. It was holding it down, and removing it
reverses a conclusion I published last round.

Your four claims, checked first

The punctuation table reproduces on upstream RAIVNLab digit for digit: 1384 of
1384 differing pairs have the foil as the dotted one, no counter-examples in
7,511. Upstream swap_obj is 245, and your swap_att duplicate is there ("A
computer desk with two monitors and a laptop.").

The tied-control argument is correct and I concede it as stated. On swap_obj I
get 64.5% bag-identical and 90.2% same word count, on swap_att 59.6% and 85.4%,
within tokenizer distance of your 66.9/90.6 and 61.3/85.4. Where the foil is a
permutation, caption length and mean unigram log-frequency are equal by
construction, so the regression had no variance to fit. Reading 0.563 there as
"not length either" was reading a tie as a result.

Your pricing bounds independently. The period differs on 31 of 245 swap_obj
pairs, so a perfect reader of the tell that coin-flips elsewhere scores 0.5633,
which is 6.3 points over chance against the ablation's 21.4. At most 30%, which
arrives at your 33% from the other side.

Answering the question you asked

It needed a re-encode, not a harness change: text states are keyed by the raw
caption string. So both arms were encoded on one box in one session,
transformers 5.15.1, raw and normalised (trailing period stripped, first
character lowercased), which puts the re-encode floor inside the comparison
rather than beside it.

Clean-five, raw re-encode against the shipped cache: -0.0004, +0.0020, +0.0030,
-0.0011 across the four cells. The floor is not carrying anything.

Then the answer. Margin over the derangement null, clean five:

Real accuracy rises 5 to 9 points in all four cells. Against the null that
varies per cell, the measured alignment roughly triples under qwen text and
gains a third under gemma text. The formatting was not a crutch. It was a
smudge on every generated foil, and it was suppressing the signal.

What that costs me

swap_obj margin over its own scrambled control:

Last round I wrote that image identity is worth nothing on either swap split and
that the entire margin is object identity. That was an artifact of the
formatting. Both swap splits move from at or below their scrambled control to
clearly positive under qwen text. The inventory-versus-arrangement division of
labour is not retracted, but the sentence that swaps are worth nothing is, and
the papers get restated again.

Your own number: the mean-image ablation on swap_obj falls from 0.7102 and
0.7347 to 0.6694 under qwen text, about half the distance to the 0.63 you
predicted, and does not fall at all under gemma text (0.6408 to 0.6571).

The thing you did not find, which is worse

Our blind bigram prior reads the same tell. bigram_prior tokenises with
.lower().split() and never strips punctuation, so water. and water are
different tokens and the trailing period rides inside the final bigram.

Our shipped swap_obj prior is exactly 140/245 = 0.5714, which is the number your
punctuation-only rule gets. Not a coincidence, the same trick.

That also retracts what I told you in round 14. I said our rebuild at 0.5449 and
yours at 0.565 differed by tokenizer choice. They differ by punctuation
handling: stripping returns ours to 0.5429. Three rounds after conceding that
our nulls were blind to the thing they were nulling, the null we shipped to fix
it was reading the artifact you had not yet named.

Your binding question

You are right that n is not the constraint and that regeneration makes it worse.
The signature is a property of the pipeline, so 584 per cell reproduces it at
584. Normalisation and sample size are orthogonal problems and enlarging the set
fixes neither. Normalisation is now in the harness, not the data, for the reason
you gave.

One anomaly I could not close

In the re-encoded arms the mean-image ablation returns identical values across
cells that use different image towers: 0.6408 raw and 0.6571 stripped in both
gemma-text cells, and 0.6694 stripped in both qwen-text cells. It is not a cell
collapse, because the cached arm separates those same cells inside the same run
(0.6531 against 0.6449), and the two fixed directions are near orthogonal at cos
0.031. My verification kept dying on the box and I would rather post it as open
than explain it. If it reproduces on your rebuild I would like to know, because
three coincidences at integer counts is more than I want to accept.

Code and artifacts: scripts/sugarcrepe_encode_norm.py,
scripts/sugarcrepe_period_test.py,
artifacts/nla/q4/period_test_all_cells.json and the four per-cell files.
Both encoder arms ship, so the floor is reproducible rather than asserted.

Sixteen rounds. You have now found an artifact that was hiding a result rather
than manufacturing one, which is the first time the correction has gone in that
direction, and it landed inside the control we built because of you.

Both of your asks ran, and the second one takes your retraction further than you took it.

Your direct question first. Neither. The 0.001–0.002 shuffled control is run(..., shuffle=True) in gemma4_mlp_align.py, which permutes Ytr and then calls train_heads. It is a freshly initialised head fit on destroyed correspondence, so it prices whether the fitting procedure can manufacture alignment from noise. It is not an eval-time null for a given head, and the per-cell null you are asking for did not exist anywhere in the repo.

Worse, and you have not caught this one yet: rounds 12 and 13 shipped their controls as result files with no code path. blind_bigram_prior, margin_over_blind, prior_comparison and mean_image_control are all JSON with nothing behind them. That is the same class of thing as the dead HEAD_REPO constant you opened with, two rounds running, and it is ours. Every null now lives in one script with named arms and one scorer: scripts/sugarcrepe_controls.py.

Reproduction against the banked numbers, fresh box, transformers 5.15.1:

The matched cell reproduces exactly on all seven splits; the mixed cell reproduces exactly on three. That is the fresh-box re-encode floor, disclosed rather than explained away.

Your derangement null, all four cells

Real images, trained head, pairing deranged, 20 seeds. Clean five macro:

The margin sorts by text tower and not at all by image tower. Swapping the image tower moves it 0.8 and 0.5 points. Swapping the text tower moves it about 6. Ratio 1.86x.

So the tower attribution survives, but now against a null that takes a different value in every cell, which is precisely what the blind prior could not do and what you correctly said it could not do. Per split on the matched qwen cell: replace_obj +20.7, replace_rel +8.6, replace_att +7.4, swap_obj −0.2, swap_att −1.4. The entire margin is object identity, and both swap splits sit at or below their own scrambled control.

Your 17.8 points, where I think you stopped one arm short

You gave two readings of the mean-image ablation, word-order competence or a candidate-set artifact, and proposed position exchange to separate them. There is a third, and it falls out of the scorer. score() is a cosine, so holding the image constant makes zi one fixed direction and the whole ablation reduces to which caption embeds closer to it. That is a typicality test, and typicality is invariant to position exchange, so your test would have scored it as genuine competence.

Three arms, all text-side, no re-encode:

Another split's mean image reproduces the ablation everywhere, within 0.5 points in all four cells. A random direction gives chance. So the ablation is not measuring the images it replaces, and it is not measuring fixed directions in general. It is measuring the one direction image means occupy, along which SugarCrepe's human captions outscore its generated foils. A regression of that fixed-direction score on caption length and mean unigram log-frequency accounts for the add splits almost exactly (0.993, 0.994) but not the swaps (0.563, 0.590 against the ablation's 0.716, 0.714), so it is not length either, and your normalized prior already showed that.

That is your round-13 control failing for exactly the reason you named in round 14, confirmed from the inside. The reading I banked off it, "image content degrades a text prior everywhere subtler", is withdrawn. It never had an image in it.

And the four-cell view explains the inversion completely: the ablation beats real images only under the weaker text tower. Under gemma text it loses (0.624, 0.625 against 0.686, 0.695). Under qwen text it wins (0.693, 0.687 against 0.664, 0.661). It was a property of the text tower, not a fact about image content.

Position exchange cannot decide this

Each candidate is scored independently against the image and the two scores are compared, so the scorer carries no positional term and exchange maps accuracy to its complement identically. Verified to 1e-9 on every split, both arms, all four cells. The test only bites on a scorer that sees both candidates jointly, like a cross-encoder. Ours is not that.

One more thing we found, before you do

Building the control script, I went to load the gemma/gemma v6 head and it does not exist. Not in the repo, not in RiverRider/srt-nla-gemma4-artifacts. The paper's headline cell had no retrievable head behind it.

Rebuilt it on a fresh box from the published procrustes/train_pairs chunks plus a seed-0 regeneration of the v6 negatives. The negative set came back at 473,148 with prep_replace 75,786 and adj_transfer 54,894, matching the published counts to the item, so it is a reproduction rather than a fresh recipe.

Note the linear comparison is against the w=1.0 hard-negative row of the weight sweep, not mlp_align_full's 0.661, which is the no-hard-negative recipe. Comparing to the wrong row would have understated the rebuild by four points. Baseline and shuffled reproduce exactly.

It is published now, with the state caches, so the whole control script runs from published artifacts alone: q4/gemma_v6_head.pt, q4/sugarcrepe_img_states_gemma_L47.npz, q4/sugarcrepe_txt_states_gemma_L47.npz, q4/mlp_align_gemma_v6_rebuild.json, and q4/controls_*.json for the four cells.

What survives, stated against the null that varies per cell

Image identity is worth +6.5 to +13.0 clean-five points depending on the text tower, concentrated on replace_obj at +20.7, and worth nothing on either swap split. That is the inventory-versus-arrangement division of labour from round 11 arriving from a completely different direction, which is the first time two independent instruments in this program have agreed on it.

Both papers are corrected and the PDF is rebuilt. Ledger entry 9 records that three of our four SugarCrepe nulls were blind to the thing they were nulling, and that the sign of our conclusion was being set by null selection rather than by measurement. The acknowledgment credits the mean-image ablation as a control you proposed and then withdrew, whose withdrawal is what forced the derangement null that replaced it. That sequence is worth more than a control that had worked.

Code: scripts/sugarcrepe_controls.py, artifacts/nla/q4/controls_*.json, artifacts/nla/q4/mlp_align_gemma_v6_rebuild.json.

And your standing question is now the binding one

Every claim on swap_obj, mine and yours, the 17.8 and the floor and the margins, is read off 245 items with a 95% interval of about six points. Your arithmetic says resolving an eight-point effect needs roughly 584 per cell. I do not think SugarCrepe can be re-split into that, so it means generating a larger swap set.

Worth doing, or is swap_obj a split we should both stop quoting?

Both asks are done, and your control found something neither of us predicted.

First the concession, which is arithmetic and yours: the prior is blind to both towers, so it takes one value per split in every cell, and subtracting it moves the zero point and nothing else. "The qwen-text cells were barely measuring alignment at all" was a statement about where four numbers sit relative to a fifth, dressed as a statement about towers. The papers now say what the margin view actually licenses: an absolute floor, with tower comparisons living in the raw table unchanged.

Your prior is banked next to mine (artifacts/nla/q4/prior_comparison.json): two rebuilds agree within two points per split, with the residual down to tokenizer choice, and each split now carries its binomial interval. On swap_obj that does what you said it would: against BOTH rebuilds, every cell's margin interval straddles zero (gemma/gemma [-0.03, +0.14] vs mine, [-0.05, +0.12] vs yours; the other cells alike). The papers now state it as a direction all four cells share that no cell confirms. Margins are reported as intervals from here on.

Then I ran your mean-image control on the two cells whose states I have clean locally (both qwen-text; the gemma-text cells need a re-encode and will follow the same protocol). Same head, same texts, every image replaced by the split's mean image vector. It does not just adjudicate, it inverts:

The ablation BEATS the real images. Per split: real images add about +15 points on replace_obj and subtract about 11 on both swap splits, where the trained text projection's caption prior alone scores 0.71+. So the picture your control exposes: image content contributes exactly where object identity is at stake, and everywhere subtler it actively degrades a text prior that was doing better without it. That is a sharper statement of the whole thirteen-round conclusion than anything I had, and it came from the control you proposed while retracting your own convention. Banked at artifacts/nla/q4/mean_image_control.json.

So the standing conventions, after this round: margins over a blind prior as intervals against both rebuilds, and the mean-image ablation as the per-cell control wherever a cell is claimed. Ledger updated accordingly.