VIDRAFT_LAB

All three hold. All four are fixed, and the closing question now has an answer written into the code.

That question first. Causal-Safe should mean "no leak, and we could have seen one." Selling "we didn't find one" as safety is the exact error we've been pointing at throughout this thread. The badge answering the weaker question wasn't a design choice — it was simply the piece that never moved.

So the badge now requires a complete control. Your arithmetic came out exactly:

Causal-Safe 36 -> 33
rows with no flag 33
complement violations 0 / 39
badge + flag on one row 0

The badge is the complement of the flag rather than a fourth opinion.

Your statement of the precedence is better than ours. We wrote the rule that 위험 outranks control-completeness, but we never drew out what it means: the flag reports the worst thing known about a row, not the last thing checked, which is precisely what makes it safe to filter on. Zyphra/Zamba2-1.2B at 3/3 with 위험 is the proof case, as you said. And our 34 should have been 33.

Solar was the one that stung. Both surfaces agreed while the object disagreed with itself, which our cross-surface check cannot catch by construction. Counting the unique top-level keys gives 13, not 12 — safety_flag is the extra one.

Deciding the spelling from the object's own caveats was the decisive move. With White-box D1/D7 remains pending sitting right there, model-level ... not confirmed reads as a result from a pass we haven't run. Same shape as 2/3 versus 0/3, and as you put it, the looser string is the one that escaped. causal and meta are the single source now; the five top-level duplicates are gone.

positive_control went with it. A row where the probe never ran now carries the key in causal with a null value, so one model no longer returns two answers.

The ?model_id= regression we did not know about. Confirmed and restored as an alias on both the backend and the Space route. Renaming a public API parameter is a silent break; that one's on us.

Thanks for retiring the causal_score push as well. One channel doing the job cleanly is the right call.

complement violations 0 / 39
leaderboard vs model_report 0 / 39
Solar top-level causal_verdict removed
?id / ?model_id 200 / 200
sha 9b7b76b5fe

Good discussion.

Both were real — the two surfaces disagreeing, and the flag channel running backwards. Both are fixed, and they had one cause.

Yesterday we attached the positive_control caveat to the row only. The place that produces the verdict was left alone, so two endpoints with opposite precedence rules ended up emitting different strings. A value patched in one presentation layer will diverge from every other one; that lesson landed.

The root was in the generator:

live = pc_tot > 0 and pc_ok >= max(1, pc_tot - 1)

That single line let 2/3 count as a full control, so it went out with the full-confidence string. The asymmetry you flagged — a caveat at 0/3 and nothing at 2/3 — was being manufactured there. It's now three-way. score is untouched: moving it moves DHS and the grade, which isn't what this fix is about.

The flag point was the more useful one. A caveat that lives only in a verdict string is invisible to anyone reading filters or warnings. 인과안전.flag now derives from the same basis as the verdict:

leaked → 위험 (risk)
control short of full → 주의 (caution)
probe never run → 주의 (caution)
control complete → no flag

Current state:

tiiuae/Falcon3-10B-Instruct-1.58bit 0/3 클린 (미검증 — 양성대조 약함) 주의
tiiuae/Falcon3-10B-Base-1.58bit-prequantized 2/3 인과 안전 (미검증 — 양성대조 2/3) 주의
tiiuae/Falcon3-10B-Instruct 2/3 인과 안전 (미검증 — 양성대조 2/3) 주의
nvidia/Nemotron-H-8B-Base-8K 2/3 인과 누설 (LEAK) 위험
Zyphra/Zamba2-1.2B 3/3 인과 누설 (LEAK) 위험
upstage/Solar-Open2-250B none API scoring caveat 주의

39 rows · 0 disagreements between leaderboard and model_report · the 34 complete-control rows carry no flag

One thing worth recording: our first patch removed Solar's 주의, since it has no control value at all. That pointed an unprobed row toward looking safer than a probed one, so we corrected the rule. A caution shouldn't be lowered in either direction.

And to answer what you asked earlier — yes, a different writer. The current row writer emits all eight fields. The sixteen thin rows were left by an older one. They're rebuilt from reports/ now, and positive_control is in the writer so it rides along from here.

The rebuild surfaced something else. A row that existed in reports/ but had never reached the table appeared on the public board — a model whose verdict is 측정 실패 (measurement failed), sitting at rank 38 with grade F. That was our harness failing to load it, not a property of the model, and it contradicts our own rule that a missing measurement is not a zero. It's off the board, and failed measurements are now blocked from it in code. A row rebuild is itself an act of publication, and we hadn't been treating it as one.

Space sha 06e1cd4b22.

Good catch.

Confirmed, and the distribution matches yours exactly — 3/3 on 34, 2/3 on 3, 0/3 on 1. One clarification: it's 38 reports rather than 39. upstage/Solar-Open2-250B has no control value because no causality probe was run on it, which is the same reason its xray is null.

The part worth fixing was less the missing field than the fact that our own schema wasn't consistent with itself. At 0/3 it already says so — 클린 (미검증 — 양성대조 약함). So the language for a weakened control existed; it just wasn't applied at 2/3, which went out at full confidence. That asymmetry is exactly where you pointed.

One distinction we'd add: a weak control degrades a verdict asymmetrically by direction. A dropped control means reduced detection sensitivity, so on a CLEAN row "we didn't find one" stops underwriting "there isn't one." On a LEAK row the finding is already positive and a missed control doesn't move it. Splitting the three that way:

nvidia/Nemotron-H-8B-Base-8K 2/3 LEAK conclusion unaffected
tiiuae/Falcon3-10B-Base-1.58bit-prequantized 2/3 CLEAN needs the caveat
tiiuae/Falcon3-10B-Instruct 2/3 CLEAN needs the caveat

So two rows, not three.

Deployed. positive_control now rides on the row and is visible in the API, and any CLEAN verdict whose control falls short of 3/3 carries the wording we already used at 0/3. No new vocabulary was needed. LEAK rows are untouched.

positive_control on rows absent -> 38/39 (Solar null, correctly)
distribution 3/3 34 · 2/3 3 · 0/3 1

tiiuae/Falcon3-10B-Instruct
before 인과 안전 (CAUSAL-SAFE)
after 인과 안전 (미검증 — 양성대조 2/3)

tiiuae/Falcon3-10B-Base-1.58bit-prequantized same change
nvidia/Nemotron-H-8B-Base-8K unchanged (LEAK, 2/3)

Two verdict strings changed, nothing else. No field lost a value, and the preserved columns are identical across all 39. Space sha 72dc3d24b3.

For the record, the value wasn't entirely off-screen before — the report modal renders 양성대조 2/3 as it always did. But it wasn't on the row, wasn't in the API, and the verdict string gave no hint of it. That reading stands.

Good catch.

Answering your question first: yes, and the drop is in the bake, not the run.

I opened all sixteen reports. Every one of them had already been measured — causal_score 100, leaked false, positive control 3/3, with the full T-profile. The one exception is Falcon3-10B-Instruct at 2/3, which we're looking at separately. The row writer was dropping eight fields on the way out.

The trap you flagged was real, and there were two faults rather than one. The Space overlay was replacing a backend row wholesale instead of merging it field by field, so a thin baked row could erase what the backend supplied. Fixing the writer alone would have been undone on the next bake. Both are now fixed: rows rebuilt from reports/, and the overlay changed so a null can no longer overwrite a value.

Measured after deploy:

              before     after

causal_verdict 16/39 0/39
leaked 16/39 0/39
causal_score 16/39 0/39
behavior 16/39 0/39
arch 16/39 0/39
params_M 16/39 0/39
created 16/39 0/39
xray 17/39 1/39 upstage/Solar-Open2-250B

The remaining cell stays as it is, for the reason you gave — there is no white-box x-ray for an API-only audit, and that null is the schema saying so. Preserved fields (dhs, grade, badges, categories, ts) are unchanged across all 39, and no row lost a value it previously had. Space sha ffed591171.

On the mamba2 timeline, we reached the same conclusion independently. We only checked two points per family — first release and 5.14.1 — and got 12/133, 63/514, 25/134. Walking every release is the better method, and the numbers agreeing is a useful cross-check for both of us. We also probed seven models behaviorally on 5.16.1; max_delta 0.000e+00 across the board, Nemotron-H included.

On the trigger: model-add is the cheaper one, but it isn't the same kind of check.

When a new modeling_*.py first lands there are usually no weights yet, so what's available at that moment is a structural read of the chunk-recurrence block — text-level, effectively free, and it fires on exactly the two events you named, 4.49.0 and 5.3.0. A behavioral probe needs a checkpoint, which puts it on the release side. So it isn't one or the other. Entry gets caught at model-add; proof happens at release. Two tiers, and the expensive one only ever confirms what the cheap one flagged.

Good catch. It saved us a run that would have produced nothing.

All three points hold. Let me correct our own numbers first.

The 88 releases / 539 days I quoted earlier came from counting GitHub tags, which isn't the right denominator — a pinned requirement resolves against PyPI final releases. Recounted on that basis it matches yours, and it splits per model:

model first release w/ file defect window exposure
zamba2 4.49.0 2025-02-17 4.49.0 – 5.14.1 63 releases / 513 days
nemotron_h 5.3.0 2026-03-04 5.3.0 – 5.14.1 25 releases / 133 days

Your point about the missing floor is right: without one, the row claims a leak on releases where Nemotron-H has no file at all. modeling_nemotron_h.py is 404 through v5.2.1 and appears at v5.3.0 already carrying the divergent form. Intervals should be per model, with both edges named.

Same view on the ceiling. >= 5.15.0 is a claim about releases that don't exist yet, and 5.16.0 and 5.16.1 landing today makes that concrete. We pulled both as well — six files, states_permuted 0, transpose(1, 3) 1, states[:, :, None, ...]).sum(dim=1) 1, in every one. So the field should be the range a verdict has actually been tested across, with a release number on the right edge rather than an inequality:

Zamba2-1.2B LEAK 4.49.0 – 5.14.1
CLEAN 5.15.0 – 5.16.1 (verified 2026-08-26)

On your last question — the unlock is separating the causality probe from the full audit. A complete diagnostic run is hours per model, which is why it can only ever be calendar-driven. The causality check on the pure-PyTorch path is minutes on CPU. Once it stands alone, the trigger can be the release itself: watch PyPI for a new final, install it into a probe environment, run the affected families, and either extend the right edge of verified_clean to that release number or flip the verdict and flag a regression. No calendar involved.

The six-family framing was useful. bamba and falcon_h1 aren't on our board yet, and the granite entries are there with the causality field empty. We're filling those in now, on 5.16.1.

Good discussion.

Thanks for taking the time to check this. Let me put the facts and the measurements side by side.

The verdict you're looking at was measured on 2026-07. The timestamp is in the record itself (ts=1785031072 for Zamba2, 1785033374 for Nemotron-H). The fix you're referring to shipped in v5.15.0 on August 10, so our measurement predates it by 15 days. Every release available at that time carried the defect, so the verdict was accurate as of measurement. Publication slipped to August for IP reasons on our side; the measurement itself was not taken in August.

I also checked how far back it goes. The defect was present from v4.49.0 (2025-02-17), when Zamba2 support landed, through v5.14.1 (2026-07-16) — 88 releases across 539 days. The fix is 16 days old.

And yes, it is resolved now. We verified that ourselves rather than taking it on faith. Same checkpoint, same seeds, CPU fp32, no mamba_ssm / causal_conv1d (pure PyTorch path). Only the library changed:

T transformers 5.14.1 transformers 5.15.0
128 0.0 0.0
256 0.0 0.0
320 7.549e-03 LEAK 0.0 CLEAN
512 2.574e-03 LEAK 0.0 CLEAN
768 6.917e-03 LEAK 0.0 CLEAN

three seeds @ T=320
7.5e-3 / 1.2e-2 / 5.4e-2 0 / 0 / 0

Leakage opens exactly at the chunk boundary (256 → 320), and on 5.15.0 all three seeds are exactly zero. We diffed the source as well: the offending permute form appears twice in 5.14.1 and zero times in 5.15.0, replaced by transpose(1, 3) + sum(dim=1). Nemotron-H-8B is likewise clean across the full range on 5.15.0.

For context, we independently identified this defect in July and filed a report and a patch upstream. It wasn't picked up, and the actual fix landed through a separate refactor. No claim to credit here — just noting this wasn't news to us.

How to present a verdict like this is something we're still refining. Our current thinking is that the answer isn't to retract the verdict but to scope it:

Zamba2-1.2B — LEAK on transformers ≤ 5.14.1 / CLEAN on ≥ 5.15.0

"Fixed upstream" and "safe in the stack you are actually running" are different statements. Pinned requirements and container images don't move on their own. Our own diagnostic server was still on 5.14.1 as of today, which is why the re-run reproduced the same leakage. Drop the verdict entirely and anyone still on an older pin is left with nothing.

Which is, I think, the more interesting part of this. Not a single weight changed, yet swapping one library flipped the verdict. A defect like this doesn't show up in a model card or in the weights — it's only visible when you measure the stack as it executes. And because this path is the one used in training, prefill and teacher-forced evaluation, it fails quietly.

We agree that causality is a property of (checkpoint, library version, kernel path) rather than of the checkpoint alone. We'll take your input into account as we refine version and kernel metadata in our records and the cadence for re-measuring against upstream changes.

Good discussion. Thanks.

I’ve had DeepSeek pro trading on a $1k account for 3 months with a minimal prompt. It does one review pass with a mid-level OpenAI model for the time (right now GPT 5.6 Terra)

It’s down $23 rn (2.3%). Started out super conservative (would enter exit trades for $1-2 profit). Told it to “aim for 10% growth in one month) about halfway in

Thanks for posting a real number instead of a screenshot of a good week — that is rarer than it should be.

Here is the thing your −2.3% runs into, and it is exactly why we built this. Over three months on one account, that result is inside the band where skill and luck are indistinguishable. Not because 2.3% is small, but because nobody has told you how wide the band is. We measured it first: twenty thousand zero-skill players per asset, charged the same fees. Over a 122-day season, pure chance clears +86.6% on Bitcoin and +51.7% on NVIDIA at the 95th percentile — and only +9.2% on gold. The same number means completely different things depending on what you traded.

So a conservative agent that ends slightly negative is not evidence of a bad agent, and the one that ended +40% on Bitcoin is not evidence of a good one. Without that line drawn in advance, three months of work produces a number you cannot interpret.

Your setup is a good fit for this. Same DeepSeek agent, same minimal prompt, same review pass — point it at our MCP server and it gets four tools (get_rules, get_data, submit_position, check_score). One line to connect:

claude mcp add --transport http finchal https://final-bench-finchal.hf.space/mcp
--header "X-Finchal-Key: "

Then you tell it to enter, and it reads the rules itself. No real money, leverage fixed at 1, and the position holds until it is replaced — so an agent that reviews once a day works fine.

One detail from your message that we think about a lot: you started conservative, then told it to aim for 10% a month. That midcourse change is the hardest thing to account for in any self-reported track record, and it is why the season runs a fixed 122 days on a fixed clock. Whatever you change, the whole record stands, start to finish.

The scorer is public. Read what it does before you enter — including the eight self-tests it runs on itself, one of which checks that entering on the bar that gaps does not capture the gap.

If the prize were the point, you would be right — and $2,000 across four assets would be a strange way to monetise a working edge.

The prize is not the payoff. It is the entry fee we pay so the thing gets run at all. What we are actually building is a record: a fixed 122-day window, leverage locked at 1, live prices, real execution costs, one public scorer, and every entrant's curve standing from start to finish — including the ones that lose. That does not exist anywhere right now. What exists is winners getting interviewed and losers leaving no trace.

The organisers do not compete, for the same reason. Three textbook baselines sit in the table instead — buy and hold, volatility targeting, random — because a leaderboard without a scale cannot be read.

And "if you can forecast the market" is doing a lot of work in that sentence. We tested sixty candidate variables directly and found no confirmed predictive power over short-horizon direction. That is a real finding, and this contest starts from there rather than assuming the opposite. If it turns out nobody clears the luck ceiling in four months, that is also a result — and it is one nobody has written down yet.