PrentisAI/Prentis-EQ-2
Image-Text-to-Text • 27B • Updated • 57 • 2
Geo Maciolek PRO
GeoMaciolek
AI & ML interests
None yet
Recent Activity
liked a model 2 days ago
PrentisAI/Prentis-EQ-2 liked a model 10 days ago
ggerganov/preset-dev liked a model 12 days ago
mradermacher/Prentis-EQ-2-i1-GGUFOrganizations
None yet
liked a
model 2 days ago
liked a
model 10 days ago
liked a
model 12 days ago
updated a
model 15 days ago
published a
model 15 days ago
Can't wait to try it!
1
#1 opened 16 days ago
by GeoMaciolek
PrentisAI (formerly HiveLabsAI) EQ-2 https://huggingface.co/PrentisAI/Prentis-EQ-2
1
#2957 opened 16 days ago
by GeoMaciolek
reacted to dipankarsarkar's post with 🔥 18 days ago
Post
3,176
I audited one of my own evaluations. The ranking did not hold up the way I expected.
Eight open models, one task: infer the structure of a prompt. Then ask again with the identical call. Caching off.
- Agreement between repeated identical calls (mean Jaccard) ranged from 0.39 to 0.96 across models.
- Only 35 of 127 prompt-model cells were perfectly reproducible on every run.
- I bootstrapped the reproducibility ranking over prompts. The two least reproducible models kept their rank in 99% and 86% of resamples. The middle four kept theirs in 27% to 48%.
So the table reliably finds the worst model. It does not reliably find the best.
Reproducible is also not the same as correct. F1 against gold annotations ran from 0.56 to 0.99.
By the audit date, 4 of the 8 model variants had been retired (HTTP 410). The study as specified can no longer be re-run. The saved outputs are what survives, so I published all of them: every inferred structure, every run, the prompts and the annotations.
Paper: How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure (2609.30074)
Dataset: dipankarsarkar/llm-evaluation-self-audit
Code: https://github.com/sarkar-dipankar/llm-evaluation-self-audit
How many of the leaderboard rankings you rely on would survive re-running the same calls?
Eight open models, one task: infer the structure of a prompt. Then ask again with the identical call. Caching off.
- Agreement between repeated identical calls (mean Jaccard) ranged from 0.39 to 0.96 across models.
- Only 35 of 127 prompt-model cells were perfectly reproducible on every run.
- I bootstrapped the reproducibility ranking over prompts. The two least reproducible models kept their rank in 99% and 86% of resamples. The middle four kept theirs in 27% to 48%.
So the table reliably finds the worst model. It does not reliably find the best.
Reproducible is also not the same as correct. F1 against gold annotations ran from 0.56 to 0.99.
By the audit date, 4 of the 8 model variants had been retired (HTTP 410). The study as specified can no longer be re-run. The saved outputs are what survives, so I published all of them: every inferred structure, every run, the prompts and the annotations.
Paper: How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure (2609.30074)
Dataset: dipankarsarkar/llm-evaluation-self-audit
Code: https://github.com/sarkar-dipankar/llm-evaluation-self-audit
How many of the leaderboard rankings you rely on would survive re-running the same calls?
- 2 replies
Any chance for a 1.1 on updated Qwen 3.8?
❤️👍 2
10
#1 opened about 1 month ago
by 
BigBeavis
liked 6
models 22 days ago
sky7350/Mica-v0.1-4B
Text Classification • 4B • Updated • 16.7k • 49
gghfez/gemma-4-31b-it-control-vectors
Text Generation • 306k • Updated • 607 • 22
Cudecnik/Qwen3.8-Flash-Next-refusal-projection
120k • Updated • 484 • 10
MonumentalSystems/Qwen3.8-Flash-Next-control-vectors
105k • Updated • 138 • 1
ginigen-ai/Edge-4B-TELL
Any-to-Any • 7B • Updated • 1.86k • 35
inclusionAI/Realtime-Venus
Any-to-Any • Updated • 475 • 109
liked a
Space 22 days ago
liked a
model 23 days ago
liked a
model 24 days ago
Surprisingly good low-slop creative writing benchmarks
1
#2 opened 28 days ago
by GeoMaciolek
liked a
model 28 days ago
