☁️ out of ideas

CodeSoft

25 5 45

https://codesft.dev

codesftdev

AI & ML interests

Working on small language models

Recent Activity

updated a bucket about 12 hours ago

CodeSoft/slm-arena-data

liked a dataset 6 days ago

AtomicChat/calib-corpora

liked a model 8 days ago

google/embeddinggemma-2

View all activity

Organizations

Sorbet Mini Experimental is out!

It's not the best model in the 5M parameter range, but it's going to be a really useful model to train on top of. This release was mostly to prove that the model actually works. I trained it on 150M tokens from TinyStories in about 12 minutes.

Does anyone have any tips on training models in this size range? I want to make the full Sorbet Mini release as good as it can be.

Post

194

Over the past week or so, I've been working on some models, those released being CodeSoft/sorbet-25m and CodeSoft/sorbet-v2-25m. In general, I'm a little confused because no matter what hyperparameters I change or datasets I add/remove, the benchmarks never move up.

In a recent project, where I attached a TN-gram block to Sorbet-v2-25M, it still stayed the same on benchmarks despite the TN-gram clearly learning (due to the perplexity being lower with the TN-gram attached). When I changed the corpus to favor higher density text (the first paragraphs of Wikipedia articles and synthetic math), the benchmarks either stayed flat or went down.

Does anyone have ideas on what I can do to improve my models? I'd really appreciate any feedback!