Slayer1B 1.07 — training checkpoints
Native training checkpoints of the run that produced Slayer1B 1.07,
for resuming or studying training (full trainer state, not a Transformers/safetensors package): pretraining continued
from the final 1.06 checkpoint ckpt_13848.pt (sha256 1fc7396c…, published in
Slayer1B-1.06-checkpoints). Each file holds the model
weights and the optimizer state.
Status: training complete at step 22327 (6 November 2026). The table below lists only the files that are in this repository now; the remaining saved checkpoints up to 22327 are added one by one. Publishing these native training checkpoints is separate from a 1.07 model release (converted weights and model card), which has not been published. Every file listed was compared after upload with the checksum list written by the trainer: the LFS object id (sha256) and the size match.
| Start | ckpt_13848.pt (final state of 1.06, constant lr 6e-4) |
| Steps | 13849 → 22327 (8 479 steps × 1 048 576 tokens), 256 sequences × 4096 tokens per step |
| Schedule | constant lr 6e-4, no decay |
| Data | SlayerLab/slayer-1.07-pretrain, 8 891 005 707 tokens, one pass |
| Hardware | 4 × NVIDIA RTX 4090, DDP, gradient accumulation 32; global batch identical to the 8-GPU runs |
| Interruptions | stopped on 4 November at step 17130 and resumed on 5 November from ckpt_17000.pt; steps 17001–17130 were trained twice, and the loss at step 17010 was identical in both runs. Stopped again on 5 November at step 19330 and resumed from the rolling checkpoint at step 19250 (not published); steps 19251–19330 were trained twice, and the loss at step 19260 was identical in both runs. Checkpoints up to step 17000 were saved before the first interruption, 18000–19000 between the two, later ones after the second resume |
| Trainer | train_gpt_ref r24.2 (sha256 46f83b51…); model definition train_gpt_ref_r23.py (unchanged) |
| Training curve and evaluations | https://track.fabryka.ai/run/4973c068-8dd5-59d8-b6c4-671ca3beb925 |
SHA-256
| step | sha256 |
|---|---|
| 13998 | f1ec1380a956f7fa002446902464df26314901d9cf2afe98ecbd00dec34d3529 |
| 14000 | 752d9d8ec48a6473be78dbccdebfd61621d3b24d78932b1b5f4616da415d28b8 |
| 15000 | f89d7f3c6c223ec00ac4d85cb85546d08194808e419dcda3821ebd7ba4915ae1 |
| 15730 | dd7170909294dbdec1282617a12d8c215037c27be5075e421a68ae6b1245e70b |
| 16000 | c18be246a5ae83be67a61fddf479fe6f906936ccd31f93dccf52fb4f03ca2b48 |
| 17000 | c4961c9ced633fef2efd8d28cc16217154f82b25f4a178ce31865b4523eed95b |
| 18000 | fd6e2360badc1ed79e931ab36864440235d67d86c835592b6c9d788a9b9f0980 |
| 18590 | 6779c0bbfe84af003e5a708c1e9a02914e492a40c3f24278bbd096635a254467 |
| 19000 | 9ae730323e29b929f2ba0f753ee75be43626b804f1a5c452d5cee16d0dacfaf7 |
| 20000 | 33146051461e6102cb97f72b6d5a0913e228ca55f165fa9dc1db8eaa057aeefe |
| 21000 | 05fc22d03e934f39bf70d57ac8f7cf20186df5aa4458250338f76d650c5a5233 |
| 21450 | 7c1dcd9e3d0d4bc03f296788c196eadf1e3855e76f414bb2ea2b96d5e9aff10d |
| 21877 | 2fa5e3e0919b4d8a33b5fac717f593b6615866db1a090f538fc2f2e6bda03c68 |
| 21927 | 03b30237dcfabecc7f99fd1703835f7b3a03ce19fed2f3a44ad8578538621bc7 |
| 21977 | 49768faad2a36bdb20beab807a7a44e4c5660e7047644fa2826cc6f3b0c7326c |
| 22000 | 8e5b5f33cedbe2b3523f1fe87470eb522d165926303e2eeb59fdd5852fd89b06 |
| 22027 | a12862b21bde36b7a3a3ec47df5f27891f0b055ef3636400af99320d6a33051a |
| 22077 | ff2e2ca16028fb66c607b0696b34beee1e9e1089624056741b82d8099c461338 |
| 22127 | 793962376892ed6ea8ebb5add94b98b14487ef742cc78cfb907303cdf2b250b7 |
| 22177 | f95c4a7df9262eee3b7121944583251e1c72c3bca5adaf2d1d9647977198d661 |
| 22227 | ab4dedce91acdef64e689ef699bda59e548490b4a96b52b4d9faebbd37b4d61c |
| 22277 | f1d7fe0397d809cff8aade1c2e3f5668dff8e7ea778e9266afdcd0339adda257 |
| 22327 | 48cdb5eb16b69e0ee9a7a2ed72895ff47c0ba57aca0f4516b1f7e9cba286685c |
Available training-time evaluations are in the Slayer1B 1.07 model card.
File format
The files are PyTorch pickles (torch.save dictionary of the trainer: model, opt (Muon + AdamW), step,
config); they contain only dictionaries, numbers, strings and tensors. Load with
torch.load(path, map_location="cpu", weights_only=True) (checked with torch 2.13 on ckpt_17000.pt; mmap=True
avoids reading the whole file into memory). Model class: train_gpt_ref_r23.py in the model repositories
(1.00–1.06). Every file is 8 860 287 151 bytes.
License
Apache-2.0 for the checkpoints. The training data includes CC BY-SA sources (the share-alike parts of the Polish core and legal texts, repeated from slayer-pl-8x3b); see the dataset cards of slayer-pl-8x3b and slayer-1.07-pretrain for the licenses of the data.