| --- |
| license: mit |
| language: |
| - en |
| - ja |
| --- |
| |
| # Model Card |
|
|
| ## Overview |
|
|
| Rize v0.6 is a causal language model for research and general text generation. |
|
|
| It uses a decoder-only Transformer architecture with Mixture-of-Experts (MoE) layers and KDA attention based on Kimi Linear-style attention. |
| The model is intended for experimental use, including architecture research, optimizer research, and Japanese/English text generation evaluation. |
|
|
| ## Model Size and Architecture |
|
|
| Rize v0.6 has approximately **4.5B total parameters** and approximately **1B active parameters per token**. |
|
|
| Main architecture points: |
|
|
| * decoder-only Transformer |
| * 19 hidden layers |
| * hidden size: 1536 |
| * 12 attention heads |
| * KDA / Kimi Linear-style hybrid attention |
| * 64 routed experts |
| * top-4 expert routing per token |
| * 1 shared expert |
| * vocabulary size: 151,669 |
| * maximum context length: 32,768 tokens |
|
|
| ## Training |
|
|
| The model is trained with autoregressive next-token prediction. |
|
|
| Training summary: |
|
|
| * pretraining data: approximately **1T tokens** with 8k context. |
| * midtraining data: approximately **50B tokens** with 32k context. |
| * posttraining data (SFT & modified-SLIME): approximately **5B tokens**. |
| * optimizer: **ROCA Optimizer** |
| * dtype: bfloat16 |
|
|
| ROCA is a Muon-based optimizer designed to improve convergence for MoE language models. |
| The technical report, **“ROCA: Routing-Aware Sparse Row Correction for Muon Optimization in Mixture-of-Experts Language Models,”** is planned for release on arXiv. |
|
|
| ## Evaluation |
|
|
| | Benchmark | Score | |
| | ----------------- | ----: | |
| | MT-Bench Japanese | 4.89 | |
|
|
| ## Intended Use |
|
|
| This model is intended for: |
|
|
| * language modeling research |
| * MoE architecture experiments |
| * optimizer research |
| * general text generation |
| * Japanese and English generation evaluation |
|
|
| This model is not intended to be used as a source of factual truth or professional advice. |
|
|
| ## Limitations |
|
|
| The model may: |
|
|
| * generate incorrect or misleading information |
| * reflect biases in training data |
| * produce unsafe or inappropriate text |
| * perform inconsistently across languages, domains, and prompts |
| * fail on high-stakes or specialized tasks |
|
|
| ## Safety and Responsible Use |
|
|
| Users should review outputs before real-world use. |
|
|
| The model should not be used on its own for: |
|
|
| * medical advice |
| * legal advice |
| * financial advice |
| * safety-critical decisions |
| * sensitive personal decisions |
|
|
| Human oversight is required. |
|
|
| ## Disclaimer |
|
|
| This model is provided for research and experimental purposes only. |
|
|
| The FA Research Team makes no guarantees regarding accuracy, reliability, safety, or fitness for a particular purpose. Use of this model and its outputs is at the user’s own risk. |
|
|
| ## Contact |
|
|
| FA Research Team |
