Transformers documentation

FBGEMM

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v5.23.0).
Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

FBGEMM

FBGEMM (Facebook GEneral Matrix Multiplication) FP8 quantization stores weights in FP8 with one scale per output channel, and quantizes activations to FP8 with one scale per token at runtime (W8A8). The matmul runs on FBGEMM’s row-wise FP8 kernel. Use it for FP8 weight and activation quantization on Hopper or newer NVIDIA GPUs, or on Intel XPUs, without calibration data.

The table below summarizes what FBGEMM supports in Transformers.

ConfigFbgemmFp8Config
quant_methodfbgemm_fp8
WorkflowQuantize while loading
Bits8
HardwareCUDA (compute capability 9.0+), XPU
Save✅
Fine-tune❌
Auto-compile❌

Install FBGEMM

On a CUDA GPU, install the fbgemm-gpu and Accelerate libraries. If you have installation issues, try the fbgemm-gpu nightly release.

pip install -U fbgemm-gpu accelerate

On an XPU, Transformers downloads the FP8 quantization kernel from the kernels-community/fp8-fbgemm repository on the Hub instead. Install the kernels and Accelerate libraries.

pip install -U "transformers[kernels]" accelerate

Quantize a model with FBGEMM

Create a FbgemmFp8Config and pass it to from_pretrained().

from transformers import AutoModelForCausalLM, FbgemmFp8Config

quantization_config = FbgemmFp8Config()
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-8B",
    quantization_config=quantization_config,
    device_map="cuda",
)

FBGEMM always loads the model in bf16. If you pass a different dtype, Transformers logs a warning and switches it to torch.bfloat16.

activation_scale_ub caps the per-token activation scale when activations are quantized at runtime. It defaults to 1200.0, and you can change it when you load a pre-quantized checkpoint.

Limitations

  • FBGEMM needs a CUDA GPU or an XPU. Loading without one raises an ImportError.
  • On CUDA, FBGEMM needs compute capability 9.0 or higher, like an H100. An older GPU raises a ValueError.
  • Quantizing while loading can’t offload to the CPU or disk. A device_map that includes "cpu" or "disk" raises a ValueError. A pre-quantized checkpoint can offload.
  • FBGEMM quantizes nn.Linear layers and Llama 4 text experts. Fused expert modules in other mixture-of-experts models stay unquantized.
  • FBGEMM-quantized models can’t be fine-tuned.

Resources

Update on GitHub