GlobalDiff checkpoints and retrained motion generators

Official checkpoints for Mitigating Error Accumulation in Co-Speech Motion Generation via Global Rotation Diffusion and Multi-Level Constraints (AAAI 2026), a diffusion approach to long-horizon co-speech gesture generation.

• Paper · Hugging Face Paper page
• Project page · Code
• Published version
• Generated inference data

Retrained motion generators (2026-11-13)

The released motion generators were retrained on the BEAT2 English training split (25 speakers). Both use the all-speaker setting; --pid 2 selects Speaker 2 at inference.

Each complete bundle contains the generator, matching CONFIG.json, the official frozen SimpleSpeechModel, required source files, an example eight-frame training seed, infer_wav.py, and SHA-256 verification files.

Checkpoint	Complete runnable bundle	Raw generator	Matching configuration
Epoch 880, CFG=1.25	Download ZIP	generator.pt	CONFIG.json
Epoch 630, CFG=1.25	Download ZIP	generator.pt	CONFIG.json

The generator architecture is DiffusionDITNetPartsFixedExpressions2PostNorm with 384 hidden channels, 6 layers, 8 seed frames, 333 motion channels and 31 person embeddings. Each raw file is a native PyTorch state dictionary with 249 tensors. In the legacy zero-based naming convention, epoch 880 corresponds to epoch_879, and epoch 630 to epoch_629.

The earlier best_pid_2.zip and all-speaker inference archives contain generated predictions, not motion-generator weights. The existing VAE, speech and AESK assets on Hugging Face are retained.

Custom WAV inference

Use the complete bundle for inference on new speech; BEAT2 test data is not required. The bundle was tested with Python 3.12 and PyTorch 2.10.0 (CUDA 12.8 on Linux). Install the appropriate PyTorch build for your platform. For the tested Linux CUDA environment:

python3.12 -m venv globaldiff-infer-env
source globaldiff-infer-env/bin/activate
python -m pip install torch==2.10.0 --index-url https://download.pytorch.org/whl/cu128
curl -fL 'https://huggingface.co/X-Zhang/GlobalDiff/resolve/main/retrained_20261113/globaldiff_e880_all_cfg1p25.zip' -o globaldiff_e880_all_cfg1p25.zip
unzip globaldiff_e880_all_cfg1p25.zip
cd globaldiff_e880_all_cfg1p25
python -m pip install -r requirements-inference.txt
python verify_sha256.py
python infer_wav.py --checkpoint generator.pt \
  --wav /path/to/input.wav \
  --seed-pose seed_pose_speaker2.npz --pid 2 \
  --out motion.npz --device cuda

For epoch 630, download its ZIP and use the corresponding extracted directory; the inference command is the same. CONFIG.json automatically sets CFG=1.25. Use --device cpu with a CPU-compatible PyTorch installation for CPU inference, and --seed 1 to change sampling noise. Use a new output filename for each invocation.

• WavLM: patrickvonplaten/wavlm-libri-clean-100h-large, pinned to revision e70e3a062ec399c46008ee55d1fb52c7ba338d5c. It downloads on first use. For offline inference, pass --wavlm /path/to/that/local/snapshot; the bundle verifies its required files against the supplied manifest.
• Seed pose: eight SMPL-X frames at 30 FPS, in a NumPy .npz containing local axis-angle poses of shape [>=8, 165] and trans of shape [>=8, 3]; optional betas contains the 300-dimensional body shape. The included example comes from a Speaker 2 training clip. For another speaker/avatar, supply a matching seed and trained person ID.
• Trained IDs: 1,2,3,4,5,6,7,9,10,11,12,13,15,16,17,18,20,21,22,23,24,25,27,28,30. ID 0 is the unconditional identity; unsupported IDs are rejected.
• Preprocessing and sampling: the script mixes audio channels to mono, resamples to 16 kHz, uses two-second/60-frame windows with eight seed frames, and applies five midpoint steps. The audio hop is 27,734 samples. It converts seed rotations to global 6D internally, pads the final partial audio window, and trims output to the audio duration. No manual motion normalization is required.
• Output: local SMPL-X axis-angle poses [T,165], trans [T,3], expressions [T,100], body shape and 30 FPS metadata. Parameter inference does not require an SMPL-X mesh asset; rendering requires the separately obtained SMPL-X model. CPU FP32 and CUDA FP16 sampling are not bitwise identical.

The two frozen VAEs are used for training losses and are not required by this custom-WAV entrypoint. Third-party datasets, pretrained encoders and SMPL-X assets remain subject to their respective terms.

Original auxiliary files

The checkpoint files mirror the official repository while preserving their repository-relative paths:

Path	Role
Scripts/VAE/ckpt/split/global/MaskedVAE2-HorizonFlip/best.pt	Global rotation VAE checkpoint
Scripts/VAE/ckpt/split/global/MaskedVAE3-HorizonFlip/best.pt	Global rotation VAE checkpoint
Scripts/FM/ckpt/split/SimpleSpeechModel/best.zip	Speech model checkpoint archive
Scripts/FM/EMAGE_VAE/AESKConv_240_100.bin	Evaluation/representation asset
These original auxiliary assets are preserved. See the custom-WAV instructions above for the new runnable bundles.

Verify the files from the repository root with sha256sum -c SHA256SUMS.

Terms

This repository mirrors research artifacts released by the authors. No new license is granted by this model card. Code, BEAT2, SMPL-X, pretrained encoders, and other third-party assets remain subject to their respective terms.

Citation

@inproceedings{zhang2026globaldiff,
  title={Mitigating Error Accumulation in Co-Speech Motion Generation via Global Rotation Diffusion and Multi-Level Constraints},
  author={Zhang, Xiangyue and Li, Jianfang and Ren, Jianqiang and Zhang, Jiaxu},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={40},
  number={15},
  pages={12834--12842},
  year={2026},
  doi={10.1609/aaai.v40i15.38281}
}