--- title: MiniMax-H3 Audio to Video emoji: πŸ”Š colorFrom: gray colorTo: blue sdk: gradio sdk_version: 6.20.0 app_file: app.py pinned: false short_description: A first frame plus your own soundtrack, driving MiniMax-H3 suggested_hardware: zero-a10g --- # MiniMax-H3 β€” a first frame and a soundtrack you provide MiniMax-H3 denoises video and audio as **one packed sequence**, under two schedules (`shift = 12` for video, `shift = 3` for audio). In the `fl2va` workflow the keyframe anchors the leading video rows and the audio rows are drawn from noise like any other generated row, so the model invents its own soundtrack. This Space hands it the audio rows instead. The upload is encoded by the model's own audio VAE, normalized the way the pipeline normalizes any soundtrack, and imposed on the audio rows at every step β€” so the video is denoised alongside a soundtrack it does not get to choose. **The first frame is optional.** With one, the request is `fl2va` and the keyframe anchors the leading video rows. Without one it is `t2va`, the same audio conditioning against a shorter packed sequence β€” `[text | target audio | target video]`, no keyframe condition block β€” and the canvas comes from the Canvas dropdown alone, since there is no aspect ratio to follow. None of the three conditioning blocks reads the keyframe, so the two branches share them; what differs is the core they sit in. Expect more seed-to-seed variance without the keyframe: that is the anchor doing its job in `fl2va`, not a fault in `t2va`. ## How the soundtrack is imposed The audio rows are **locked**: your clip is written into them clean and written back clean after every scheduler step, and the row-timestep plan presents them at `t = 1.0` β€” finished β€” which is the presentation `ref2va` gives a *reference* soundtrack. The video noise is still the request generator's first draw, so one seed gives one starting video latent. `h3_a2v_blocks.py` also implements a `blended` mode (rows re-noised to that step's audio sigma, off the shift-3 audio schedule) and an `off` control leg. Neither is reachable from this app: comparison is finished and locked is what the Space runs. They stay in the module because the a2v LoRA sweep imports it. The file you get back carries **your** upload as its soundtrack β€” with the audio rows given, the model's own audio output is that same waveform round-tripped through the audio VAE (Pearson 0.988, 16.2 dB SNR on the example clip), so muxing the upload just skips a lossy step. ## Prompt upsampling **Upsample prompt** hands the rewrite to the conditioner Space, which rewrites the prompt into MiniMax-H3's trained format on its own Qwen3-VL β€” *looking at your keyframe*, which is why the rewrite belongs on that side β€” encodes the rewrite instead of what you typed, and returns it. It comes back in the **Upsampled prompt** panel, which stays hidden until a request asks for one. It is a second GPU booking on the conditioner, ahead of the encode's own; nothing about this Space's own reservation changes, since that only ever covers the denoise. ## Speed: the turbo LoRA The Space runs [`larryvrh/MiniMax-H3-Turbo-Lora`](https://huggingface.co/larryvrh/MiniMax-H3-Turbo-Lora) (`minimax_h3_turbo_v4_step600_ema.safetensors`) at scale 1.0, at **6 NFE**. In the ten-leg a2v LoRA sweep it was the only adapter that beat its own step-matched control rather than merely matching it (mouth-motion proxy 0.4308 vs 0.2623, sharpness 252.5 vs 230.3) while still running ~3.7x faster than the base schedule's 27 NFE. `num_inference_steps` counts sigma grid points **including the terminal one, which has no model evaluation**, so 6 NFE is **7 steps**. The Steps slider stays adjustable β€” larryvrh's card says 6–8 β€” and `get_duration` sizes the GPU reservation from the step value in the request, so a hand-raised count still reserves what it will use. It is trained against the *released* checkpoint, so on the pruned one served here its AdaLN factors go through the shipped fullβ†’pruned projection and 51 constant-term offsets ride alongside them. A correct attach is therefore **363 modules** (51 AdaLN + 312 attention/feed-forward) **and 51 AdaLN projection offsets**; both counts are checked at startup and printed. A mismatch deletes the adapter β€” offsets included, which PEFT's `delete_adapters` does not touch β€” falls back to the base model at the base step count, and says so in the status line rather than quietly running half-attached. `H3_TURBO=0` as a Space variable loads the base model instead, at the base step count. Env only: a debugging escape hatch, not a user-facing choice. ## Split deployment MiniMax-H3 is 195.9 GiB in bfloat16 and a ZeroGPU Space is evicted at 150 GB of storage, so `MiniMaxH3Blocks` is cut at its `text_encoder` step. This Space is the denoising half and runs the pruned checkpoint, [`multimodalart/MiniMax-H3-Pruned`](https://huggingface.co/multimodalart/MiniMax-H3-Pruned) β€” the released weights with their AdaLN input projections folded onto their reachable rank, 51.3 GB instead of 77.3 GB and latency-neutral. The 62.14 GiB Qwen3-VL conditioner runs in [`qwen3vl-conditioner`](https://huggingface.co/spaces/multimodalart/qwen3vl-conditioner), which this Space calls over the gradio API for every request; `prompt_embeds` + `text_token_tags` is the whole wire format between them. `diffusers` is pinned to a **main** commit (`d5baa4fb`), not to the MiniMax-H3 follow-up PR the sibling Spaces run: that PR branched before the LoRA work landed and carries no `MiniMaxH3LoraLoaderMixin`, so no turbo LoRA can be read against it. `h3_a2v_blocks.py` holds the conditioning itself: an encode step that turns the waveform into channel-major audio rows, a timestep step that presents those rows as clean when they are locked, and a loop scheduler step that imposes them after every update. Those three go into both an `fl2va`-shaped core and a `t2va`-shaped one, and conditional wrappers of the same shape as main's own pick between them on `image`. Two-way rather than main's three-way: the upstream `ref2va` branch declares `transformer_ref`, the second checkpoint partition, which this Space must never load. ## Example assets `examples/subject.png` (a studio portrait, free to use) and `examples/voice.wav` (6.5 s of speech) are the ones the sibling [`minimax-h3-reference`](https://huggingface.co/spaces/multimodalart/minimax-h3-reference) Space ships.