MiMo V2.5 Pro on one and two GPUs

I ran Xiaomi MiMo V2.5 Pro (1.02T total / 42B active) IQ2_S on RTX PRO 6000 Blackwell Max-Q GPUs. The checks cover speed and tool use for a resident multi-agent orchestrator candidate.

Video link: https://www.youtube.com/watch?v=tviNguY-HRE

Decode increased about 33%, from 12.4–12.5 tok/s on one GPU to about 16.5 tok/s on two. The gain is modest for a second GPU. The MIT-licensed model remains a batch dataset-generation candidate.

The main points are:

• decode was 12.4-12.5 tok/s on single GPU and 16.5-16.6 tok/s on dual GPU
• a head / tail placement did not scale well, so this run used a more balanced expert tensor placement
• llama.cpp has merged MTP support, but MiMo’s mimo2 MTP tensors were not used in this run
• ik_llama.cpp already has MiMo 2.5 support, but it could not load this quant because of the fused QKV layout

Environment

The model is IQ2_S from AesSedai/MiMo-V2.5-Pro-GGUF. The Hugging Face quant table lists it as 297.45 GiB and 2.50 BPW. The base model, XiaomiMiMo/MiMo-V2.5-Pro, is MIT licensed, and the model card describes it as 1.02T total / 42B active with hybrid attention, 3-layer MTP, and up to 1M context.

Server Commands

The single GPU baseline uses only GPU 0 and places part of the expert tensors on CUDA0.

  podman run --rm \
  --device nvidia.com/gpu=0 \
  -p 8000:8000 \
  --shm-size 8g \
  --cap-add=SYS_NICE \
  -v /mnt/data/models/models--AesSedai--MiMo-V2.5-Pro-GGUF:/models:ro,Z \
  ggml-org/llama.cpp:full-cuda13 \
  --server \
  -m /models/snapshots/a8205a14c95fd13018a7776bf0778d1edb6ba2be/IQ2_S/MiMo-V2.5-Pro-IQ2_S-00001-of-00008.gguf \
  --ctx-size 49152 \
  -fa on -fit on \
  -ngl 99 \
  -ctk q8_0 -ctv q8_0 \
  -ot "blk\.([0-3]|1[6-7]|2[6-7]|3[6-7]|4[6-7]|5[5-7]|6[6-9])\.ffn_.*_exps.*=CUDA0,exps=CPU" \
  --no-mmap \
  --parallel 1 \
  --threads 15 \
  --threads-batch 27 \
  --batch-size 12288 \
  --ubatch-size 6144 \
  --jinja \
  --kv-unified \
  --host 0.0.0.0 --port 8000 \
  --alias grandpa
  

The dual GPU run distributes expert tensors across CUDA0 and CUDA1. Non-expert tensors stay on GPU via -ngl 99, preserving Flash Attention.

  podman run --rm \
  --device nvidia.com/gpu=all \
  -p 8000:8000 \
  --shm-size 8g \
  --cap-add=SYS_NICE \
  -v /mnt/data/models/models--AesSedai--MiMo-V2.5-Pro-GGUF:/models:ro,Z \
  ggml-org/llama.cpp:full-cuda13 \
  --server \
  -m /models/snapshots/a8205a14c95fd13018a7776bf0778d1edb6ba2be/IQ2_S/MiMo-V2.5-Pro-IQ2_S-00001-of-00008.gguf \
  --ctx-size 49152 \
  -fa on -fit on \
  -ngl 99 \
  -ctk q8_0 -ctv q8_0 \
  -ot "blk\.([0-3]|1[6-7]|2[6-7]|3[6-7]|4[6-7]|5[5-7]|6[6-9])\.ffn_.*_exps.*=CUDA0,blk\.([4-6]|1[8-9]|2[8-9]|3[8-9]|4[8-9]|5[8-9]|6[2-5])\.ffn_.*_exps.*=CUDA1,exps=CPU" \
  --no-mmap \
  --parallel 1 \
  --threads 15 \
  --threads-batch 27 \
  --batch-size 12288 \
  --ubatch-size 6144 \
  --jinja \
  --kv-unified \
  --host 0.0.0.0 --port 8000 \
  --alias grandpa
  

Expert Tensor Placement

On single GPU, about 18 layers worth of experts sit on CUDA0 and about 52 layers remain on CPU. On dual GPU, about 18 layers sit on CUDA0, 17 on CUDA1, and about 35 remain on CPU.

The 297.45 GiB IQ2_S quant does not fit directly into 192GB of VRAM. So attention / norm / embedding stay on GPU, while only MoE expert tensors are split with the -ot regex. The goal of this run was first to confirm behavior and speed, so I did not do a deeper layer-placement search.

Benchmark Results

The raw timings were:

  Single GPU (1x RTX PRO 6000):
  prompt eval time =   27716.99 ms / 15594 tokens (    1.78 ms per token,   562.62 tokens per second)
  eval time         =   13512.61 ms /   169 tokens (   79.96 ms per token,    12.51 tokens per second)
  prompt eval time =    4185.57 ms /   136 tokens (   30.78 ms per token,    32.49 tokens per second)
  eval time         =  164812.58 ms /  2048 tokens (   80.47 ms per token,    12.43 tokens per second)

Dual GPU (2x RTX PRO 6000):
  prompt eval time =   24150.56 ms / 15594 tokens (    1.55 ms per token,   645.70 tokens per second)
  eval time         =    7892.13 ms /   131 tokens (   60.25 ms per token,    16.60 tokens per second)
  prompt eval time =    3109.48 ms /   160 tokens (   19.43 ms per token,    51.46 tokens per second)
  eval time         =  123901.10 ms /  2048 tokens (   60.50 ms per token,    16.53 tokens per second)
  

Decode increased to 16.5–16.6 tok/s, about +33%. More GPU-resident experts likely reduced CPU computation and transfers.

About 35 expert layers still remain on CPU in the dual setup. CPU computation and GPU synchronization appear to limit decode near 16.5 tok/s.

Prefill improved +14.8% for long input and +58.4% for short input. GPU parallelism may explain the larger short-input gain, but this test does not isolate the cause.

MTP Is Still Unused

MiMo V2.5 Pro includes 3-layer MTP heads. AesSedai’s GGUF includes tensors for blk.70-blk.72, but llama.cpp did not use them in this run.

  W model has unused tensor blk.70.attn_output.weight (size = 106954752 bytes) -- ignoring
W model has unused tensor blk.70.attn_norm.weight (size = 24576 bytes) -- ignoring
W model has unused tensor blk.70.attn_sinks.weight (size = 512 bytes) -- ignoring
W model has unused tensor blk.70.ffn_norm.weight (size = 24576 bytes) -- ignoring
W model has unused tensor blk.70.ffn_gate.weight (size = 106954752 bytes) -- ignoring
W model has unused tensor blk.70.ffn_down.weight (size = 106954752 bytes) -- ignoring
W model has unused tensor blk.70.ffn_up.weight (size = 106954752 bytes) -- ignoring
W model has unused tensor blk.70.nextn.eh_proj.weight (size = 80216064 bytes) -- ignoring
W model has unused tensor blk.70.nextn.enorm.weight (size = 24576 bytes) -- ignoring
W model has unused tensor blk.70.nextn.hnorm.weight (size = 24576 bytes) -- ignoring
W model has unused tensor blk.70.layer_output_norm.weight (size = 24576 bytes) -- ignoring
... (same for blk.71, blk.72)
  

llama.cpp PR #22673 merged MTP support into master on 2026-05-16. The PR examples use --spec-type draft-mtp and --spec-draft-n-max. But for this MiMo GGUF, the mimo2 MTP tensors are still not recognized by the runtime and are discarded at load time.

Xiaomi’s model card reports 3x output speed with MTP. This run did not use MTP, so that gain was not measured. A hypothetical +50% would reach about 24 tok/s, near my current 23–40 tok/s orchestrators. I plan to retest after support works. The video shows the React SPA Spec run’s initial behavior.

ik_llama.cpp Stops on Fused QKV

I also tried the merged MiMo 2.5 support in ik_llama.cpp, expecting better TG, but this GGUF did not load.

The model fails to load because of the fused QKV layout.

  Image: registry.home.arpa/ik_llama:latest (main branch with #1723 "Support Mimo-2.5")

Memory required for model tensors + cache: 314076 MiB
Memory available on all devices - compute: 92720 MiB
llm_load_tensors: ggml ctx size =    0.66 MiB
llama_model_load: error loading model: check_tensor_dims: tensor 'blk.0.attn_q.weight' not found
llama_model_load_from_file: failed to load model
llama_init_from_gpt_params: error: failed to load model
  

As covered in ik_llama.cpp issue #1769, the cause is the fused QKV layout in the GGUF. Xiaomi’s released MiMo V2.5 Pro safetensors use fused QKV, and ggml-org-style conversion keeps that as attn_qkv in GGUF. ik_llama.cpp expects separate attn_q / attn_k / attn_v, so it cannot find blk.0.attn_q.weight. Other quantizations may appear on HF later, but downloading a model this large takes time. Before pulling one, it is worth checking the model card to see whether the QKV layout matches what ik_llama.cpp expects.

If a separate Q/K/V GGUF appears, or if ik_llama.cpp starts un-merging fused QKV at load time, it can be retested. For now, MiMo V2.5 Pro IQ2_S is practically a llama.cpp mainline hybrid inference target.

Code Generation Behavior

Separately from speed, I also tried a React SPA code generation task. The benchmark asks the model to generate a hair salon booking SPA on top of a React SPA Spec scaffold. I stopped before it completed, so I do not have a score.

Even so, the initial tool-use behavior was solid.

• it inspected package.json, vite.config.ts, tsconfig.json, and src/ before starting
• it ran npm install to settle dependencies before implementation
• it created directories before writing files
• it designed reservation status with TypeScript discriminated unions
• it used str_replace for incremental edits instead of clobbering existing files
• it noticed that package.json had changed after npm install and reread it

The run used the terminal, existing files and dependencies successfully. For development assistance in LLM integration, longer completion and output-quality checks remain necessary.

Licensing

MiMo is a MIT-licensed 1T-class MoE aimed at agentic work. My 200B+ candidates at the time of writing were DeepSeek V4, GLM-5.1 and MiMo. Other models such as Kimi-K2.6 may need revenue-based license checks before commercial provision, so terms need to be distinguished by model.

Scope of this run

The run confirmed about 12 tok/s on one GPU and 16.5 tok/s on two. Working MTP and ik_llama.cpp loading remain unresolved.

I use Claude / Codex for debugging and implementation and want to finish a local alternative within the year. Possible shifts toward token-based pricing and tighter usage limits are concerns, not established future changes.

References

• AesSedai/MiMo-V2.5-Pro-GGUF
• XiaomiMiMo/MiMo-V2.5-Pro
• ik_llama.cpp issue #1769: Mimo V2.5 Pro GGUF Fails to Load
• llama.cpp PR #22673: MTP Support
• Xiaomi MiMo-V2.5-Pro official page