SGLang FP8 with dynamic LoRA ran at 20-30 tok/s. I considered merged or fully tuned models for EAGLE, but weight duplication across instances costs VRAM. I tested sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP in vLLM as an alternative, with the agent layer controlling MTP and LoRA switching.

Video link: https://www.youtube.com/watch?v=HP1Bl-h45bE

Setup

I ran sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP on vLLM v0.19.2rc1 (V1 engine). GPUs are RTX PRO 6000 Max-Q Blackwell 96GB x 2, TP=2 (TP=1 was also tried but is omitted here). MTP is enabled with num_speculative_tokens=3, KV cache is FP8, and Prefix Caching and Chunked Prefill are both ON.

  podman run --rm -it \
  --name naughty \
  --pull=always \
  --publish 8001:8001 \
  --device nvidia.com/gpu=all \
  --shm-size=8g \
  -v /mnt/data/models/lora-adapters:/loras:ro \
  -v /mnt/data/models:/hf/hub:ro \
  -e HF_HOME=/hf \
  -e HF_HUB_CACHE=/hf/hub \
  -e HF_HUB_OFFLINE=1 \
  -e TRANSFORMERS_OFFLINE=1 \
  -e TORCH_CUDA_ARCH_LIST=12.0 \
  -e VLLM_TARGET_DEVICE=cuda \
  -e VLLM_USE_V1=1 \
  -e VLLM_FLASHINFER_MOE_BACKEND=throughput \
  -e VLLM_USE_FLASHINFER_MOE_FP4=1 \
  registry.home.arpa/vllm-openai:cu130-nightly-fe9c3d6 \
  sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP \
    --host 0.0.0.0 \
    --port 8001 \
    --served-model-name Qwen3.6-27b-NVFP4 \
    --trust-remote-code \
    --language-model-only \
    --quantization modelopt \
    --enable-prefix-caching \
    --dtype auto \
    --tensor-parallel-size 2 \
    --gpu-memory-utilization 0.88 \
    --max-model-len 262144 \
    --max-num-seqs 8 \
    --kv-cache-dtype fp8 \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --default-chat-template-kwargs '{"enable_thinking": true}' \
    --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}' \
    --override-generation-config '{"temperature":1.0,"top_p":0.95,"top_k":20,"presence_penalty":0.0,"repetition_penalty":1.0}'

Initialization took roughly 102 seconds total: model load 1.45s (target) + 0.21s (drafter), torch.compile 45.95s + 6.97s, CUDA Graph Capture 2s. FB Used was 87.6 GB/GPU and Free was 9.69 GB/GPU. KV cache reserved 1,115,200 tokens.

LoRA Adapter (axolotl, bf16)

The LoRA used here was trained in bf16 with axolotl. The adapter is applied at runtime against the NVFP4-quantized base.

  base_model: /mnt/data/models/models--Qwen--Qwen3.6-27B/snapshots/5d316fa25c3a0b6251198e9e7a94e863a435536a
tokenizer_type: AutoTokenizer
trust_remote_code: true

datasets:
  - path: datasets/familiar-sft-v2/train.jsonl
    type: chat_template
    ds_type: json
    field_messages: messages
    roles_to_train:
      - assistant
test_datasets:
  - path: datasets/familiar-sft-v2/val.jsonl
    type: chat_template
    ds_type: json
    field_messages: messages
    roles_to_train:
      - assistant

chat_template: tokenizer_default
sequence_len: 16384
sample_packing: false
pad_to_sequence_len: false

adapter: lora
lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
lora_target_modules:
  - q_proj
  - k_proj
  - v_proj
  - o_proj
  - gate_proj
  - up_proj
  - down_proj

output_dir: /mnt/data/models/lora-adapters/qwen3.6-27b-familiar-bf16-lora
save_safetensors: true
save_total_limit: 2
save_steps: 50
eval_steps: 50
logging_steps: 5

micro_batch_size: 1
gradient_accumulation_steps: 8
num_epochs: 2
learning_rate: 5.0e-5
lr_scheduler: cosine
warmup_ratio: 0.05
optimizer: adamw_torch

bf16: true
fp16: false
tf32: true
load_in_8bit: false
load_in_4bit: false
gradient_checkpointing: true

I mounted /mnt/data/models/lora-adapters as :ro and applied LoRA after startup. The seq=4 comparison was:

  • with LoRA: 85-95 tok/s
  • without LoRA: 85-105 tok/s

Throughput Measurements

I ran Django-like tasks and aggregated 10-second vLLM log intervals.

Generation Throughput (TG)

  • Mean (all): 155.7 tok/s
  • Mean (active, >50 tok/s): 161.4 tok/s
  • Max: 190.4 tok/s
  • Min: 32.6 tok/s
  • Active samples: 43 / 45

Prompt Processing (PP)

  • Mean (active, >0): 831.2 tok/s
  • Max: 2,773.3 tok/s
  • Min (active): 226.6 tok/s
  • Active samples: 28 / 45

MTP Speculative Decoding

  • Mean Acceptance Length (MAL) avg: 3.64
  • MAL max: 4.00
  • Draft Acceptance Rate avg: 87.9%
  • Draft Acceptance Rate max: 100.0%
  • Draft Acceptance Rate min: 70.3%

Caching

  • Prefix Cache Hit Rate: 68.7% → 91.2% (final)
  • GPU KV Cache Usage max: 2.1%

In agent workloads with repeated tool-call patterns, Prefix Cache had headroom to grow and reached 91% by the end of the session. The KV cache usage staying as low as 2.1% is because max_num_seqs=8 and the access pattern was effectively close to single-stream.

GPU State (DCGM)

Values read from the live Grafana DCGM dashboard during the run.

  • GPU Utilization: 92% / 93%
  • Memory Utilization: 53% / 43%
  • Temperature: 84°C / 85°C
  • Power: 281 W / 299 W
DCGM GPU Monitoring dashboard during vLLM NVFP4+MTP run
GPU state during the run. Utilization 92-93%, Temperature 84-85°C, Power 281-299W. FB Used 87.6 GB/GPU, Free 9.69 GB/GPU.

Agent Real-Task Comparison: vLLM NVFP4 vs SGLang FP8

I ran similar tasks through my own agent system on both vLLM NVFP4+MTP (alias: naughty) and SGLang FP8 (alias: frisky). The numbers below come from the Familiar Tool Calls Grafana dashboard, comparing one session to one session.

Familiar Tool Calls dashboard - vLLM NVFP4+MTP vs SGLang FP8
Left: vLLM MTP-NVFP4 (naughty). Right: SGLang FP8 (frisky). Tool-call distribution and failure rates.
  • vLLM NVFP4: 140 calls / 21 failed (15.0%)
  • SGLang FP8: 221 calls / 38 failed (17.2%)

The failure-rate gap was about 2 percentage points. These are only one session per backend, not a statistical comparison. NVFP4 stalled later and did not finish; FP8 completed. The video also shows NVFP4 tool errors.

Prefix Cache Diverged Across Backends Under My CTXManager

For context composed by CTXManager, vLLM had intervals stuck at 0 prefix-cache hits. The same workload stayed at 0.8-1.0 on SGLang and ik_llama. I suspect vLLM cache behavior or startup options rather than synthesis keys, but the cause is unresolved.

Takeaways

  • SGLang FP8 (about 30GB) with dynamic LoRA and no EAGLE ran at 20-30 tok/s. Separate merged / fully tuned instances are an option, but duplicating weights limits VRAM.
  • vLLM MTP-NVFP4 uses about 15GB + LoRA for weights, making multi-instance LLM integration easier to budget. Each instance also has separate context. Combining it with LMCache remains a proposal.
  • The record reports about 90 tok/s average with LoRA versus 161 tok/s without it, or 3-4x the SGLang FP8 + LoRA range. Quality was not evaluated; tool-error rates were close.
  • At the time, NVFP4 was unsupported in LMCache, so I could not combine them. Tracking issue: https://github.com/LMCache/LMCache/issues/3163