Llama 4 Maverick Q4 and Q8 CPU comparison
A Q4_K_M versus Q8_0 test on EPYC 9175F with 768GB memory. For LLM integration choices, it records throughput, TTFT, memory behavior, and qualitative batch/aider quality differences.
Q4_K_M and Q8_0 on CPU
Meta Llama-4-Maverick is a 128-expert MoE with around 17B active parameters per token.
On EPYC 9175F with 768GB DDR5, I compared Q4_K_M at roughly 290GB with Q8_0 at 426GB on the same tasks. The LLM integration choice balances batch throughput with coding quality.
I measured Llama-4-Maverick-17B-128E-Instruct through llama.cpp, using CPU and RAM rather than fitting it into GPU VRAM.
EPYC Zen 5 / Turin provides 12-channel DDR5 and large L3 for sparse-model CPU inference, including trillion-scale model proposals.
Objective
- Compare Q4_K_M and Q8_0 Prefill/Decode speeds on identical hardware
- Quantify TTFT (Time To First Token) difference
- Evaluate output quality difference on practical tasks
- Establish selection criteria for batch processing vs aider workflows
Test Environment
| Item | Specification |
|---|---|
| CPU | AMD EPYC 9175F (Zen 5, 16C, L3 512MB) |
| Memory | DDR5-6400 768GB (12ch) |
| GPU | Not used (CPU inference) |
| OS | Ubuntu 24.04 LTS |
| Runtime | llama.cpp server (Podman rootless) |
| Threads | 14 |
| Context | 16,384 |
Fourteen threads on 16 cores leave OS and network IO headroom while considering bandwidth saturation.
Model Specifications
| Item | Q4_K_M | Q8_0 |
|---|---|---|
| Architecture | Llama-4-Maverick (MoE, 128E) | Same |
| Model Size | ~290GB | ~426GB |
| Quantization | 4-bit mixed | 8-bit |
Results
Speed Comparison
| Metric | Q4_K_M | Q8_0 | Delta |
|---|---|---|---|
| PP(tok/s) | 65-68 | 50-52 | Q4 ~30% faster |
| TG(tok/s) | 21-24 | 15-16 | Q4 ~40% faster |
| TTFT (800-1000tok) | 12-17s | 16-20s | Q4 3-5s faster |
Q4_K_M Measurements
| Prompt(tok) | PP time(s) | PP(tok/s) | Gen(tok) | TG(tok/s) |
|---|---|---|---|---|
| 819 | 12.0 | 68.4 | 165 | 23.7 |
| 1,101 | 16.8 | 65.4 | 814 | 21.3 |
Q8_0 Measurements
| Prompt(tok) | PP time(s) | PP(tok/s) | Gen(tok) | TG(tok/s) |
|---|---|---|---|---|
| 819 | 15.6 | 52.4 | 104 | 16.6 |
| 1,000 | 19.7 | 50.8 | 916 | 15.2 |
Memory Behavior
- Q4_K_M: mmap + page cache loads only needed portions. Prompt cache ~180-380 MiB per entry
- Q8_0: Hundreds of GB in buff/cache. RSS appears small (normal mmap behavior)
- MoE (128E): only active experts are frequently accessed; unused pages stay cold
mmap and page cache load weights. Hundreds of GB can appear in free buff/cache while RSS remains small; this is normal for llama.cpp.
128E accesses active experts repeatedly, while unused pages stay cold.
Analysis
Speed Gap Explained
Q4_K_M is roughly 68% of Q8_0 size (290GB versus 426GB). Smaller weights reduce per-token memory reads.
TG differs by 40% versus a 32% size gap. The interpretation is that random decode access amplifies cache misses and memory latency.
Quality Difference in Practice
Q8_0 felt better at:
- Retaining long conversation context
- Avoiding destructive aider edits
- Producing output I could rely on
The difference was smaller for scaffolding and summaries. Q4_K_M rarely made critical mistakes in aider code or document scaffolds here. Complex repository work and long output showed more difference. This was a qualitative judgment, not a scored quality test.
Why Q8_0 Is Viable on 768GB RAM
With 768GB, Q8_0 at 426GB plus KV and page cache can fit. This choice depends on the available capacity.
Selection Criteria by Use Case
My use-case choices:
- High-throughput oneshot generation, Dagster/dbt batch: Q4_K_M. 40% speed difference compounds over volume
- Aider workflows, complex repo operations, long-form: Q8_0. Reduced destructive edit risk outweighs speed loss
- Daily on/off operation: Q4_K_M. Faster startup
- Always-on resident: Q8_0. Quality stability pays off over long sessions
- Local knowledge processing: Both work for structuring/distilling sensitive internal data locally
Q8_0 reached 15-16 tok/s in this 128E CPU setup.
I favor Q8_0 for quality-sensitive work and Q4_K_M for high-turnover tasks.
Reproduction Steps
Q4_K_M
podman run --rm -it \
-p 8081:8080 --shm-size 16g --cap-add=SYS_NICE \
-v "$MO":/models:ro,Z $IMG \
--host 0.0.0.0 --port 8080 \
-m /models/Llama-4-Maverick-17B-128E-Instruct-Q4_K_M.gguf \
--jinja -c 16384 \
--threads 14 --threads-batch 14 \
-b 1024 -ub 256 \
--parallel 1 --flash-attn on
Q8_0
Use the same command with the Q8_0 file.
Technical Notes
MoE mmap Behavior
Repeated topics reuse experts and warm mmap page cache. Frequent topic changes tend to increase page faults.
Prompt Cache
Both support LCP similarity prompt caching. Fixed System Prompts shorten TTFT on later requests.