Kimi-K2.5 CPU inference and prompt cache

CPU inference measurements

I measured Kimi-K2.5 CPU inference for asynchronous batches: Dagster jobs, dataset expansion, distillation teacher output and local validation. These uses can tolerate waiting rather than needing immediate chat responses.

The setup used Kimi-K2.5 Q4_K_S, llama.cpp server, AMD EPYC 9175F and 768GB DDR5-6400. CPU-only inference ran, with operation shaped by 12-channel bandwidth and prompt-cache reuse.

Background

Kimi-K2.5 is a 1.03 trillion parameter MoE model by Moonshot AI. It selects 8 out of 384 experts, keeping active parameters at ~32B. Built on the DeepSeek-V2 architecture (MLA: Multi-head Latent Attention), it achieves compressed KV cache for better memory efficiency.

RSS was ~523 GiB for Q4_K_S and ~579 GiB for Q4_K_M. Both fit within 768GB RAM, allowing CPU-only startup.

The intended workload was:

• Dagster pipelines
• asynchronous batch generation
• dataset expansion
• distillation teacher generation
• support or validation for a GPU-based agent stack
• a local fallback LLM

The comparison checks tolerable waiting times and context use for asynchronous work.

Comparison criteria

1. Benchmark Q4_K_S CPU inference speed (baseline performance)
2. Measure thread count vs throughput relationship, identify optimal thread count
3. Measure Q4_K_M Prefill/Decode speed at 32K context
4. Validate Prompt Cache (LCP similarity) effectiveness
5. Determine viability for Dagster pipeline batch operations

Test Environment

Model Specifications

Methodology

Running llama-sweep-bench

  MODEL=/models/snapshots/386fed8b054275941d6a495a9a7010fbf31b560d/Q4_K_S/Kimi-K2.5-Q4_K_S-00001-of-00013.gguf
IMG=compute.home.arpa/ik_llama-cuda:latest
podman run --rm -it \
  --device nvidia.com/gpu=all \
  --shm-size 16g \
  --cap-add=SYS_NICE \
  -v /mnt/data/hf/hub/models--unsloth--Kimi-K2.5-GGUF:/models:ro,Z \
  $IMG \
  /app/llama-sweep-bench \
    --model "$MODEL" \
    --no-mmap --merge-qkv \
    -mla 3 -amb 512 \
    -b 4096 -ub 4096 \
    -ctk f16 -ctv f16 \
    -c 131072 \
    -ngl 999 -ot exps=CPU \
    --threads 13 \
    --threads-batch 26 \
    --warmup-batch \
    -n 128
  

This command performs:

• llama-sweep-bench: Dedicated benchmarking tool
• -ngl 999 -ot exps=CPU: Full GPU offload with Expert weights on CPU
• -c 131072: 131K context support
• -ctk f16 -ctv f16: KV cache in f16 precision
• --threads 13 --threads-batch 26: Thread configuration
• -mla 3 -amb 512: MLA (Multi-head Latent Attention) parameters

Thread-Scaling Launch Commands

The benchmark varied thread counts while keeping other parameters fixed. Below are representative CPU-only launch commands.

th=16 (Maximum Performance)

  podman run --rm -p 8081:8080 --shm-size 16g --cap-add=SYS_NICE \
  -v /mnt/data/hf/hub/models--unsloth--Kimi-K2.5-GGUF:/models:Z \
  compute.home.arpa/llamacpp-zen5:latest \
  -m /models/snapshots/386fed8b054275941d6a495a9a7010fbf31b560d/Q4_K_S/Kimi-K2.5-Q4_K_S-00001-of-00013.gguf \
  --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on \
  --ctx-size 8192 --parallel 1 --threads 16 --threads-batch 16 \
  --batch-size 2048 --ubatch-size 512 --jinja --host 0.0.0.0 --port 8080
  

th=13 (coexisting workloads)

  podman run --rm -p 8081:8080 --shm-size 16g --cap-add=SYS_NICE \
  -v /mnt/data/hf/hub/models--unsloth--Kimi-K2.5-GGUF:/models:Z \
  compute.home.arpa/llamacpp-zen5:latest \
  -m /models/snapshots/386fed8b054275941d6a495a9a7010fbf31b560d/Q4_K_S/Kimi-K2.5-Q4_K_S-00001-of-00013.gguf \
  --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on \
  --ctx-size 8192 --parallel 1 --threads 13 --threads-batch 13 \
  --batch-size 2048 --ubatch-size 512 --jinja --host 0.0.0.0 --port 8080
  

Memory Layout (Q4_K_S)

Memory Layout (Q4_K_M / ctx=32K)

A 768GB RAM machine can hold a 16k context workload comfortably. At this scale, the dominant cost is the model itself, not the KV cache.

CPU Inference Live Demonstration

To verify CPU inference output quality, I recorded real-time execution on the EPYC 9175F.

Video link: https://www.youtube.com/watch?v=n8htU2pmzNI

The video demonstrates:

• llama.cpp server startup and model loading (quantized weights)
• Prefill phase token generation speed and content
• Token-by-token Generate phase output
• Verification of actual generated text quality

Results

Q4_K_S Baseline (th=14, ctx=16K)

The first request is slow for chat. A generation taking about eighty seconds can still be useful for overnight or background batches.

The third request reused its prefix through LCP similarity, reducing cache lookup to 62ms. Saved prompt state lowers waiting time for repeated jobs.

Thread Optimization (ctx=8K)

Decode stopped improving around th=13–14, suggesting memory bandwidth was a limit.

The rationale for th=13: the remaining 3 cores are freed for Dagster/Trino and other data pipeline processes. 90% inference speed retained while enabling process coexistence.

Q4_K_M Long Context (th=13, ctx=32K)

16K bulk prefill taking 44 minutes suggests that “fill 256K from scratch every time” is unrealistic. At 20 tok/s, 256K would take ~3.5 hours.

Later requests prefilled only 356, 12 and 1,050 token differences. A fixed digest preserved the reusable prefix.

The prefill slowdown was also pronounced. Throughput started at 10 tok/s, dropped to 7 tok/s midway, and bottomed out near 4 tok/s. Decode landed at 2.44 tok/s, meaning 1k output takes about 6-7 minutes and 2k takes 13-14 minutes. Too slow for interactive chat, but workable for asynchronous batch tasks.

Prompt Cache Effectiveness (Q4_K_S)

Long context logs show repeated LCP similarity reuse:

• selected slot by LCP similarity, sim_best = 0.978 (> 0.100 thold), f_keep = 0.980
• selected slot by LCP similarity, sim_best = 0.999 (> 0.100 thold), f_keep = 0.870
• selected slot by LCP similarity, sim_best = 0.939 (> 0.100 thold), f_keep = 0.940

Later requests are not rebuilding the entire fixed prefix. The server reprocesses only the delta, which is exactly why a stable digest design matters.

Addendum: Improvements with ik_llama.cpp (Expert CPU + Attention GPU Hybrid)

Using the optimized ik_llama.cpp build with Expert weights on CPU and Attention layers on GPU (-ngl 999 -ot exps=CPU):

Execution Command

  podman run --rm -it --device nvidia.com/gpu=all \
  -p 8081:8080 \
  --shm-size 32g \
  --cap-add=SYS_NICE \
  -v /mnt/data/hf/hub/models--unsloth--Kimi-K2.5-GGUF:/models:ro,Z \
  $IMG \
  --host 0.0.0.0 --port 8080 \
  -m "$MODEL" --no-mmap --jinja \
  -c 131072 \
  -n 128 \
  --threads 13 --threads-batch 26 \
  -b 2048 -ub 512 \
  -ngl 999 -ot exps=CPU \
  -ctk f16 -ctv f16 \
  --merge-qkv -mla 3 -amb 512
  

Benchmark Results (Initial)

Subsequent Measurements

After applying Prompt Cache (LCP) and template optimizations:

Assessment

Observed values:

• Runs 1 and 3 reached 100-130 t/s prefill.
• S_TG stayed around 19 t/s across all runs under this Blackwell + Q4_K_S configuration.
• Partial cache hits in Runs 2-3 gave S_PP of 50-110 t/s.

Current constraints:

• Generation bottleneck: S_TG fixed at ~19 t/s. Perceived latency dominated by prefill time and output token count
• Cache consistency: Runs 2-3 show “Common part does not match fully” warnings. Even minor changes to System Prompt or templates (whitespace, timestamps) fragment the cache

Analysis

Memory Bandwidth and Why th=13

Decode speed saturates at th=13-14. The 12-channel DDR5-6400 theoretical bandwidth is ~614 GB/s, but MoE random access patterns cannot fully utilize it. th=16 gives 12.94 tok/s, th=13 gives 11.67 tok/s – a 3-thread reduction for only 10% speed loss.

Proposed long-input configuration

Long inputs favor avoiding repeated full prefill.

The practical solution:

• Limit ctx to 16K-32K
• Prefill a System Digest (8K-16K) once at startup
• Use Prompt Cache (LCP similarity) for diff-only processing on subsequent requests
• Keep output length at 1K default, 2K only when necessary

Two quantizations and context lengths

Q4_K_S / ctx=16K delivered 10 tok/s; Q4_K_M / ctx=32K delivered 2.4 tok/s. Attention over the 4.1GB KV cache may contribute. Both context and quantization changed, so this does not isolate a single cause.

L3 cache and CPU hypotheses

L3 reuse may contribute to CPU performance; the full model cannot fit in cache. Candidate high-reuse areas include:

• Router / Gating logic
• Projection-adjacent compute
• Recent layer weights and intermediate tensors
• KV reuse paths

The EPYC 9175F characteristics may also contribute:

1. L3 (512MB) and 16 cores: the large cache per core may reduce cross-core contention.
2. 12 channels and 16 cores: memory-controller queues may stay short.
3. Zen 5 BF16 / AVX-512: the 512-bit datapath and native BF16 may suit llama.cpp optimizations.

Input design for Prompt Cache

• System Digest must be byte-identical across requests (whitespace/date changes break cache)
• RAG context goes in user messages, not system (cache preservation is priority)
• Keep output style short and controlled (since generation is slow, shorter output directly improves usability)

The aichat .file /path/to/file approach is also effective: keep a fixed digest in the system prompt and inject only necessary documents on the user side for better cache stability.

Conclusion

The run confirmed CPU-only inference of this 1T model. Decode at 10 tok/s is a candidate for the intended batch generation, dataset expansion and teacher output.

The LLM integration plan puts interactive work on GPU (RTX PRO 6000 / vLLM) and batch work on CPU llama.cpp at th=13. Model use of 523 GiB from 768GB leaves an estimated 200GB+ for DataFrame or Trino work. Concurrent load still needs verification.

The measured operating points are th=13 and LCP similarity cache reuse. They inform resource allocation between inference and data services.

Recommended Parameters (Batch Operations)

Recommended Launch Command

  podman run --rm -p 8081:8080 --shm-size 16g --cap-add=SYS_NICE \
  -v /mnt/data/hf/hub/models--unsloth--Kimi-K2.5-GGUF:/models:Z \
  compute.home.arpa/llamacpp-zen5:latest \
  -m /models/snapshots/386fed8b054275941d6a495a9a7010fbf31b560d/Q4_K_S/Kimi-K2.5-Q4_K_S-00001-of-00013.gguf \
  --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on \
  --ctx-size 131072 --parallel 1 --threads 13 --threads-batch 13 \
  --batch-size 2048 --ubatch-size 512 --jinja --host 0.0.0.0 --port 8080
  

Reproduction Steps

1. Download Models

  # Q4_K_S
huggingface-cli download unsloth/Kimi-K2.5-GGUF \
  --include "Q4_K_S/*" \
  --local-dir /mnt/data/hf/hub/models--Kimi-K2.5-GGUF

# Q4_K_M
huggingface-cli download unsloth/Kimi-K2.5-GGUF \
  --include "Q4_K_M/*" \
  --local-dir /mnt/data/hf/hub/models--Kimi-K2.5-GGUF
  

2. Run

See commands in “Methodology” section. Requires llama.cpp with flash-attn and prompt cache support.

3. Measure

  curl -s http://localhost:8081/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"kimi","messages":[{"role":"user","content":"Explain MoE architecture"}],"max_tokens":512}'
  

Extract prompt eval time and eval time from server logs.

Technical Notes

Q4_K_S vs Q4_K_M

Q4_K_M is about 60GB larger, from 520 to 579 GiB. It remains a quality-oriented candidate; Q4_K_S is the proposed choice for these ctx=16K batches.

Reference Benchmark: Llama-4-Maverick-17B-128E

For calibration, I also kept one comparison table for another MoE model:

The table provides reference values for quantization and architecture effects on CPU inference.