Planning a GPU/CPU Division for Local LLM, and the Reality of Daily Trial and Error

Summary

I planned a GPU/CPU division of labor for local LLM on a Blackwell 96GB + EPYC 9175F workstation with 12-channel memory bandwidth.

The plan: Use the CPU lane for a Dagster pipeline — idempotent, asynchronous, running large MoE models for quality output. Use the GPU lane as the user-facing interactive window with smaller, fine-tuned models specialized through daily LoRA adapters generated from Dagster assets.

The reality: I’m using ik_llama.cpp with partial GPU offloading, still experimenting daily with model combinations to avoid memory bandwidth contention between GPU and CPU. The Dagster + LoRA pipeline isn’t built yet.

The Plan: CPU/GPU Division Architecture

CPU Lane — Dagster Idempotent Pipeline

Leverage the 12-channel memory bandwidth of EPYC for large MoE models (100B–400B class) in async mode.

• Dagster manages the pipeline as idempotent assets. Jobs can be re-run safely and produce the same results
• Use cases: repository-wide analysis, documentation generation, test code generation, large-scale refactoring reasoning — work where quality matters more than speed
• No responsiveness requirement. Fire and forget, then collect results

MoE models often don’t fit in a single GPU’s VRAM even when their active parameters are small, because total parameters are enormous. With 12-channel CPU memory bandwidth, they run at practical speeds for async workloads.

GPU Lane — Interactive Window + Daily LoRA Updates

The GPU lane serves as the user-facing interactive endpoint.

• Run a ~30B-class model optimized for responsiveness
• Fine-tune for specialization: my codebase, my design patterns, my languages
• Generate LoRA adapters daily from Dagster assets, applying them to the base model on a daily cycle. The heavy CPU processing feeds back into GPU-side conversational quality

This feedback loop is the core of the plan. Knowledge extracted through slow, thorough CPU-side processing gets distilled into LoRA adapters that make the GPU-side interactive model progressively better. From the user’s perspective, it becomes a local LLM that gets smarter the more you use it.

Planned Pipeline Diagram

  CPU (EPYC 12ch):
  Dagster Pipeline → MoE 100B-400B (ik_llama.cpp / vLLM)
    → asset: code analysis results, test generation, refactoring proposals
    → asset: LoRA training data
    → daily LoRA adapter generation

GPU (Blackwell 96GB):
  Inference server → Dense ~30B (vLLM) + LoRA adapter
    → Zed MCP / Aider / CLI
    → immediate responses
    → daily LoRA update (from CPU-side assets)
  

Reality: Model Combination Trial and Error with ik_llama.cpp

The planned Dagster + LoRA pipeline isn’t built yet. The current state is ik_llama.cpp with partial GPU offloading, experimenting with model combinations that don’t create memory bandwidth contention.

Focus of Experimentation

• Adjusting how much weight goes to GPU to balance interactive responsiveness against CPU-side bandwidth availability
• Avoiding patterns where running multiple models simultaneously causes both to slow down due to bandwidth competition
• Measuring how quantization choices (GGUF IQ, NVFP4, FP8) affect quality and speed per model

Current Tool Stack

vLLM is in the plan but ik_llama.cpp’s partial offloading currently offers more flexibility for trying different combinations.

Model Trial Timeline

Models tested from January through March 2026. Retrieved from cold storage archive.

January 2026

February 2026

March 2026

Trends

Looking back at three months of experimentation:

• January: Baseline model selection. Comparing Dense coding models (IQuest-Coder 40B) against MoE (Scout, Maverick), and evaluating NVFP4 quantization quality
• Early February: Increased evaluation of Chinese-origin models (GLM, Kimi, DeepSeek). Also testing Flash-class lightweight models
• Late February: Qwen3.5-generation MoE models (397B, 122B, 35B) become the focus. More per-model quantization comparisons (downloading GGUF/NVFP4/FP8 variants of the same model)
• March: Shift to Japanese-specialized models (Nemotron-Nano-9B-v2-Japanese) and embedding models (Perplexity)

Throughout, the same model often appears multiple times with different quantizations. This is because quantization level directly affects GPU/CPU division efficiency during ik_llama.cpp partial offloading.

Next Steps

The planned Dagster pipeline + daily LoRA updates are not yet implemented. Based on what I’ve learned from the current experimentation:

• Stabilize a model combination for ik_llama.cpp partial offloading
• Build the Dagster idempotent pipeline and dedicate the CPU lane to async processing
• Begin daily LoRA adapter generation from Dagster assets for the GPU-side base model
• Integrate ctree + pathfinder MCP tools as Dagster asset inputs