I ran DeepSeek V4 Flash 0731 on two RTX PRO 6000 Blackwell Max-Q 96GB GPUs. The runtime stack was custom vLLM Gilded Gnosis r24 with Tensor Parallel 2, DSpark K5, FP8 compressed MLA KV, and native CPU KV offload.

The following video covers the coding-agent workload from execution through inspection of the generated artifacts.

Video link: https://www.youtube.com/watch?v=E7JroDgE9no

The server started with MAX_MODEL_LEN=262144 and exposed 285,336 GPU KV cache tokens. A single 262,144-token request therefore has a theoretical concurrency of 1.09x. On the coding-agent workload, generation throughput peaked at 359.0 tok/s, while the median for a single active request was 230.9 tok/s.

Results

ItemMeasured value
Modeldeepseek-ai/DeepSeek-V4-Flash-0731
RuntimevLLM Gilded Gnosis r24
GPURTX PRO 6000 Blackwell Max-Q 96GB x2
Tensor Parallel / DCP2 / 1
Expert pathmodel-native FP4 / B12X W4A8
KV cacheFP8 DeepSeek compressed MLA
Configured context262,144
Maximum sequences2
GPU KV capacity285,336 tokens
262K request concurrency1.09x
Peak GPU KV usage79.4%
Peak generation throughput359.0 tok/s
Single-active-request median230.9 tok/s
Single-active-request IQR167.9-279.3 tok/s
Nonzero-window mean205.6 tok/s
Peak prompt throughput5,790.1 tok/s
DSpark acceptance lengthmedian 5.12 / best 5.90
DSpark draft acceptancemedian 82.4% / best 98.1%
CPU KV offload8 GiB total across TP ranks
Observed KV copy rateapproximately 50-55 GB/s
Captured access-log results101 / 101 HTTP 200

I calculated the throughput and DSpark statistics from vLLM server logs with Codex. They were not estimated from Grafana telemetry.

How This Differs from Earlier DeepSeek V4 Flash Articles

Previously, I ran a Q2 GGUF build on a single GPU with the dedicated DwarfStar 4 runtime and also distributed it across two compute nodes as an orchestrator.

This time I used the newly released DeepSeek-V4-Flash-0731:

  • The runtime is custom vLLM rather than DwarfStar 4
  • The checkpoint is DeepSeek-V4-Flash-0731
  • The expert weights are model-native FP4
  • The model runs on two RTX PRO 6000 GPUs in one host rather than on a single GPU
  • DSpark K5 speculative decoding is enabled
  • The GPU KV cache uses FP8 compressed MLA
  • Native CPU KV offload uses /dev/shm

Validation Environment

ItemConfiguration
Host CPUAMD EPYC 9175F
Host RAM768GB DDR5 ECC
GPUNVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition x2
VRAM96GB x2
GPU linkPCIe 5.0 x16
OSUbuntu 24.04
ContainerPodman / Quadlet
Modeldeepseek-ai/DeepSeek-V4-Flash-0731
Model snapshot7872f01b1d1fe23eabc4c98b48bffcef5a386062
Image usedregistry.home.arpa/vllm:deepseek-v4
Base releaseGilded Gnosis v20 r24
Runtime versionv0.11.2.dev280 custom build

The vLLM version that actually started was:

  Aug 04 01:45:09 compute-server grandpa[415455]: (APIServer pid=52) INFO 08-04 01:45:09 [api_utils.py:345]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.11.2.dev280+gilded.gnosis.v20.vllmf5981f1.si2b9bf2a.fi801d57a.cu132.20260803.r24
Aug 04 01:45:09 compute-server grandpa[415455]: (APIServer pid=52) INFO 08-04 01:45:09 [api_utils.py:345]   █▄█▀ █     █     █     █  model   /models/snapshots/7872f01b1d1fe23eabc4c98b48bffcef5a386062

The release image behind the private mirror was this build:

  voipmonitor/vllm:gilded-gnosis-v20-vllmf5981f1-si2b9bf2a-fi801d57a-cu132-20260803-r24
Docker manifest: sha256:64b94299abdd3bcf5bb5050ca91b378f9ee4e0b0eff4748375b95352371d7cb2
Local image ID: sha256:dc0bc459b8c1d59f84e945a4b77f65ea474778b58f4a95d3e3d1c97632daeb1d

Model-Native FP4 Experts and B12X W4A8

The expert weights are model-native FP4; this is not a generic post-training Q4 conversion. The startup log also confirms FP4 experts, W4A8, compressed MLA KV, and the FP8 Lightning Indexer.

  (Worker_TP0 pid=496) INFO [quant_config.py:75] DeepSeek V4 expert_dtype resolved to 'fp4'
(Worker_TP0 pid=496) INFO [attention.py:98] Using DeepSeek's fp8_ds_mla KV cache format.
(Worker_TP0 pid=496) INFO [mxfp4.py:426] Using 'B12X' Mxfp4 MoE backend.
(Worker_TP0 pid=496) INFO [attention.py:1023] Using FP8 indexer cache for Lightning Indexer.
(Worker_TP0 pid=496) WARNING [b12x_moe.py:810] B12X MoE force-A8 enabled: using quant_mode=w4a8_mx for E8M0 FP4 weights.

Model loading used the InstantTensor BUFFERED backend. Model weights occupied 81.01 GiB on each GPU.

DSpark K5

K5 is a fixed-depth profile in which DSpark proposes five draft tokens in each verification step.

  Environment=MODE=dspark
Environment=DSPARK_DEPTH_MODE=fixed
Environment=DSPARK_TOKENS=5

With this configuration, the effective scheduled-token budget was 8,184 rather than MAX_NUM_BATCHED_TOKENS=8192, after subtracting the slots needed for speculative decoding.

  Aug 04 02:44:37 compute-server grandpa[572062]: (EngineCore pid=179) WARNING 08-04 02:44:37 [vllm.py:1758] max_num_scheduled_tokens is set to 8184 based on the speculative decoding settings. This may lead to suboptimal performance. Consider increasing max_num_batched_tokens to accommodate the additional draft token slots, or decrease num_speculative_tokens or max_num_seqs.

In this run, acceptance length reached 5.90 while Current speculative depth: 5. The metric includes the normal target token in addition to the five draft tokens. Draft acceptance remained prompt-dependent and was not always high.

Quadlet

This image does not take a list of arguments through the standard vllm-openai entrypoint. The included serve-ds4-flash.sh assembles the DeepSeek V4-specific vLLM arguments, SparkInfer backend, and DSpark settings from environment variables.

The Quadlet Exec therefore consists of one line:

  Exec=/usr/local/bin/serve-ds4-flash.sh

This is the Quadlet configuration I used:

  [Unit]
Description=DeepSeek V4 Flash 0731 / DSpark r24
After=network-online.target
Wants=network-online.target

[Container]
ContainerName=grandpa
Image=registry.home.arpa/vllm:deepseek-v4
Pull=always

Network=host
AddDevice=nvidia.com/gpu=0
AddDevice=nvidia.com/gpu=1
PodmanArgs=--ipc=host --privileged
RunInit=true

Volume=/mnt/data/hf/hub/models--deepseek-ai--DeepSeek-V4-Flash-0731:/models:ro
Volume=/mnt/data/hf/jit/ds4-v20-r24:/cache
Volume=/mnt/data/hf/tmp/ds4-v20-r24:/container-tmp

Environment=BACKEND=b12x-a8
Environment=CUDA_VISIBLE_DEVICES=0,1
Environment=DCP_SIZE=1
Environment=DSPARK_DEPTH_MODE=fixed
Environment=DSPARK_TOKENS=5
Environment=GPU_MEMORY_UTILIZATION=0.975
Environment=INSTANTTENSOR_BACKEND=BUFFERED
Environment=KV_OFFLOADING_SIZE=8
Environment=LOAD_FORMAT=instanttensor
Environment=MAX_MODEL_LEN=262144
Environment=MAX_NUM_BATCHED_TOKENS=8192
Environment=MAX_NUM_SEQS=2
Environment=MODE=dspark
Environment=MODEL_PATH=/models/snapshots/7872f01b1d1fe23eabc4c98b48bffcef5a386062
Environment=PORT=8000
Environment=SERVED_MODEL_NAME=grandpa
Environment=TP_SIZE=2

Ulimit=memlock=-1:-1
Ulimit=nofile=1048576:1048576
Ulimit=stack=67108864:67108864

Exec=/usr/local/bin/serve-ds4-flash.sh

[Service]
ExecStartPre=/usr/bin/mkdir -p /mnt/data/hf/jit/ds4-v20-r24 /mnt/data/hf/tmp/ds4-v20-r24
TimeoutStartSec=3600
TimeoutStopSec=120
Restart=on-failure
RestartSec=30
LimitMEMLOCK=infinity
LimitNOFILE=1048576
LimitSTACK=67108864

Because this configuration uses --ipc=host, it cannot also specify ShmSize=. Podman rejects host IPC and a container-specific shared-memory size when both are configured, so one of them must be selected.

The --privileged setting is also not a least-privilege configuration. I kept it here because this validation prioritized getting the runtime started with the settings I actually used.

GPU Memory and KV Capacity

After graph capture, the per-GPU breakdown was:

ItemPer GPU
Model weights81.01 GiB
Peak activation2.45 GiB
Non-torch memory0.20 GiB
CUDA graph0.14 GiB
GPU KV cache8.59 GiB

The raw log records both the memory breakdown and KV capacity in the same startup sequence.

  Aug 04 02:44:21 compute-server grandpa[572062]: (Worker_TP0 pid=264) INFO 08-04 02:44:21 [gpu_worker.py:617] Available KV cache memory: 8.59 GiB
Aug 04 02:44:21 compute-server grandpa[572062]: (EngineCore pid=179) INFO 08-04 02:44:21 [kv_cache_utils.py:2428] GPU KV cache size: 285,336 tokens
Aug 04 02:44:21 compute-server grandpa[572062]: (EngineCore pid=179) INFO 08-04 02:44:21 [kv_cache_utils.py:2429] Maximum concurrency for 262,144 tokens per request: 1.09x
Aug 04 02:44:27 compute-server grandpa[572062]: (Worker_TP0 pid=264) INFO 08-04 02:44:27 [gpu_worker.py:918] Free memory on device (93.73/94.97 GiB) on startup. Desired GPU memory utilization is (0.975, 92.59 GiB). Actual usage is 81.01 GiB for weight, 2.45 GiB for peak activation, 0.2 GiB for non-torch memory, and 0.14 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory-bytes=9288604570` (8.65 GiB) to fit into requested memory, or `--kv-cache-memory-bytes=10511114752` (9.79 GiB) to fully utilize gpu memory. Current kv cache memory in use is 8.59 GiB.
Aug 04 02:44:36 compute-server grandpa[572062]: (EngineCore pid=179) INFO 08-04 02:44:36 [core.py:345] init engine (profile, create kv cache, warmup model) took 47.53 s (compilation: 12.64 s)

Allocating the Active Sequence Budget

Treating the current GPU KV capacity conservatively as an active working set of approximately 256K gives the following operational model:

Active sequencesApproximate total tokens per sequence
1256K
2128K
464K

Here, total tokens means the current prompt plus generated output, not input alone. A sequence with a 64K input and another 32K of generated output consumes 96K.

For more flexible operation, I could keep MAX_MODEL_LEN=262144, raise MAX_NUM_SEQS to 4, and have the agent loop or orchestrator limit the aggregate active-sequence token budget.

  sum(active prompt tokens + generated tokens + output headroom) <= approximately 256K

Native CPU KV Offload

KV_OFFLOADING_SIZE=8 creates one 8 GiB host-RAM pool shared by TP0 and TP1.

  Aug 04 02:44:22 compute-server grandpa[572062]: (Worker_TP0 pid=264) INFO 08-04 02:44:22 [shared_offload_region.py:85] Created mmap file /dev/shm/vllm_offload_f5388900-1119-46b2-b2eb-178e4b7e21c5.mmap (8.59 GB)
Aug 04 02:44:22 compute-server grandpa[572062]: (Worker_TP1 pid=265) INFO 08-04 02:44:22 [shared_offload_region.py:93] Opened existing mmap file /dev/shm/vllm_offload_f5388900-1119-46b2-b2eb-178e4b7e21c5.mmap
Aug 04 02:44:22 compute-server grandpa[572062]: (Worker_TP0 pid=264) INFO 08-04 02:44:22 [shared_offload_region.py:176] Unlinked mmap file /dev/shm/vllm_offload_f5388900-1119-46b2-b2eb-178e4b7e21c5.mmap after 2 workers mapped it

Although the log calls it an mmap file, it is not writing to SSD. /dev/shm is normally tmpfs backed by host DRAM. Transfers between that memory and the GPU travel over PCIe 5.0 x16.

Calculating bytes / recorded copy time from the transfer metrics gives a weighted store rate of 54.38 GB/s and a weighted load rate of 51.46 GB/s.

DirectionWeighted rateObserved range
GPU to host RAM (store)54.38 GB/s52.09-55.56 GB/s
Host RAM to GPU (load)51.46 GB/s48.02-54.91 GB/s

The largest store interval transferred 1.415 GB in 27.17 ms. The largest load interval transferred 185.3 MB in 3.37 ms.

  Aug 04 02:49:49 compute-server grandpa[572062]: (APIServer pid=53) INFO 08-04 02:49:49 [metrics.py:103] KV Transfer metrics: vllm:kv_offload_store_bytes=1415335680, vllm:kv_offload_store_time=0.027173248052597042, vllm:kv_offload_store_size_count=18, vllm:kv_offload_store_size_sum=1415335680, vllm:kv_offload_cpu_cache_usage_perc=0.0, vllm:kv_offload_cpu_allocation_size_count=7, vllm:kv_offload_cpu_allocation_size_sum=503, vllm:kv_offload_cpu_cache_write_usage_perc=0.0, vllm:kv_offload_cpu_cache_read_usage_perc=0.0
Aug 04 02:49:59 compute-server grandpa[572062]: (APIServer pid=53) INFO 08-04 02:49:59 [metrics.py:103] KV Transfer metrics: vllm:kv_offload_cpu_cache_usage_perc=0.0, vllm:kv_offload_cpu_cache_write_usage_perc=0.0, vllm:kv_offload_cpu_cache_read_usage_perc=0.0, vllm:kv_offload_lookup_sync_delay_seconds_count=5, vllm:kv_offload_lookup_sync_delay_seconds_sum=9.603999751561787e-05, vllm:kv_offload_load_bytes=185264640, vllm:kv_offload_load_time=0.0033739200830459597, vllm:kv_offload_load_size_count=2, vllm:kv_offload_load_size_sum=185264640, vllm:kv_offload_cpu_allocation_size_count=5, vllm:kv_offload_cpu_allocation_size_sum=139, vllm:kv_offload_store_bytes=299439360, vllm:kv_offload_store_time=0.005468831906095147, vllm:kv_offload_store_size_count=10, vllm:kv_offload_store_size_sum=299439360

The raw one-way bandwidth of PCIe 5.0 x16 is approximately 64 GB/s, so an observed 50-55 GB/s is not an implausible result.

The maximum observed kv_offload_cpu_cache_usage_perc for this workload was approximately 3.86%. This workload did not fill the 8 GiB pool. It confirms that the offload path was active, but it does not evaluate a very large CPU tier.

What 768GB of RAM Adds: KV Storage Capacity

Using this host’s 768GB of RAM for CPU KV offload does not increase the maximum context of one request or the amount of KV that can be processed concurrently on the GPU. What increases is the amount of previously computed prefix KV that can remain on the CPU for reuse by later requests.

In this configuration, 17.18 GiB of total GPU KV across TP2 corresponds to 285,336 logical tokens. Assuming the CPU tier preserves the same KV block layout, a 512 GiB CPU pool has the following storage equivalent:

  285,336 tokens x (512 GiB / 17.18 GiB)
  = approximately 8.5 million token-block equivalents

That is roughly enough storage for 32 histories of 262,144 tokens under the simple conversion. It does not mean that 32 such requests can run concurrently. KV in the CPU tier must return to the GPU over PCIe before it can be reused, and actual capacity also varies with block rounding, metadata, eviction, and shared prefixes.

This native offload path moves inactive prefix KV to the CPU for later reuse; it does not continue an active request entirely from CPU memory. Increasing KV_OFFLOADING_SIZE to 512 GiB alone does not change:

  • MAX_MODEL_LEN
  • The 285,336-token GPU KV cache
  • The GPU-resident active working set
  • The configured MAX_NUM_SEQS

A 512 GiB shared mmap also requires sufficient /dev/shm capacity, actual free RAM, memlock allowance, and successful CUDA host registration.

  df -h /dev/shm

This is an upper-bound conversion from the 8 GiB run. Startup, latency, and effective capacity with 512 GiB remain untested.

Generation Throughput and DSpark Acceptance

I aggregated the vLLM metrics in 10-second windows.

  • Median and Tukey IQR come from 22 windows with Running: 1
  • The nonzero mean comes from 34 windows with nonzero generation throughput
  • Acceptance statistics come from 34 windows containing DSpark metrics
  • CPU KV bandwidth is sum(bytes) / sum(recorded transfer time)
MetricValue
Peak generation throughput359.0 tok/s
Single-active-request median230.9 tok/s
Single-active-request IQR167.9-279.3 tok/s
Mean across nonzero windows205.6 tok/s
Peak prompt throughput5,790.1 tok/s
Acceptance length median5.12
Acceptance length mean4.86
Acceptance length best5.90
Draft acceptance median82.4%
Draft acceptance mean77.2%
Draft acceptance best98.1%
Draft acceptance range41.9-98.1%
Accepted speculative throughput median161.2 tok/s
Accepted speculative throughput best298.2 tok/s

In the window where DSpark was most effective, generation throughput reached 359.0 tok/s with 98.1% draft acceptance.

  Aug 04 02:46:09 compute-server grandpa[572062]: (APIServer pid=53) INFO 08-04 02:46:09 [loggers.py:314] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 359.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 88.1%, External prefix cache hit rate: 0.8%
Aug 04 02:46:09 compute-server grandpa[572062]: (APIServer pid=53) INFO 08-04 02:46:09 [metrics.py:125] SpecDecoding metrics: Mean acceptance length: 5.90, Current speculative depth: 5, Accepted throughput: 298.18 tokens/s, Drafted throughput: 303.98 tokens/s, Accepted: 2982 tokens, Drafted: 3040 tokens, Per-position acceptance rate: 1.000, 0.998, 0.993, 0.972, 0.941, Avg Draft acceptance rate: 98.1%

For some prompts, acceptance fell as low as 41.9%.

  Aug 04 02:47:59 compute-server grandpa[572062]: (APIServer pid=53) INFO 08-04 02:47:59 [loggers.py:314] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 193.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 95.2%, External prefix cache hit rate: 9.3%
Aug 04 02:47:59 compute-server grandpa[572062]: (APIServer pid=53) INFO 08-04 02:47:59 [metrics.py:125] SpecDecoding metrics: Mean acceptance length: 3.09, Current speculative depth: 5, Accepted throughput: 131.19 tokens/s, Drafted throughput: 313.47 tokens/s, Accepted: 1312 tokens, Drafted: 3135 tokens, Per-position acceptance rate: 0.789, 0.542, 0.362, 0.247, 0.152, Avg Draft acceptance rate: 41.9%

The 359 tok/s peak is real, but practical performance in this runtime moves significantly with acceptance. For this workload, the approximately 231 tok/s median for one active request is the better central value.

The 5,790.1 tok/s prompt-throughput value is a window peak from a mixed long-context workload, not an independent fixed-length prefill benchmark.

  Aug 04 02:49:39 compute-server grandpa[572062]: (APIServer pid=53) INFO 08-04 02:49:39 [loggers.py:314] Engine 000: Avg prompt throughput: 2649.6 tokens/s, Avg generation throughput: 107.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.4%, Prefix cache hit rate: 95.9%, External prefix cache hit rate: 3.7%
Aug 04 02:49:49 compute-server grandpa[572062]: (APIServer pid=53) INFO 08-04 02:49:49 [loggers.py:314] Engine 000: Avg prompt throughput: 5790.1 tokens/s, Avg generation throughput: 115.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.2%, Prefix cache hit rate: 95.9%, External prefix cache hit rate: 3.7%

Peak GPU KV usage was 79.4%. A simple conversion of 285,336 slots puts that at approximately 226K token slots. Because this runtime gauge includes multiple requests, shared prefixes, and block allocation, it does not represent the length of one prompt.

Prefix Cache

The prefix-cache hit rate reached 97.4%, while the external prefix-cache hit rate reached 11.4%.

  Aug 04 02:49:29 compute-server grandpa[572062]: (APIServer pid=53) INFO 08-04 02:49:29 [loggers.py:314] Engine 000: Avg prompt throughput: 293.2 tokens/s, Avg generation throughput: 136.6 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 97.4%, External prefix cache hit rate: 6.7%
Aug 04 02:50:39 compute-server grandpa[572062]: (APIServer pid=53) INFO 08-04 02:50:39 [loggers.py:314] Engine 000: Avg prompt throughput: 250.6 tokens/s, Avg generation throughput: 166.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 95.5%, External prefix cache hit rate: 11.4%

External prefix cache and native CPU KV offload are not the same thing. In the Gilded Gnosis r24 runbook, native offload is the qualified host-cache path, while LMCache remains experimental for DS4.

Coding-Agent Workload

The client did not send requests directly to vLLM. It went through Familiar, my agent-loop runtime, which applies tool-call correction and telemetry.

A representative request contained 44 messages, 10 tools, 41,230 prompt tokens, and 800 completion tokens.

  {"time":"2026-08-04T11:49:58.642929+09:00","level":"INFO","msg":"backend POST","url":"http://compute.home.arpa:8000/v1/chat/completions","model":"grandpa","stream":true,"req_stream_arg":true,"payload_bytes":180640,"total_content_chars":136124,"n_messages":44,"tools":10,"max_tokens":32768,"max_completion_tokens":32768,"reasoning_format":null,"reasoning_effort":null,"temperature":0.7}
{"time":"2026-08-04T11:50:02.952783+09:00","level":"INFO","msg":"http request","method":"POST","path":"/v1/chat/completions","status":200,"duration":4317612500,"model":"familiar","message_count":44,"prompt_tokens":41230,"completion_tokens":800}

The captured vLLM access log contained 101 HTTP 200 responses. Over the same range, it contained zero non-200 responses, CUDA OOMs, fatal CUDA errors, or tracebacks.

  Aug 04 02:50:28 compute-server grandpa[572062]: (APIServer pid=53) INFO:     10.10.10.2:52922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
Aug 04 02:50:30 compute-server grandpa[572062]: (APIServer pid=53) INFO:     10.10.10.2:52922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
Aug 04 02:50:34 compute-server grandpa[572062]: (APIServer pid=53) INFO:     10.10.10.2:52922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
Aug 04 02:50:36 compute-server grandpa[572062]: (APIServer pid=53) INFO:     10.10.10.2:52922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
Aug 04 02:50:38 compute-server grandpa[572062]: (APIServer pid=53) INFO:     10.10.10.2:52922 - "POST /v1/chat/completions HTTP/1.1" 200 OK

For this workload, the model generated a Django restaurant reservation system. The artifact contained 25 domain models, nine public service methods, 12 API endpoints, Django Admin, and seed data. All 52 tests passed.

  uv run --frozen --offline python manage.py test --verbosity 2
  
  Ran 52 tests in 0.419s
OK

After loading the seed data, Django Admin showed three exceptional closures for Lakeside Bistro: Private Event, Maintenance, and Holiday.

Django Admin showing seed data in the restaurant reservation system generated by DeepSeek V4 Flash 0731
Django Admin after loading the generated seed data. Three exceptional closures—Private Event, Maintenance, and Holiday—are registered for Lakeside Bistro.

A separate review still found prototype-level issues in timezone-aware availability calculation, waitlist and hold expiry enforcement, and invalid foreign-key handling. The generated artifact is substantially larger than a simple scaffold, but I would not call it production-ready.

Tool calling also needs to be interpreted as a result obtained through Familiar’s correction path, not as a direct vLLM request.

Caveats

256K Capacity and 256K Correctness Are Different

What I confirmed is that the server started with a 262,144-token context setting, allocated 285,336 GPU KV tokens, and completed the mixed long-context workload without runtime instability.

CPU Offload API Is Experimental

The startup log marks the CPUOffloadingSpec API as experimental. The native offload runtime test passed in r24, but the API contract may still change.

  Aug 04 02:44:21 compute-server grandpa[572062]: (Worker_TP0 pid=264) WARNING 08-04 02:44:21 [base.py:493] Initializing OffloadingSpec. This API is experimental and subject to change in the future as we iterate the design.

First-Run JIT

The first run compiles SparkInfer, TileLang, and CUDA graph artifacts. Engine initialization took 47.53 seconds in this run, including 12.64 seconds of compilation. Reusing the JIT cache volume is preferable.

Tool Calls

Client requests pass through Familiar. Familiar applies tool-call correction and telemetry, so direct vLLM tool-calling behavior may differ slightly. That said, the transparent processing used from an editor is primarily tool-call recovery and telemetry delivery to Vector, so I do not expect the behavior to change dramatically.

The throughput and DSpark numbers themselves come from vLLM server-side metrics logs.

Summary

The model-native FP4 expert checkpoint for DeepSeek V4 Flash 0731 reached a practical 262K-context configuration on two RTX PRO 6000 Blackwell 96GB GPUs. Its output quality in terminal use also looked excellent. I had been running StepFun for a long time, but I now plan to try retuning the setup around DS4.

  1. FP8 compressed MLA exposed 285,336 logical tokens from 8.59 GiB of KV memory on each GPU
  2. DSpark K5 delivered a large gain in high-acceptance windows and still reached a single-active-request median of approximately 231 tok/s
  3. Native CPU KV offload operated through one TP2-shared mmap and recorded approximately 50-55 GB/s of KV copy bandwidth over PCIe 5.0 x16

Managing the active GPU KV working set as approximately 256K while using host RAM as a tiered cache for reusable prefix KV fits my custom agent platform well. Admission control in the agent loop—1 x 256K, 2 x 128K, or 4 x 64K—would allow the same runtime to support one long session or several medium-sized sessions.

References