Using DeepSeek-V4-Flash-Vision-Exp and V4.1-Flash with DeepSeek Harness

Introduction

I made a big change to how I build familiar’s agent.

Until now, I had built familiar myself as a monolith that contained the gateway, the agent, and the infrastructure. This time, I removed the agent part entirely. Instead, I put DeepSeek Harness (dsh from here on) inside a container Pod and let it run freely with full access.

I split familiar into two parts, familiar-daemon and agent-gateway. ancestor, which manages the inference backend, already existed, so I kept using it as is. These three run dsh inside a Pod.

I did the port in about a week after Codex Astra was released, which also served as a performance check.

I wrote about how familiar started and its early design in the previous article. This article continues from there.

Videos

I recorded videos of dsh running with full access inside a Pod, with inference served by vLLM on the compute host.

The first one runs deepseek-ai/DeepSeek-V4-Flash-Vision-Exp from the official checkpoint as is and has it implement a Django domain layer. The settings and numbers are in “The Vision-Exp run” later in this article.

Video link: https://www.youtube.com/watch?v=KD-4jm46izk

The second one was recorded separately with the recently released DeepSeek-V4.1-Flash in EXL3 2.0bpw. The model is the quantized model published by diffbot.

Video link: https://www.youtube.com/watch?v=CBkIkCtIxJI

familiar before

At the time of the previous article, familiar was a monolith that contained the gateway, the agent, and the infrastructure. I had packed quite a lot into it.

• Compaction
• Memory
• Summarization
• Per-role memory management
• Operating the knowledge platform
• Custom tools
• MCP. Since it was written in Go, I even embedded entire MCP sources with embed
• Observability into OpenTelemetry, Vector, Prometheus, and Grafana
• What-if and replay built with Grafana and GraphQL
• A Dagster assets pipeline
• A container mechanism for verifying shell execution
• Tool recovery

As I wrote in “I was building tools more than the main body” in the previous article, most of the work was compensating for what local LLMs are bad at with mechanisms around them.

Switching to DeepSeek Harness

What got me started was dsh’s pluggable design. It was really good.

Plugins keep being added. Lately, when I search before implementing something, I have sometimes found what I wanted.

The what-if and replay features I had built myself with Grafana and GraphQL were also covered by plugins. For example, ThoughtDAG lets you see and edit the context passed to the LLM as a graph, and branch it to try a different flow. It is also provided as a plugin for dsh, and in the desktop dsh I use it from the DAG tab next to Chat.

Plugins can be used like libraries. I don’t have to build them myself, and someone else develops and maintains them, which I appreciate.

Three components

I pulled out everything except the agent and ported only the infrastructure part. The roles are split like this.

famd owns agent execution and publishing its own operational events to NATS, and nothing beyond that. It does not own an OpenAI-compatible inference endpoint or the capture of prompts, tool calls, and responses. Those belong to agent-gateway, which publishes captures to NATS asynchronously. Observability such as OpenTelemetry was also implemented by reusing agent-gateway’s events.

Starting and stopping inference servers is ancestor’s job. Submitting a request to famd does not start or stop an inference server on its own.

Current architecture

flowchart LR
  subgraph desktop["desktop.home.arpa"]
    CLI["famdctl"]
    UI["dsh Web UI"]
  end
  subgraph compute["compute.home.arpa"]
    FAMD["familiar-daemon<br/>control / launcher / finalizer"]
    GW["agent-gateway"]
    ANC["ancestor"]
    VLLM["vLLM"]
    RELAY["famd-egress<br/>inference relay"]
    subgraph pod["runtime Pod"]
      DSH["DeepSeek Harness<br/>Full access"]
      WEB["Web relay"]
    end
  end
  NATS[("NATS")]
  CLI --> FAMD
  CLI --> GW
  CLI --> ANC
  FAMD --> pod
  ANC --> VLLM
  DSH --> RELAY --> GW --> VLLM
  GW -. discover .-> ANC
  FAMD -. events .-> NATS
  GW -. capture .-> NATS
  UI -. "SSH tunnel<br/>pod forward" .-> WEB

On compute.home.arpa, famd’s control, launcher, finalizer, and related services run as native services, and so does agent-gateway.

famd-egress is the inference entry point that Pods can see. dsh inside a Pod sends inference requests to http://famd-egress:8080/v1, and famd-egress relays them to agent-gateway. It is a Quadlet container with a read-only root filesystem and all capabilities dropped.

I operate everything from famdctl on the desktop. A request is submitted as a delegation, and once it is accepted, a runtime Pod is created and dsh starts running. The Pod contains two things, the dsh container and a Web relay, and dsh runs freely with full access inside the Pod. Why it has full access, and what to watch out for, is in “About full access” later in this article.

Using famdctl

famdctl has the following command set.

  ksh3@desktop.home.arpa ~ % famdctl
famdctl [--json] [--config path] [--idempotency-key key] <command>

  ancestor    List, inspect, plan, ensure or stop Ancestor services
  delegation  List, submit, inspect, wait for, retrieve or cancel a delegation
  session     List, start, recover, observe or close a DSH session
  events      Stream lifecycle events, resuming with --after
  artifact    Inspect or download a verified artifact
  pod         List, start, stop, remove, drain or forward runtime Pods
  dsh-local   Prepare, sync and open the local DSH Web UI
  workbench   Alias for dsh-local open
  workspace   Initialize or locate the local work records and DSH mirror
  gateway     List public models
  config      Initialize or inspect saved settings
  doctor      Check configuration; --online probes supported service APIs
  completion  Print a bash or zsh completion script

Use famdctl <command> --help for details. Output is formatted for reading.
Add --json anywhere before -- for raw JSON responses and JSONL events.
Exit status: 0 success/waiting, 1 service/local/terminal failure, 2 usage error.
Root submit/run, status, wait, result and cancel are delegation shortcuts.
Save endpoints once with config init, then use alias fam=famdctl.
  

ancestor operates ancestor, gateway operates agent-gateway, and delegation, session, events, artifact, and pod operate famd. Each of the three services has its own endpoint and credentials.

Here is an example of submitting the Django domain implementation task. With --detach, only the acceptance receipt comes back.

  ksh3@desktop.home.arpa ~/familiar-workspace % famdctl delegation submit --workflow django --detach /Users/ksh3/familiar-workspace/task/task.json
Work record: /Users/ksh3/familiar-workspace/workflow/django/2026-09-15/18-03-25
STATUS    DELEGATION           SESSION              ACCEPTED
accepted  del_W9hVQjDgA2zzXQ   ags_NPoXgKU5IGYmAg   2026-09-15 18:04:17
Receipt only; execution is not confirmed.
Next: famdctl delegation wait del_W9hVQjDgA2zzXQ
  

To see dsh inside the Pod, I open a tunnel through compute.home.arpa with pod forward and view it in a browser.

  ksh3@desktop.home.arpa ~/familiar-workspace % famdctl pod forward del_W9hVQjDgA2zzXQ
http://127.0.0.1:13080/?token=<redacted>
Pod pod_EaTd58jSdfE2kA is forwarded through compute.home.arpa; press Ctrl-C to close the tunnel.
  

The Web UI has Chat and Trajectory tabs, where you can see the To-dos work plan and the files in /workspace inside the Pod. Sessions that ran in a Pod are also synced to ~/familiar-workspace/dsh-local/Remote/ on the desktop and can be opened from the desktop dsh as Remote / artifact_….

Here is podman ps on compute-server when neither vLLM nor a Pod is running.

  ksh3@compute-server:~$ podman ps
CONTAINER ID  IMAGE                                                                                                            COMMAND               CREATED         STATUS         PORTS       NAMES
77af06a3024b  registry.home.arpa/ml-foundry-mlflow@sha256:e0916a7bce42adc92f5b688e212ece81c0b1ce363e92eaf02b1118161f23cf3b     mlflow server --h...  5 hours ago     Up 5 hours                 mlflow
2144fc65532d  registry.home.arpa/ml-foundry@sha256:df3d06efff1570a47085c8002d980d66085c01c69b5d07d5afab98c4f482c888            dagster api grpc ...  5 hours ago     Up 5 hours                 dagster-user-code
381a9063ea00  registry.home.arpa/ml-foundry@sha256:df3d06efff1570a47085c8002d980d66085c01c69b5d07d5afab98c4f482c888            dagster-webserver...  5 hours ago     Up 5 hours                 dagster-webserver
9df064d0e85b  registry.home.arpa/ml-foundry@sha256:df3d06efff1570a47085c8002d980d66085c01c69b5d07d5afab98c4f482c888            dagster-daemon ru...  5 hours ago     Up 5 hours                 dagster-daemon
c9849a9fd9a5  registry.home.arpa/famd-inference-relay@sha256:702149bdc55611ef25ba40d98c0181d55e9d786507ad5b991e17ccbc96bd4ed4  --upstream http:/...  17 minutes ago  Up 17 minutes              famd-egress
  

famd-egress is the inference relay. famd’s control and agent-gateway are native services, so they don’t show up here.

Dagster and MLflow are ml-foundry. It is the former model-foundry, also rebuilt to fit agent-gateway, and it now has an ML pipeline for my hobby investing in addition to models.

During the Vision-Exp run, the vLLM container ancestor-deepseek-v4-flash-vision-exp started by ancestor and the Pod’s dsh runner and dsh web were also running. The two Pod containers use the same famd-dsh-runner image: --managed is dsh itself, and --proxy-only is the Web relay. The Pod publishes only one port on 127.0.0.1, and the Web UI connects to it through the tunnel.

The Vision-Exp run

In the first video, DeepSeek-V4-Flash-Vision-Exp runs on vLLM on the compute host, and dsh inside a Podman Pod is given full access to implement a Django domain layer. Requests to the model never leave the host.

Configuration

  MODEL: deepseek-ai/DeepSeek-V4-Flash-Vision-Exp @ 6821d6ad3681a4b137b066b76094fa82ebd0a380
IMAGE: registry.home.arpa/voipmonitor/vllm:ds4-jovian-r9-5bea08859798
HOST:  compute.home.arpa:8000 (engine) <- pod forward <- desktop.home.arpa (client)
POD:   dsh runner, dsh web
GPU:   NVIDIA RTX PRO 6000 Blackwell Max-Q ×2 (GPU 0,1 / 300W cap)
  

This is the ExecStart of the vLLM service. The env file is read from ancestor’s active directory.

  ExecStart=/usr/bin/podman run --name=ancestor-deepseek-v4-flash-vision-exp --cidfile=%t/%N.cid --replace --rm --cgroups=split --network=host --init --sdnotify=conmon -d --ulimit stack=67108864:67108864 --device=nvidia.com/gpu=0 --device=nvidia.com/gpu=1 -v /mnt/data/hf/hub/models--deepseek-ai--DeepSeek-V4-Flash-Vision-Exp:/models:ro -v /mnt/data/hf/jit/ds4-vision-exp-jovian-r9:/cache -v /mnt/data/hf/tmp/ds4-vision-exp-jovian-r9:/container-tmp --env-file /home/ksh3/.local/state/ancestor/active/ancestor-deepseek-v4-flash-vision-exp.env --pull never --entrypoint=/usr/local/bin/lmcache-mp-wrapper.sh --ipc=host registry.home.arpa/voipmonitor/vllm:ds4-jovian-r9-5bea08859798 /usr/local/bin/serve-ds4-flash.sh
  

The serving profile is based on the Vision spec of DeepSeek-V4-Flash Jovian Judgement r9 published in local-inference-lab/rtx6kpro. The only exception is gpu_memory_utilization, which I set to 0.968 instead of either documented value (0.975 for GPU KV only, 0.970 with LMCache).

The entrypoint goes through the LMCache wrapper (lmcache-mp-wrapper.sh), but LMCACHE_MODE is not set. So the host tier is not used, and this is a GPU-KV-only run.

I wrote about measuring DeepSeek V4 Flash 0731 with DSpark K5 on the same two RTX PRO 6000 GPUs in another article. The model and settings are different, so I don’t put the numbers side by side.

Engine log numbers

Across a roughly 7-minute window of the engine log, there were 70 chat completion requests.

The prefix cache hit rate rose from 87.5% to 98.0% over the window. It dipped three times along the way when new prefixes came in.

Comparing decode-only windows with windows that include prefill, chunked prefill takes roughly 28% off decode on the same engine.

The dsh status bar in the video shows 201 tok/s and a 93% cache hit. Those are the client’s own numbers, counted over a different span with a different denominator, so they are not expected to match the engine counters above.

Caveats on the numbers

All numbers are averages of 10-second windows from the engine log. I did not capture TTFT, ITL, or per-request latency. Token totals are approximations from “rate × 10 seconds”, not exact counters.

Also, the published r9 numbers were measured on 600W Workstation cards. These are 300W Max-Q cards, so the numbers are not directly comparable.

About full access

I give it full access because what I struggled with most in the old familiar was excessive guard mechanisms. The guards pulled it into a negative spiral, and quality kept dropping. Even in the old version, I had moved to a mechanism where only complex shell commands, such as pipelines, were dry-run on the container side before being executed, and I removed the guards.

Compared with around March this year, OSS model performance has improved a lot. I use ds4, glm5.3-flash, and qwen38-flash-next for regular work.

That said, running without restrictions leaves every risk of destructive operations in place. If you try it, isolate the environment and take a snapshot first.

Thoughts

I spent about six months building my own agent in between other work, and it was a lot of fun. I came to understand model selection and the problems that show up when you actually run things, and I did a lot of MCP development along the way. I used to prefer borrowing code from OSS as much as possible and writing as little as possible myself, but thanks to AI coding, for the past half year I was absorbed in the fun of reinventing the wheel.

dsh supports plugins, and a community is forming around it. I think it won’t be long before I can borrow things that are continuously maintained instead of building them from scratch.

On top of that, Chinese engineering is impressive, and all kinds of tools are still being released at a remarkable pace. The scene is lively, and I also figured I could benefit technically in many ways, so I threw away the agent I had built and moved onto dsh.

I hope this is a useful reference for running agents in a local environment.