familiar: a local LLM development platform and its observation tools
A 55-minute familiar demo generated a Django reservation system. Local infrastructure, model selection, observation, and recovery for AI system development and LLM integration.
Introduction
Over three months from February, I implemented familiar, a local-LLM multi-agent development platform connected to a data stack. The name refers to family and a familiar spirit. Its internal aliases are grandpa, frisky, and naughty.
I recorded a 55-minute demo on YouTube.
Video link: https://www.youtube.com/watch?v=tSAguJzTINs
The demo planned, implemented, tested, and reviewed a Django restaurant reservation system without human intervention. It produced 33 commits, used about 11 million tokens, passed all 13 tests, and received approval from all reviewers.
React worked in light tests. I also prepared Tauri, Flutter, Hugo, and other work or personal stacks. X-Role Header variables include planning (Strategist), investing (Analyst), and PR (Spokesman). Orchestrators and workers can gain roles and swap models.
This article records the development path and implementation decisions.
What familiar is
familiar splits work among local LLMs and manages execution, retries, integration, and completion. It combines an on-prem runtime, observation, and a training pipeline.
A human supplies the first request and final check. Eventually I want only systemctl start familiar.service to be needed. The orchestrator plans, assigns, and judges results. Workers implement, test, review, design visuals, and sometimes integrate. The runtime stores requests, completion reports, evidence, artifacts, and verdicts.
I used the runtime and tools to support weak model behaviors and keep work moving after failures.
Orchestrator decisions and worker results share a timeline. Successes, failures, blocked dependencies, retries, and reproducibility show whether to change instructions, execution order, or tools.
Model development is difficult for me because of cost and data rights, so I archive OSS models with suitable licenses. grandpa can use GLM-5.1, Kimi-K2.5, or DeepSeek V4 Flash. Newer models can be substituted and supported by accumulated datasets. At the time, GLM-5.1 was the most stable orchestrator for my requirements, with Qwen3.6 and gemma4 workers offering faster implementation and growing MTP support. More candidates appeared from February through April.
Observation grew with the runtime. Records include a Grafana Familiar folder backup with 54 dashboard generations synchronized. Views include Orchestrator Decisions, Workflow & Replay, MOLsEV, C&F Latency, Arch Stats, Tool Calls, Guard Calls, LoRA/Prompt ROI, and Model/Quant Failure Heatmap.
Prompts and model choice needed support from OSS observation, search, and execution tools.
It started with Qwen Next
I started with Qwen Next around last July.
MoE, long context, and multiple models led me to expect more CPU inference. I designed CPU, memory bandwidth, L3 cache, storage, networking, and GPU together.
Concern about semiconductor shortages also led me to plan the hardware around August.
PC building is my hobby, but this was a large expense and risk. I ordered the HPC configuration and waited three months. EPYC 9175F has 16C/16T and 512MB L3, which appealed to me for CPU inference.
I split the setup into compute, storage, desktop, and edge.
- compute.home.arpa: EPYC 9175F, large DDR5 memory, and two NVIDIA GPUs for inference.
- storage.home.arpa: PostgreSQL, NATS JetStream, MinIO, Gitea, OCI Registry, Prometheus, Loki, Vector, and Caddy. A Mac mini Late 2018 (x86) runs Linux 24/7. Used units cost about 30,000 yen, avoid Arm compatibility work, and I collect suitable Mercari units as spares.
- desktop.home.arpa: daily work, UI, and external-network access.
- edge.home.arpa: RouterOS, Step-CA, and Caddy for internal TLS and the network boundary.
Although incorporated, I work as a freelance engineer. This was a large investment.
With little prior knowledge and many planned experiments, I designed around owned hardware. After building the prototype, I think a used x86 Mac mini around 30,000 yen plus cloud GPUs could work, but I could not judge that beforehand.
I planned to reduce a long-running client project and focus from December, including preparing to leave that work.
In December, an engineer at a long-time client became seriously ill, leaving less time available. Things settled in February, when I began the smaller tasks.
I first separated networks, service placement, inference, persistence, observation, and development. Agents came after infrastructure that could run continuously and expose failures.
At first it was a proxy for observing LLM usage
The first version was a lightweight backend for observing LLM use.
Rust + axum proxied LLM requests. Knowledge.Gate collected prompts, responses, tool calls, and failures, with Dagster, MLflow, NATS, Prometheus, and Grafana around it.
For cloud and local models, output alone did not show what to improve. I kept histories, failure patterns, token use, and model behavior.
I then wanted decisions and recovery in the same platform.
I defined what to observe, how much autonomy to allow, where decisions belonged, how to recover, and which logs could support learning.
I replaced Rust + axum with Go. Goroutines and channels made parallel execution, state, scheduling, and subscriptions easier. I wanted fewer external MCP / Python dependencies and a resident platform close to one binary.
The video ends by stopping the session, but the runtime accepts external pub/sub requests. I also designed directory-based tasks with KV eviction and success | fail | loop states. Go suited low-cost waiting and resumption.
Selecting models with short tasks
Model selection began around January.
Without time for large features, I downloaded models and sent short tasks. I watched prompt failures, tool calls, long text, empty output, XML errors, unclosed tools, CPU-offload waits, and quantization limits.
The storage-side Hugging Face archive records those trials.
ksh3@storage-server:~$ ls -ltr /srv/archive/cold/hf/hub/
total 0
drwxr-xr-x 5 ksh3 ksh3 64 Jan 4 19:44 models--Qwen--Qwen3-VL-32B-Instruct
drwxr-xr-x 5 ksh3 ksh3 64 Jan 4 20:21 models--AaryanK--IQuest-Coder-V1-40B-Instruct-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 Jan 4 23:30 models--NousResearch--Hermes-4.3-36B
drwxr-xr-x 5 ksh3 ksh3 64 Jan 5 00:11 models--unsloth--gpt-oss-120b-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 Jan 5 01:56 models--NousResearch--Hermes-4-70B-FP8
drwxr-xr-x 5 ksh3 ksh3 64 Jan 5 08:23 models--lmstudio-community--Llama-4-Scout-17B-16E-Instruct-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 Jan 9 10:23 models--nvidia--Llama-3.3-70B-Instruct-NVFP4
drwxr-xr-x 5 ksh3 ksh3 64 Jan 9 11:33 models--nvidia--Llama-4-Scout-17B-16E-Instruct-NVFP4
drwxr-xr-x 5 ksh3 ksh3 64 Jan 9 11:46 models--Firworks--command-a-reasoning-08-2025-nvfp4
drwxr-xr-x 5 ksh3 ksh3 64 Jan 9 19:59 models--Elias-Schwegler--IQuest-Coder-V1-40B-Loop-Instruct-NVFP4
drwxr-xr-x 5 ksh3 ksh3 64 Jan 9 21:51 models--miromind-ai--MiroThinker-v1.5-30B
drwxr-xr-x 5 ksh3 ksh3 64 Jan 9 22:37 models--IQuestLab--IQuest-Coder-V1-40B-Loop-Instruct
drwxr-xr-x 5 ksh3 ksh3 64 Jan 9 22:37 models--IQuestLab--IQuest-Coder-V1-40B-Instruct
drwxr-xr-x 6 ksh3 ksh3 85 Jan 11 01:03 models--unsloth--Llama-4-Maverick-17B-128E-Instruct-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 Jan 11 01:47 models--pfnet--plamo-2-translate
drwxr-xr-x 5 ksh3 ksh3 64 Jan 11 02:15 models--google--functiongemma-270m-it
drwxr-xr-x 5 ksh3 ksh3 64 Jan 11 02:17 models--Firworks--gemma-3-270m-it-nvfp4
drwxr-xr-x 5 ksh3 ksh3 64 Jan 11 05:44 models--BCCard--gemma-3-27b-it-NVFP4A16
drwxr-xr-x 5 ksh3 ksh3 64 Jan 11 07:01 models--openai--gpt-oss-20b
drwxr-xr-x 5 ksh3 ksh3 64 Jan 11 07:27 models--google--gemma-3-27b-it
drwxr-xr-x 5 ksh3 ksh3 64 Jan 11 10:46 models--GAlex535--Qwen3-Coder-30B-A3B-Instruct-NVFP4
drwxr-xr-x 5 ksh3 ksh3 64 Jan 11 13:17 models--openai--gpt-oss-120b
drwxr-xr-x 5 ksh3 ksh3 64 Jan 11 15:20 models--mratsim--Monstral-123B-v2-NVFP4
drwxr-xr-x 6 ksh3 ksh3 85 Jan 11 15:37 models--Lightricks--LTX-2
drwxr-xr-x 5 ksh3 ksh3 64 Jan 12 01:04 models--ChristianAzinn--mixtral-8x22b-v0.1-imatrix
drwxr-xr-x 5 ksh3 ksh3 64 Jan 20 08:57 models--Firworks--NVIDIA-Nemotron-3-Nano-30B-A3B-nvfp4
drwxr-xr-x 5 ksh3 ksh3 64 Jan 20 10:13 models--Qwen--Qwen3-Coder-30B-A3B-Instruct-FP8
drwxr-xr-x 5 ksh3 ksh3 64 Jan 22 04:07 models--zai-org--GLM-4.7-Flash
drwxr-xr-x 5 ksh3 ksh3 64 Feb 5 15:11 models--nvidia--NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4
drwxr-xr-x 5 ksh3 ksh3 64 Feb 13 22:57 models--unsloth--Qwen3-Next-80B-A3B-Thinking-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 Feb 14 00:46 models--AesSedai--Step-3.5-Flash-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 Feb 15 06:15 models--unsloth--GLM-5-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 Feb 15 11:44 models--ubergarm--GLM-4.7-Flash-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 Feb 15 18:10 models--DavidAU--GLM-4.7-Flash-Uncensored-Heretic-NEO-CODE-Imatrix-MAX-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 Feb 15 19:11 models--bartowski--Qwen_Qwen3-Coder-Next-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 Feb 16 00:52 models--ubergarm--DeepSeek-V3.2-Speciale-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 Feb 16 03:44 models--DavidAU--Openai_gpt-oss-120b-NEO-Imatrix-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 Feb 17 08:49 models--ubergarm--MiniMax-M2.5-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 Feb 17 19:40 models--GadflyII--Qwen3-Coder-Next-NVFP4
drwxr-xr-x 5 ksh3 ksh3 64 Feb 18 15:12 models--ACE-Step--Ace-Step1.5
drwxr-xr-x 5 ksh3 ksh3 64 Feb 18 15:41 models--mistralai--Voxtral-Mini-4B-Realtime-2602
drwxr-xr-x 5 ksh3 ksh3 64 Feb 18 17:18 models--black-forest-labs--FLUX.2-klein-9B
drwxr-xr-x 5 ksh3 ksh3 64 Feb 18 23:14 models--LiquidAI--LFM2-8B-A1B
drwxr-xr-x 5 ksh3 ksh3 64 Feb 19 00:09 models--LiquidAI--LFM2-8B-A1B-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 Feb 19 05:58 models--LiquidAI--LFM2.5-1.2B-Thinking
drwxr-xr-x 5 ksh3 ksh3 64 Feb 19 06:03 models--LiquidAI--LFM2.5-VL-1.6B
drwxr-xr-x 5 ksh3 ksh3 64 Feb 19 06:07 models--LiquidAI--LFM2.5-1.2B-Instruct
drwxr-xr-x 5 ksh3 ksh3 64 Feb 19 19:49 models--ubergarm--Devstral-2-123B-Instruct-2512-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 Feb 23 20:30 models--AesSedai--MiniMax-M2.5-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 Feb 24 06:41 models--nvidia--Qwen3-Next-80B-A3B-Instruct-NVFP4
drwxr-xr-x 5 ksh3 ksh3 64 Feb 24 06:47 models--nvidia--Qwen3-Next-80B-A3B-Thinking-NVFP4
drwxr-xr-x 5 ksh3 ksh3 64 Feb 25 01:54 models--AesSedai--Qwen3.5-35B-A3B-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 Feb 25 03:16 models--AesSedai--Qwen3.5-122B-A10B-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 Feb 25 16:57 models--Sehyo--Qwen3.5-122B-A10B-NVFP4
drwxr-xr-x 5 ksh3 ksh3 64 Feb 25 19:39 models--Qwen--Qwen3.5-27B
drwxr-xr-x 5 ksh3 ksh3 64 Mar 2 04:18 models--mmnga-o--NVIDIA-Nemotron-Nano-9B-v2-Japanese-gguf
drwxr-xr-x 5 ksh3 ksh3 64 Mar 2 06:23 models--nvidia--NVIDIA-Nemotron-Nano-9B-v2-Japanese
drwxr-xr-x 5 ksh3 ksh3 64 Mar 3 02:56 models--bartowski--Qwen_Qwen3.5-27B-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 Mar 3 03:53 models--ubergarm--Qwen3.5-27B-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 Mar 6 05:21 models--perplexity-ai--pplx-embed-context-v1-0.6b
drwxr-xr-x 5 ksh3 ksh3 64 Mar 6 05:24 models--perplexity-ai--pplx-embed-v1-0.6b
drwxr-xr-x 5 ksh3 ksh3 64 Mar 6 08:22 models--perplexity-ai--pplx-embed-context-v1-4b
drwxr-xr-x 5 ksh3 ksh3 64 Mar 6 08:23 models--perplexity-ai--pplx-embed-v1-4b
drwxr-xr-x 5 ksh3 ksh3 64 Mar 14 00:30 models--AesSedai--NVIDIA-Nemotron-3-Super-120B-A12B-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 Mar 14 11:03 models--nvidia--NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4
drwxr-xr-x 5 ksh3 ksh3 64 Mar 17 23:30 models--fishaudio--s2-pro
drwxr-xr-x 5 ksh3 ksh3 64 Mar 17 23:36 models--mradermacher--Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled-i1-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 Mar 18 00:00 models--bartowski--Qwen_Qwen3.5-4B-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 Mar 18 16:24 models--AesSedai--Mistral-Small-4-119B-2603-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 Mar 23 04:31 models--AesSedai--Nemotron-Cascade-2-30B-A3B-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 Mar 27 03:02 models--mconcat--Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled-NVFP4
drwxr-xr-x 5 ksh3 ksh3 64 Mar 27 03:37 models--Qwen--Qwen3.5-9B
drwxr-xr-x 5 ksh3 ksh3 64 Mar 27 04:54 models--chankhavu--Nemotron-Cascade-2-30B-A3B-NVFP4
drwxr-xr-x 5 ksh3 ksh3 64 Mar 27 21:50 models--Jackrong--Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled
drwxr-xr-x 5 ksh3 ksh3 64 Mar 27 21:57 datasets--OpenMOSS-Team--OmniAction
drwxr-xr-x 5 ksh3 ksh3 64 Mar 27 21:59 datasets--Roman1111111--claude-opus-4.6-10000x
drwxr-xr-x 5 ksh3 ksh3 64 Mar 27 22:11 datasets--nvidia--Nemotron-Terminal-Corpus
drwxr-xr-x 5 ksh3 ksh3 64 Mar 28 05:43 models--nvidia--Nemotron-Cascade-2-30B-A3B
drwxr-xr-x 5 ksh3 ksh3 64 Mar 28 09:17 datasets--nvidia--Nemotron-Terminal-Synthetic-Tasks
drwxr-xr-x 6 ksh3 ksh3 85 Mar 29 14:33 models--HauhauCS--Qwen3.5-35B-A3B-Uncensored-HauhauCS-Aggressive
drwxr-xr-x 5 ksh3 ksh3 64 Mar 29 17:33 models--Qwen--Qwen3.5-4B-Base
drwxr-xr-x 5 ksh3 ksh3 64 Mar 29 18:47 models--Qwen--Qwen3.5-35B-A3B-Base
drwxr-xr-x 5 ksh3 ksh3 64 Mar 29 18:55 models--mistralai--Mistral-Small-4-119B-2603-NVFP4
drwxr-xr-x 5 ksh3 ksh3 64 Mar 30 08:24 models--Qwen--Qwen3.5-9B-Base
drwxr-xr-x 5 ksh3 ksh3 64 Mar 30 08:24 models--zed-industries--zeta-2
drwxr-xr-x 5 ksh3 ksh3 64 Mar 30 11:24 models--nvidia--MiniMax-M2.5-NVFP4
drwxr-xr-x 5 ksh3 ksh3 64 Mar 30 12:59 models--sentence-transformers--all-MiniLM-L6-v2
drwxr-xr-x 5 ksh3 ksh3 64 Mar 30 13:02 models--datalab-to--chandra-ocr-2
drwxr-xr-x 5 ksh3 ksh3 64 Apr 2 14:29 datasets--ianncity--KIMI-K2.5-450000x
drwxr-xr-x 5 ksh3 ksh3 64 Apr 2 14:42 datasets--open-index--hacker-news
drwxr-xr-x 5 ksh3 ksh3 64 Apr 3 00:49 models--bartowski--google_gemma-4-26B-A4B-it-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 Apr 3 02:31 models--lightonai--LateOn-Code-edge
drwxr-xr-x 5 ksh3 ksh3 64 Apr 3 02:32 models--mixedbread-ai--mxbai-edge-colbert-v0-17m
drwxr-xr-x 5 ksh3 ksh3 64 Apr 3 11:33 models--google--gemma-4-26B-A4B-it
drwxr-xr-x 5 ksh3 ksh3 64 Apr 3 17:15 models--ubergarm--Step-3.5-Flash-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 Apr 6 07:00 models--bartowski--Qwen_Qwen3.5-9B-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 Apr 8 15:31 datasets--nohurry--Opus-4.6-Reasoning-3000x-filtered
drwxr-xr-x 5 ksh3 ksh3 64 Apr 8 16:17 datasets--ianncity--KIMI-K2.5-1000000x
drwxr-xr-x 6 ksh3 ksh3 85 Apr 8 16:38 models--ubergarm--Qwen3-Coder-Next-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 Apr 11 05:57 models--google--translategemma-4b-it
drwxr-xr-x 5 ksh3 ksh3 64 Apr 11 06:20 models--llm-jp--llm-jp-4-32b-a3b-base
drwxr-xr-x 5 ksh3 ksh3 64 Apr 12 18:35 models--MiniMaxAI--MiniMax-M2.7
drwxr-xr-x 6 ksh3 ksh3 85 Apr 13 02:52 models--ubergarm--MiniMax-M2.7-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 Apr 14 09:24 models--llm-jp--llm-jp-4-32b-a3b-thinking
drwxr-xr-x 5 ksh3 ksh3 64 Apr 16 09:22 models--AesSedai--Qwen3.5-397B-A17B-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 Apr 17 08:51 models--AesSedai--Qwen3.6-35B-A3B-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 Apr 20 09:02 models--cyberagent--CAT-Translate-7b
drwxr-xr-x 5 ksh3 ksh3 64 Apr 20 14:25 datasets--Jackrong--GLM-5.1-Reasoning-1M-Cleaned
drwxr-xr-x 5 ksh3 ksh3 64 Apr 23 00:11 models--openai--privacy-filter
drwxr-xr-x 5 ksh3 ksh3 64 Apr 23 15:41 models--Qwen--Qwen3.6-27B
drwxr-xr-x 5 ksh3 ksh3 64 Apr 23 16:36 models--Qwen--Qwen3.6-27B-FP8
drwxr-xr-x 5 ksh3 ksh3 64 Apr 24 00:41 models--huihui-ai--Huihui-GLM-5.1-abliterated-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 Apr 24 11:19 models--deepseek-ai--DeepSeek-V4-Flash
drwxr-xr-x 5 ksh3 ksh3 64 Apr 24 15:24 datasets--Modotte--CodeX-2M-Thinking
drwxr-xr-x 5 ksh3 ksh3 64 Apr 27 01:44 models--nsparks--DeepSeek-V4-Flash-FP4-FP8-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 Apr 27 16:02 models--zai-org--GLM-5.1
drwxr-xr-x 5 ksh3 ksh3 64 Apr 29 16:51 models--Qwen--Qwen3.6-35B-A3B-FP8
drwxr-xr-x 5 ksh3 ksh3 64 Apr 29 17:22 models--Qwen--Qwen3.6-35B-A3B
drwxr-xr-x 5 ksh3 ksh3 64 Apr 29 17:43 models--AesSedai--Kimi-K2.6-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 Apr 30 18:40 models--RedHatAI--DeepSeek-V4-Flash-NVFP4-FP8
drwxr-xr-x 5 ksh3 ksh3 64 May 1 16:03 datasets--AlicanKiraz0--Cybersecurity-Dataset-Fenrir-v2.1
drwxr-xr-x 5 ksh3 ksh3 64 May 1 22:50 models--ubergarm--GLM-5.1-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 May 2 09:46 models--mistralai--Mistral-Medium-3.5-128B
drwxr-xr-x 5 ksh3 ksh3 64 May 2 09:55 models--mistralai--Mistral-Medium-3.5-128B-EAGLE
drwxr-xr-x 5 ksh3 ksh3 64 May 2 14:43 models--RedHatAI--Qwen3.6-35B-A3B-NVFP4
drwxr-xr-x 5 ksh3 ksh3 64 May 3 14:55 models--ubergarm--Qwen3.6-27B-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 May 4 03:57 models--sakamakismile--Qwen3.6-27B-Text-NVFP4-MTP
drwxr-xr-x 5 ksh3 ksh3 64 May 4 15:06 datasets--Kassadin88--GLM-5.1-OpenThoughts3-Distill
drwxr-xr-x 5 ksh3 ksh3 64 May 7 14:01 models--LilaRest--gemma-4-31B-it-NVFP4-turbo
drwxr-xr-x 5 ksh3 ksh3 64 May 7 14:04 models--nvidia--Gemma-4-31B-IT-NVFP4
drwxr-xr-x 5 ksh3 ksh3 64 May 7 14:05 models--google--gemma-4-31B-it-assistant
drwxr-xr-x 5 ksh3 ksh3 64 May 9 01:20 models--RecViking--Mistral-Medium-3.5-128B-NVFP4
drwxr-xr-x 5 ksh3 ksh3 64 May 10 02:52 models--ubergarm--Kimi-K2.6-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 May 10 07:18 models--google--gemma-4-31B-it
drwxr-xr-x 5 ksh3 ksh3 64 May 10 10:25 models--Zyphra--ZAYA1-74B-preview
drwxr-xr-x 5 ksh3 ksh3 64 May 12 16:37 models--AesSedai--MiMo-V2.5-Pro-GGUF
drwxr-xr-x 5 ksh3 ksh3 64 May 12 17:21 models--antirez--deepseek-v4-gguf
ksh3@storage-server:~$
I tried uncensored models, storytellers, image and video generation, security, coding, and OCR to compare roles, quantization, and behavior.
Adding Claude to development
I enjoy writing code.
I had limited AI assistance to inline autocomplete. Copilot was useful, but I wanted to do the writing myself.
In February, a statement against military use prompted me to try Claude. I soon wished I had used it earlier.
Shoulder stiffness and tendinitis had made me consider retirement within a few years. Claude helped me keep developing, and I wanted a system to support my habits, knowledge, design choices, and criteria for discarding work.
Around March, I passed prompts to claude -p and Codex as small dependencies, observing work similar to local orchestrator tasks. I planned to transfer that knowledge to 100B-plus local models that could run on CPU.
Removing external model integrations in April
Anthropic’s April policy change led me to change the plan.
Autonomous subprocess use of claude -p looked uncertain to me, and I saw Reddit ban reports. I removed Claude / Codex integrations based on that judgment at the time.
Discarding working code was costly, but I needed orchestration knowledge rather than dependence on that integration.
I needed to observe task size, worker inputs, retryable versus design failures, review timing, and parallel-change integration inside my own platform.
I focused on Vector, Tempo, Alloy, and PostgreSQL observation tables.
The same correlation links plan, dispatch, worker result, tool call, token usage, task failure, review verdict, and recovery.
Grafana became a way to read that sequence by role, task, tool call, and latency.
I read failed sessions in Grafana and PostgreSQL, thought through hypotheses on walks, then changed the implementation and ran it again.
Removing the integrations helped me understand execution and establish repeatable experiments.
Tools for difficult model behaviors
Around March, I started building tools for weaknesses found in observation.
Long-context memory, ambiguous file search, result interpretation, failure state, and deciding what to carry into the next turn often broke when left to the model.
I built tools for common problems.
I made about 15 MCP, management, and debugging tools. The effort felt roughly 20% familiar core and 80% tools for observation, search, structuring, verification, and recovery.
Model internals are hidden. The same input can change with context, temperature, earlier failures, and tool use. Chained models add more behavior to trace.
I checked why tools were called, which information burdened context, and whether failures needed retries or permanent fixes.
MCP tools turn ambiguous work into consistent runtime functions.
I also changed some of my usual coding practices.
I normally prefer clear symbols and outlines over comments, with test names showing intent.
I refine names and structure to reduce how much someone must read.
For the LLM implementing familiar, comments and search labels were useful. I adapted the code to its user.
Typed context was not always the best format for the LLM.
Repeated shapes could add noise. Dynamic formats hid unnecessary information and separated concerns, even where I would normally avoid them.
Training and generation management remain future work. This records several months of development; I plan to continue it as infrastructure for local AI system development.