Local parallel-agent development with Step-3.7-Flash

I used Step-3.7-Flash-NVFP4 and my familiar framework to generate a restaurant reservation API and RBAC-aware admin system. Six roles ran in parallel, with local inference and measurement and no external APIs.

Task decomposition and dependency resolution were stable. It reached a roughly 70% working state quickly, then needed more repairs later, partly because of the Guard layer.

Video link: https://www.youtube.com/watch?v=L6ryJhd2EmA

Recent Groundwork: Generation Token Estimation and Stream Continuation

I used dwarfstar commits as a reference for token estimation and budget control. I moved BackendGenerationTokenController into internal/domain/agent and added stream continuation to resume generation cut by a length cap.

The previous session repeated XML/TOML parse-error repairs. Synthesizing a recovery point before the run degraded worked better than post-processing alone. dwarfstar was moving toward a DeepSeek-4-specific single model, while I prefer replaceable OSS models, but the problems overlap.

BackendGenerationTokenController does not send requests. It observes and returns four kinds of control information:

• Watches OpenAI-compatible stream chunks and estimates generated completion tokens with the existing token estimator
• Detects overrun against max_tokens/context-window limits
• Returns control decisions such as continue / cutoff / repair-needed / terminal
• Detects malformed tool arguments at tool-call finish time, including cases where usable content can be salvaged from thinking output

Prompt synthesis, continuation, repairs, final responses, and telemetry remain in the outer agent. The controller is a state machine that observes the stream and returns decisions.

I also adjusted dynamic sampling per role. Since stream continuation can now recover from a length cap, I lowered max_output_tokens, narrowed sampling, and reduced stalls in long reasoning/output. For tester, I used repetition_penalty 1.08 with lower temperature/top_p to reduce empty verification loops.

Setup

The framework supports up to three models. This run used Step-3.7-Flash for both orchestrator and worker role sessions. Fewer models have tended to improve success and token cost in my use; I suspect differences in training distributions make shared context harder. Multiple models may fit loosely coupled roles or specialized sub-10B jobs through Rust Ort/ONNX.

The main vLLM runtime configuration was:

  Exec=/models/snapshots/36afbf6e15100cdc2d7a5b79d7e95d276ed33679 \
    --host 0.0.0.0 \
    --port 8000 \
    --served-model-name grandpa \
    --gpu-memory-utilization 0.92 \
    --tensor-parallel-size 2 \
    --max-num-seqs 8 \
    --kv-cache-dtype fp8 \
    --max-model-len 147456 \
    --disable-cascade-attn \
    --enable-expert-parallel \
    --trust-remote-code \
    --quantization modelopt \
    --reasoning-parser step3p5 \
    --tool-call-parser step3p5 \
    --enable-auto-tool-choice \
    --enable-expert-parallel \
    --async-scheduling
  

Grafana + Prometheus / Loki / PostgreSQL / Vector / Tempo / Alloy provide observation. Claude/Codex debug from the dashboards, and I use those findings in implementation.

vLLM Interval Benchmark

Aggregated:

This is an seq=8 configuration, but it is close to the batch behavior I ran with a soft cap of 7.

Single-request measurements are separate, and the benchmark is included in the video description.

Video link: https://www.youtube.com/watch?v=NDhzjHW-2SM

Running the Session

Chat DAG: Conversation Shape

vector/loki displays the conversation DAG before datastore finalization. The flow user request -> orchestrator -> Scheduler Decisions -> Dependencies -> worker -> orchestrator shows which turn spawned each worker.

Orchestrator Decisions: Inside the Decisions

This dashboard breaks down orchestrator decisions.

Guard Calls: Blocking and Recovering from Unsafe Behavior

The system has a Guard layer that stops or redirects agent behavior at runtime. In this session there were 40 Guard Calls: 11 blocking, 29 non-blocking, with a Block Rate of 27.5%.

MOLsEV: Prompt Quality and Role Synergy

Result

It took about one hour to work fully and about 24 minutes to reach roughly 70%. Final repairs took much of the time.

Out of seven deliverables, about two passed command-based startup without manual edits.

• Django admin worked as-is
• The API worked
• api/docs, the Swagger UI, was generated with auth and v1 resources listed
• Seed data was loaded

The remaining five needed light fixes. Opening api/docs showed realistic domain resources such as auth/token, reservations, reservation-holds, waitlist, availability/report, and cover-forecast/report, including both CRUD and report-style endpoints.

Notes

Like Step-3.5, Step-3.7 felt suited to orchestration. Attempt-1 structured output was 100%. TG averaged 176.6 tok/s, with median 175.7 and range 153.9-201.1. Prefix hits stayed at 81.5-81.8%. The context manager retained roughly 70-90%, and ctxMgr kept cache near 80% across roles.

In my environment, MTP did not work with NVFP4. SGLang with cu13 also failed because it only supported up to SM100. I tried GGUF through ik_llama as well, but crashes and output quality were unstable.

Claude/Codex handled about 70% of the framework and dashboard development, mainly design and outline implementation. I turned ideas and dashboard findings into experiments and changes. For agent-assisted business application development, having this observation layer first made debugging easier.