RAG and event processing in agent-gateway

OpenAI-compatible API and RAG

agent-gateway provides LLM and RAG functions through an OpenAI-compatible HTTP API. I built Phase 1 of this homelab AI infrastructure over roughly two weeks in early to mid-March. It separates inference, retrieval, and records needed for LLM integration.

The later v3 redesign split the domains, and the full-layer refactor applied Clean Architecture. This article covers the earlier configuration.

The Architecture Overview

The Go + Gin implementation separates responsibilities into layers.

  Transport Layer (Gin) [internal/transport/http/]
    -- middleware: RequestContext -> Logger -> Recovery
    -- v1/ handlers (OpenAI-compatible endpoints)

Presentation Layer [pkg/openai/]
    -- OpenAI-compatible / Anthropic-compatible request/response DTOs

Domain Layer [internal/domain/]
    |-- knowledge/  RAG orchestration
    |-- llm/        Multi-backend LLM routing
    |-- vectorstore/ Vector search
    +-- pipeline/   Pipeline integration (NATS pub/sub, Dagster triggers)

Infrastructure Layer [internal/infra/]
    |-- vllm/       vLLM client
    |-- llamacpp/   llama.cpp client
    |-- lmstudio/   LM Studio client
    |-- postgres/   PostgreSQL + pgvector
    |-- nats/       NATS JetStream pub/sub
    |-- dagster/    Dagster job triggers
    +-- reranker/   ColBERT reranker client
  

Configuration selects among three LLM backends, using GPU or CPU inference according to the task.

The catalog holds over 40 models. Name patterns route them: qwen3- to vLLM and nemotron- to llama.cpp.

Separating requests and persistence with NATS

Chat completion responses should not wait for persistence.

After responding, the gateway publishes events asynchronously to NATS. Dagster consumes and persists them. Publishing is fire-and-forget.

  User -> gateway -> LLM backend -> response returned
                \-> NATS publish (fire-and-forget)
                      \-> Dagster sensor -> JetStream pull -> PostgreSQL
  

Published topics:

  pipeline.knowledge.chat.persist    -- chat persistence
pipeline.knowledge.embedding       -- embedding generation events
pipeline.knowledge.retrieve        -- vector search events
pipeline.knowledge.flow.lineage    -- flow DAG lineage tracking
pipeline.knowledge.tool_call       -- tool call logs
  

Vector handles telemetry separately, subscribing to NATS and forwarding to Prometheus and Loki.

Dagster-Side NATS Pipeline

In model-foundry, I implemented Dagster assets and jobs using NATS JetStream consumers.

• Sensors: poll topics via NATS JetStream durable consumers
• Assets: materialize received events as DuckDB tables
• Jobs: knowledge_chat_persist, knowledge_embedding, knowledge_lineage

JetStream pull consumers may redeliver events, so persistence must be idempotent.

  INSERT INTO chat_pairs (correlation_id, prompt, response, model, created_at)
VALUES ($1, $2, $3, $4, $5)
ON CONFLICT (correlation_id) DO NOTHING;
  

RAG Orchestration Flow

The RAG flow for a chat completion request:

1. Extract user query
2. Generate embeddings (multi-bert-inference gRPC)
3. Vector search with pgvector + ColBERT reranking
4. Attach search results as context
5. Send to LLM backend
6. Publish pipeline events to NATS
7. Return response

PostgreSQL + pgvector stores vectors. The document_chunks table has an embedding vector(256) column and a cosine HNSW index.

  CREATE TABLE document_chunks (
    id SERIAL PRIMARY KEY,
    document_id TEXT NOT NULL,
    chunk_index INT NOT NULL,
    content TEXT NOT NULL,
    embedding vector(256),
    created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX idx_chunks_embedding ON document_chunks
    USING hnsw (embedding vector_cosine_ops);
  

Cache hits reuse embeddings for retrieval. Misses generate and store embeddings before passing context to the LLM.

Containerizing multi-bert-inference

The Rust gRPC server multi-bert-inference handles embeddings and reranking. I moved it from a standalone process on desktop.home.arpa into a devstack container.

3-Stage Containerfile

  # Stage 1: Build only dependencies for caching
FROM rust:1.94.0-slim-trixie AS deps
RUN apt-get update && apt-get install -y \
    protobuf-compiler pkg-config libssl-dev g++
WORKDIR /app
COPY Cargo.toml Cargo.lock build.rs proto/ ./
RUN mkdir src && echo "fn main() {}" > src/main.rs && \
    cargo build --release && rm -rf src

# Stage 2: Build actual source
FROM deps AS builder
COPY src/ src/
RUN cargo build --release && strip target/release/multi-bert-inference

# Stage 3: Minimal runtime
FROM debian:trixie-slim
RUN apt-get update && apt-get install -y ca-certificates curl && \
    rm -rf /var/lib/apt/lists/*
RUN useradd -r -s /bin/false app
USER app
COPY --from=builder /app/target/release/multi-bert-inference /usr/local/bin/
EXPOSE 50051 3000
ENTRYPOINT ["multi-bert-inference"]
  

Stage 1 builds Cargo dependencies using a dummy main.rs, allowing cached rebuilds after source changes. Multi-GB model files are volume-mounted rather than included in the image, so model updates need no image rebuild.

  podman build -t desktop.home.arpa/multi-bert-inference .
  

Devstack Integration

Added as the inference service in devstack/desktop/podman-compose.yml:

  inference:
  image: desktop.home.arpa/multi-bert-inference
  ports:
    - "50051:50051"
    - "3800:3000"
  volumes:
    - /Users/ksh3/Development/multi-bert-inference/models:/app/models:ro,z
  environment:
    RUST_LOG: multi_bert_inference=info
  healthcheck:
    test: ["CMD-SHELL", "curl -sf http://localhost:3000/healthz || exit 1"]
    interval: 10s
    timeout: 3s
    retries: 3
  restart: unless-stopped
  

Expose gRPC on 50051 and REST health checks on 3000, mapped to host port 3800. Mount the model directory read-only.

When inference is unavailable

internal/infra/inference/client.go manages the connection and provides three clients.

• EmbeddingServiceClient: Embed() (256-dimensional) and EmbedContext() (late chunking)
• RerankServiceClient: Rerank() (ColBERT MaxSim)
• SearchServiceClient: MaxSimSearch()

The inference connection is optional. The gateway can start without it.

  var inferClient *inference.Client
if strings.TrimSpace(cfg.InferenceGRPCAddr) != "" {
    var inferErr error
    inferClient, inferErr = inference.NewClient(ctx, cfg.InferenceGRPCAddr)
    if inferErr != nil {
        slog.Warn("inference gRPC unavailable", "err", inferErr, "addr", cfg.InferenceGRPCAddr)
    } else {
        slog.Info("inference gRPC connected", "addr", cfg.InferenceGRPCAddr)
        orc.SetRerankerRepository(inference.NewRerankRepository(inferClient))
        defer inferClient.Close()
    }
}
  

An unavailable service disables embed/rerank while the LLM proxy and other endpoints keep working. Development also needs fewer running services.

Embedding Calls from Dagster

Dagster uses the same gRPC endpoint. EmbeddingResource encodes and decodes protobuf directly to call /search.EmbeddingService/Embed, producing 256-dimensional embeddings for knowledge_events and chat_pairs.

The col-bert-api INT8 error

I also ran col-bert-api, an axum embed/rerank API using candle-onnx. Its INT8 model failed during inference.

onnx/model_int8.onnx loaded but produced a runtime error. The same code worked with FP32 onnx/model.onnx. The CPU lacked the i8/VNNI instructions required by INT8.

  // Before: INT8 priority fallback
// 1. onnx/model_int8.onnx -> 2. onnx/model_fp16.onnx -> 3. onnx/model.onnx

// After: FP32 fixed
let model = candle_onnx::read_file("onnx/model.onnx")?;
  

I changed the README wording from “Prefer provided INT8 models if available” to FP32. Loading a quantized model is insufficient; inference must also work on the target CPU.

What Phase 1 Delivered

Implemented Endpoints

LLM Parser Chain

A four-stage parser chain normalizes LLM output.

1. ApplyToolCallParser – tool call normalization
2. ApplyReasoningParser – thinking/reasoning tag extraction
3. ApplyVisionParser – image base64 processing
4. ApplyOutputParser – output format normalization

Middleware

The chain is RequestContext -> Logger -> Recovery. RequestContext extracts or generates X-Correlation-ID and attaches it to subsequent logs.

Service Topology

  desktop.home.arpa
  -- agent-gateway :8080
  -- NATS :4222
  -- multi-bert-inference :50051

compute.home.arpa
  -- vLLM :8000
  -- llama.cpp :8081
  -- LM Studio :1234
  -- PostgreSQL :5432
  -- Dagster :3300

storage.home.arpa
  -- Prometheus :9090
  -- Loki :3100
  -- MinIO :9000
  

Phase 1 connected the following paths.

• gateway -> NATS publish (fire-and-forget)
• Dagster sensor -> JetStream pull -> PostgreSQL
• Vector -> NATS subscribe -> Prometheus/Loki
• pgvector ANN search + ColBERT reranking

The next v3 redesign revisits the domain boundaries.