agent-gateway v3: domain separation and MLflow integration

The new structure

We split agent-gateway v3 from pipeline.knowledge.* into three domains: llm, obsidian, and mlflow. NATS subjects, Dagster sensors/jobs/assets, and Vector routing now use the same boundaries.

There was no production data, so the migration plan removed old topics and durable consumers without compatibility logic. Six workstreams and MinIO externalization produced 8 commits. go build ./... passed at each stage, and Dagster imports were checked.

Adding Obsidian vault and MLflow made knowledge too broad a label. For LLM integration, the new names distinguish conversation processing, documents, and experiment management.

This article is a sequel to Go + NATS + Dagster AI Orchestration Platform.

Prerequisites

• Target system: agent-gateway — a custom LLM orchestration platform built with Go + Python
• Runtime: EPYC 9175F (768GB DDR5, RTX 6000 Pro Max-Q), Podman rootless
• Messaging: NATS 2.11 (JetStream)
• Orchestration: Dagster (Python)
• Telemetry: Vector (Rust) → Prometheus / Loki
• New integrations: Obsidian vault, MLflow Model Registry
• Design document: docs/2nd-design-archi.md

Design Decisions

Domain Split Strategy

Chat persistence, embedding, and tool calls previously shared pipeline.knowledge.*.

• When adding Obsidian vault operations, pipeline.knowledge.embedding was indistinguishable — chat embedding or vault indexing?
• Placing MLflow experiment management events under the knowledge domain was semantically broken
• Dagster sensor routing logic could not infer intent from subject names
• Telemetry classification no longer mapped to functional boundaries

The three domains after the split:

Telemetry follows the same split: telemetry.llm.* / telemetry.obsidian.* / telemetry.mlflow.*.

No Backward Compatibility

With no production data, we removed old topics and consumers instead of writing migration logic. This reduced the implementation work.

Workstream Structure

The 6 workstreams followed dependency order, with 1 checkpoint commit per WS.

  WS1 (Lakehouse simplification) → WS2 (NATS topic migration) → WS3 + WS4 + WS5 (parallel) → WS6 (Vector config)
  

WS3 (Go domain layer) / WS4 (Go transport layer) / WS5 (Python Dagster) operate on different layers and could proceed in parallel.

Implementation

WS1: Lakehouse Simplification

We removed Nessie + Trino + dbt-fusion, introduced for earlier ELT tests. PyIceberg + JDBC catalog covered the needs at this stage.

Changes:

• Removed nessie, trino, dbt-fusion services and trino-warehouse volume from podman-compose.yml
• Removed TrinoBaseURL, NessieBaseURL fields and env var loading from internal/config/config.go
• Added Iceberg JDBC catalog metadata tables (iceberg_tables, iceberg_namespace_properties) to devstack/postgres/init.sql
• Deleted devstack/trino/ directory entirely

WS2: NATS Topic Migration

We replaced pipeline.knowledge.* constants with domain-specific names across 5 files.

Telemetry topics (telemetry.knowledge.* → telemetry.llm.* / telemetry.obsidian.*) and DAG names (knowledge.prompt_flow → llm.prompt_flow) were updated simultaneously.

grep "pipeline\.knowledge\|telemetry\.knowledge" confirmed that no old topics remained under internal/.

WS3: New domain/obsidian Package

We added 3 files under internal/domain/obsidian/.

repository.go — VaultRepository interface and local filesystem implementation LocalVault. A resolve() method prevents path traversal by rejecting access outside the vault root.

service.go — Service struct with 5 methods:

The knowledge orchestrator handles LLM calls. The obsidian domain handles vault I/O and embedding, publishing events to pipeline.obsidian.* and telemetry to telemetry.obsidian.*.

indexer.go — Vault scan → chunk split → embedding → pgvector upsert pipeline.

  vault files → scan → chunk (512 rune, 64 overlap) → embed → pgvector upsert
  

We added ObsidianVaultDir and OBSIDIAN_VAULT_DIR to internal/config/config.go, and initialized obsidian.NewService in cmd/server/main.go. The transport connection remains unimplemented.

WS4: MLflow Webhook Endpoints

We added 2 MLflow handlers to the Go transport layer.

webhook_mlflow.go — POST /v1/webhook/mlflow

The handler receives Model Registry webhooks, checks HMAC-SHA256 through X-Webhook-Signature, and publishes the event to NATS.

pipeline_mlflow.go — 2 endpoints:

Added WithPipelineService ServerOption to server.go, following the existing WithConversationRepository DI pattern. Added MLflowWebhookSecret (MLFLOW_WEBHOOK_SECRET env var) to config.go.

WS5: Dagster Sensor/Job/Asset Restructuring

The change covered 16 files: +401 lines / -229 lines.

We replaced 6 per-topic sensors with 4 wildcard sensors.

Each sensor subscribes to a wildcard subject and selects a job from the received subject suffix.

Deleted old job files (chat_persist.py, chat_pair_materialize.py, knowledge_events.py, tool_calls.py) and created domain-specific llm_jobs.py, obsidian_jobs.py, mlflow_jobs.py.

Asset key_prefix updated to domain-specific values:

• ["knowledge", "embedding"] → ["obsidian", "embedding"]
• knowledge_chat_audit → llm_chat_audit
• knowledge_events → pipeline_events

WS6: Vector Config Update

Added domain-based routing transforms to devstack/vector/vector.yaml.

  source: telemetry.> (wildcard)
    |
    v
route_by_domain (branch by subject prefix)
    |-- telemetry.llm.*     → domain: "llm"
    |-- telemetry.obsidian.* → domain: "obsidian"
    |-- telemetry.mlflow.*  → domain: "mlflow"
    |
    v
add_domain_label (.domain field injection)
    |
    +-- sink: console
    +-- sink: prometheus_exporter (with domain label)
  

Additional: MinIO Externalization

We switched devstack to the existing MinIO at storage.home.arpa:9000. Its archives, artifacts, datasets, lightdash, and models buckets remain separate from the new names.

• Removed minio service and minio-data volume from podman-compose.yml
• Updated dagster-user-code and dagster-daemon S3 endpoints to storage.home.arpa
• Added minio-init container to auto-create agw-mlflow and agw-iceberg buckets with agw- prefix (namespace isolation from existing buckets)
• Updated MLflow artifact root from s3://mlflow/ to s3://agw-mlflow/

Caveats

• The obsidian domain indexer lacks vault diff detection. Currently only full-scan IndexAll is implemented
• Old NATS JetStream durable consumer names remain as residue. Stream purge/recreation via nats-init is needed
• Dagster UI asset graph behavior has not been verified
• The obsidian domain’s transport layer connection remains a TODO
• During WS5 implementation, tool context was cleared requiring re-reads of defs.py and assets/__init__.py, but this did not affect the implementation itself
• The initial generation of indexer.go produced broken encodeJSON helper code (an unnatural pattern attempting to fallback var jsonMarshal to fmt.Sprintf). Fixed to call json.Marshal directly

Verification

• go build ./... passed at each workstream
• Dagster imports verified
• grep "pipeline\.knowledge\|telemetry\.knowledge" confirmed no old topic residue under internal/
• All commits on the feature/redesin-devstack branch

Next Actions

Commit History