Placing data on NVMe/SATA and moving management UIs

Introduction

I separated query data on NVMe from logs on SATA, and moved management UIs from Linux to the Mac. Storage and host roles now follow the workload.

This data-platform design separates continuous logging from temporary analytics I/O.

Motivation: Why Separate Storage?

All data initially used NVMe. Several workloads competed during operation:

1. Continuous writes: systemd, Loki, and Prometheus use I/O continuously. Frequent fsync and synchronous writes can exhaust buffers and affect query latency.
2. Analytics contention: Trino and dbt spill writes compete with logs.
3. UI overhead: Dagit and Grafana generate their own logs and share resources with the data engine.

I separated device roles (NVMe/SATA) and host roles (Linux/Mac).

Redefining Storage Allocation Policy

Data is split into 2 groups by I/O needs.

NVMe: Optimized for High Throughput and Burst Reads

NVMe holds workloads requiring fast random access and low latency.

• Postgres: main database files, including WAL, for query performance.
• Spill/cache: Trino spills, Iceberg staging, and dbt targets with large temporary I/O.
• LLM models: Ollama and vLLM files, from GBs to tens of GBs, where loading time matters.
• Search and intermediate data: Qdrant indexes and Dagster caches.

SATA: Optimized for Steady Sequential Writes

SATA SSDs hold continuous writes and less frequently accessed files. The choice prioritizes predictable steady writes in this setup.

• Logs and time series: /var/log, Prometheus TSDB, and Loki chunks/indexes.
• Container metadata: Podman layers and compose definitions.
• Backups: a separate physical device from the primary data.

UI Daemon Consolidation Strategy

Linux runs the data engine; the Mac hosts management UIs.

Dagit, Grafana, Lightdash, and Trino Web UI moved to Podman or Docker on the Mac.

Benefits of This Architecture

• UI logs and configuration I/O move to the Mac SSD.
• Linux rootless podman runs fewer containers, reducing management work and I/O contention.
• UIs connect through gRPC, HTTP, or SQL, keeping the same user interface.

Conclusion and Future Outlook

Linux now focuses on computation and storage. Separating devices and hosts aims to reduce competition for limited resources.

Comparing the overhead and stability of Podman on Mac and Docker Desktop remains planned work.

Allocation Policy Summary