On this page

Building the Storage Server's Always-On Monitoring Stack: Prometheus, Loki, Promtail, and Quadlet

Complete record of building a Mac mini (Ubuntu 24.04) as an always-on monitoring node running Prometheus, Loki, Promtail, and MKTXP under Quadlet with rootful/rootless separation. Covers RouterOS/Netgear syslog collection, node_exporter deployment across 3 hosts, and Grafana integration.

These articles use AI-generated summaries of Obsidian notes originally kept as technical memos.

English translations are produced with AI assistance.

warning

**March 2026 update**: Promtail reached EOL (LTS end) on March 2, 2026. Migration to [Vector](https://vector.dev/) (Datadog Telemetry) is currently in progress. For new deployments, Vector or [Grafana Alloy](https://grafana.com/docs/alloy/latest/) is recommended.

Conclusion

The monitoring layer for the local dev platform is centralized on the Storage Server (Mac mini, Ubuntu 24.04), which runs 24/7. Metrics collection and log aggregation continue on the Storage Server alone even when Compute Server and Desktop PC are powered off.

Service management evolved from the initial compose + systemd template approach to Quadlet (.container → systemd auto-generation), with rootful and rootless scopes separated by access requirements.

Collection Target	Path	Storage
Host metrics (3 hosts)	node_exporter → Prometheus	Storage Server
Disk health (2 hosts)	smartctl_exporter → Prometheus	Storage Server
RouterOS metrics	MKTXP (API) → Prometheus	Storage Server
MinIO metrics	/minio/v2/metrics/cluster → Prometheus	Storage Server
RouterOS / Netgear syslog	UDP 1514 (rfc3164) → Promtail → Loki	Storage Server
Linux logs	journal / files → Promtail → Loki	Storage Server
Visualization	Grafana (Desktop PC) → Prometheus + Loki	Desktop PC

Hardware and Storage

Item	Details
Chassis	Mac mini late 2018 (x86)
CPU	Core i3
RAM	DDR4 SO-DIMM 8GB
Storage	SSD 2TB (OS, LVM), ext M.2 SSD 4TB (LUKS → `/mnt/data`)
NIC	1GbE + ext 10GbE (RTL8159, 10.10.10.3)

  nvme0n1 (1.8T)  → LVM → / (100G)
nvme1n1 (3.6T)  → LUKS → /mnt/data (archive, backups, object storage)

OS is Ubuntu 24.04.3 LTS. NTP and DNS point to the router, timezone is UTC.

Quadlet Service Layout

The initial deployment used compose files with a [email protected] systemd template, with dedicated users (prometheus, loki, promtail) for rootless operation. The current setup uses Quadlet with rootful/rootless scope separation.

  /opt/containers/systemd/
├── rootful/
│   ├── node-exporter.container
│   ├── promtail.container
│   └── smartctl-exporter.container
└── rootless/
    ├── loki.container
    ├── minio.container
    ├── mktxp.container
    └── prometheus.container

Rootful / Rootless Split Criteria

Scope	Services	Reason
rootful	node-exporter, promtail, smartctl	Need access to host `/`, `/var/log/journal`, or raw disk devices
rootless	prometheus, loki, mktxp, minio	Network-only data collection and storage. No direct host device access needed

node_exporter requires Network=host and Volume=/:/host:ro,rslave for full host metrics. Promtail needs /var/log/journal access and UDP 1514 listening. smartctl_exporter needs raw disk device access.

For Quadlet verification details and UID mapping, see Building a Container Platform with Rootless Podman and Quadlet.

Prometheus Configuration

Seven jobs covering 3 hosts, RouterOS, and MinIO.

  global:
  scrape_interval: 30s
  evaluation_interval: 30s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets:
          - storage.home.arpa:9090

  - job_name: node-exporter
    static_configs:
      - targets:
          - storage.home.arpa:9100
          - desktop.home.arpa:9100
          - compute.home.arpa:9100

  - job_name: loki
    static_configs:
      - targets:
          - storage.home.arpa:3100

  - job_name: promtail
    static_configs:
      - targets:
          - storage.home.arpa:9080

  - job_name: smartctl
    scrape_interval: 5m
    static_configs:
      - targets:
          - storage.home.arpa:9633
          - compute.home.arpa:9633

  - job_name: mikrotik
    static_configs:
      - targets:
          - storage.home.arpa:49090

  - job_name: minio
    metrics_path: /minio/v2/metrics/cluster
    static_configs:
      - targets:
          - storage.home.arpa:9000

Port Map

Port	Service	Host(s)
9090	Prometheus	storage
9100	node_exporter	storage, desktop, compute
3100	Loki	storage
9080	Promtail	storage
9633	smartctl_exporter	storage, compute
49090	MKTXP (RouterOS)	storage
9000	MinIO	storage

node_exporter Across 3 Hosts

Storage Server and Compute Server run node_exporter via Quadlet (rootful). Desktop PC (Mac) uses Homebrew with an explicit listen address change for remote scraping.

  # Mac (Homebrew)
brew install node_exporter
sudo brew services start node_exporter
# Change listen address for external scraping
sudo sed -i '' 's/--web.listen-address=127\.0\.0\.1:9100/--web.listen-address=0.0.0.0:9100/' \
  /Library/LaunchDaemons/homebrew.mxcl.node_exporter.plist

Promtail Configuration (Log Collection)

Promtail runs as a rootful Quadlet service and collects both Linux logs and network device syslog, pushing everything to Loki.

  server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /promtail/positions.yaml

clients:
  - url: http://storage.home.arpa:3100/loki/api/v1/push

scrape_configs:
  # --- Linux file logs ---
  - job_name: security-files
    static_configs:
      - labels:
          job: security
          host: storage.home.arpa
          __path__: /var/log/alternatives.log*
      - labels:
          job: security
          host: storage.home.arpa
          __path__: /var/log/apport.log*

  - job_name: apt-files
    static_configs:
      - labels:
          job: apt
          host: storage.home.arpa
          __path__: /var/log/apt/history.log
      - labels:
          job: apt
          host: storage.home.arpa
          __path__: /var/log/apt/term.log
      - labels:
          job: apt
          host: storage.home.arpa
          __path__: /var/log/dpkg.log

  # --- systemd journal ---
  - job_name: kernel-journal
    journal:
      path: /var/log/journal
      max_age: 168h
      labels:
        job: kernel
        host: storage.home.arpa
    relabel_configs:
      - source_labels: ["__journal__hostname"]
        target_label: host
      - source_labels: ["__journal__syslog_identifier"]
        target_label: ident
      - source_labels: ["__journal_priority"]
        target_label: priority

  - job_name: systemd-units
    journal:
      path: /var/log/journal
      max_age: 168h
      labels:
        job: systemd
        host: storage.home.arpa
    relabel_configs:
      - source_labels: ["__journal__systemd_unit"]
        target_label: unit
      - source_labels: ["__journal__hostname"]
        target_label: host
      - source_labels: ["__journal__uid"]
        target_label: uid

  # --- Network device syslog (RouterOS / Netgear WiFi) ---
  - job_name: network-syslog
    syslog:
      listen_address: 0.0.0.0:1514
      listen_protocol: udp
      syslog_format: rfc3164
      use_incoming_timestamp: false
      label_structured_data: true
    relabel_configs:
      - source_labels: ["__syslog_message_hostname"]
        target_label: host
      - source_labels: ["__syslog_message_app_name"]
        target_label: app
      - source_labels: ["__syslog_severity"]
        target_label: severity
      - source_labels: ["__syslog_facility"]
        target_label: facility

Network Device Syslog Reception

RouterOS and Netgear WiFi AP send syslog over UDP 1514 in rfc3164 format.

RouterOS configuration:

  /system logging action add name=promtail target=remote remote=10.10.10.3 remote-port=1514 bsd-syslog=yes
/system logging add topics=firewall action=promtail
/system logging add topics=system,info action=promtail

Bulk firewall logging:

  /ip firewall filter set [find builtin=no && dynamic=no] log=yes log-prefix="FW: "

MKTXP: RouterOS API Metrics

RouterOS metrics use MKTXP Exporter (API-based) instead of SNMP. The API path is simpler to validate and avoids SNMP Exporter network-mode complications.

  [edge.home.arpa]
hostname = edge.home.arpa
username = metrics
password = (API user password)
port = 8728
use_ssl = False
verify_ssl = False
timeout = 5
allow_duplicates = False

Metrics are exposed at http://storage.home.arpa:49090/metrics — CPU, memory, temperature, interface stats, and DHCP data. Grafana visualizes these using the “Mikrotik MKTXP Exporter” dashboard.

Grafana Integration (Desktop PC)

Visualization runs on the Desktop PC (Mac Studio) via Grafana, pointing to the Storage Server’s data sources. This separates collection/storage from visualization.

Data Sources

Prometheus → http://storage.home.arpa:9090
Loki → http://storage.home.arpa:3100

RouterOS Log Table Panel

Item	Setting
Data source	Loki
Query	`{job="network-syslog"}`
Transformations	Extract fields → Organize fields
Columns kept	`Time`, `host`, `app`, `severity`, `line`

LogQL query example:

  {job="network-syslog"} | line_format "{{.time}} | {{.host}} | {{.severity}} | {{.line}}"

Having metrics and logs in the same Grafana makes it possible to correlate a RouterOS load spike with firewall events at the same timestamp.

Vector Migration (In Progress)

Promtail reached EOL in March 2026, so migration to Vector (Datadog Telemetry) is underway. Vector is Rust-based and handles log collection, metric transformation, and routing in a single binary.

Within the Go + NATS + Dagster AI orchestration platform, the TELEMETRY stream (telemetry.>) is already designed to be consumed by Vector and forwarded to Prometheus and Loki. Promtail’s responsibilities will transfer directly to Vector.

Caveats

Verify CRS304 L3 / filter rules do not block monitoring ports (9100, 9633, etc.)
Mac’s node_exporter defaults to localhost bind — plist modification needed for external scraping
Restrict monitoring ports to internal segments only
smartctl_exporter at 5-minute intervals is sufficient; over-scraping adds disk load
Promtail’s journal collection requires /var/log/journal access, which necessitates rootful Quadlet

Building a Container Platform with Rootless Podman and Quadlet: UID Mapping, Permission Design, and macOS DNS Resolution

Complete record of building a …

How I Stabilized smartctl-exporter and Standardized Exporter Operations Across Rootful and Rootless Scopes

Establishing a consistent …