On this page

Monitoring with Prometheus, Loki, and Quadlet

Metrics and logs collected on a Mac mini for 3 hosts and RouterOS. An infrastructure setup using Prometheus, Loki, and Quadlet, with configuration and migration notes.

These articles use AI-generated summaries of Obsidian notes originally kept as technical memos.

English translations are produced with AI assistance.

warning

**March 2026 update**: Promtail reached EOL (LTS end) on March 2, 2026. Migration to [Vector](https://vector.dev/) (Datadog Telemetry) is currently in progress. For new deployments, Vector or [Grafana Alloy](https://grafana.com/docs/alloy/latest/) is recommended.

Centralized monitoring

The local dev platform collects metrics and logs on the Storage Server (Mac mini, Ubuntu 24.04). It runs 24/7, including when Compute Server and Desktop PC are off.

We moved from compose + systemd templates to Quadlet. .container files generate systemd units, with rootful and rootless scopes selected by access needs.

Collection Target	Path	Storage
Host metrics (3 hosts)	node_exporter → Prometheus	Storage Server
Disk health (2 hosts)	smartctl_exporter → Prometheus	Storage Server
RouterOS metrics	MKTXP (API) → Prometheus	Storage Server
MinIO metrics	/minio/v2/metrics/cluster → Prometheus	Storage Server
RouterOS / Netgear syslog	UDP 1514 (rfc3164) → Promtail → Loki	Storage Server
Linux logs	journal / files → Promtail → Loki	Storage Server
Visualization	Grafana (Desktop PC) → Prometheus + Loki	Desktop PC

Hardware and Storage

Item	Details
Chassis	Mac mini late 2018 (x86)
CPU	Core i3
RAM	DDR4 SO-DIMM 8GB
Storage	SSD 2TB (OS, LVM), ext M.2 SSD 4TB (LUKS → `/mnt/data`)
NIC	1GbE + ext 10GbE (RTL8159, 10.10.10.3)

  nvme0n1 (1.8T)  → LVM → / (100G)
nvme1n1 (3.6T)  → LUKS → /mnt/data (archive, backups, object storage)

OS is Ubuntu 24.04.3 LTS. NTP and DNS point to the router, timezone is UTC.

Quadlet Service Layout

The initial setup used dedicated prometheus, loki, and promtail users with podman-compose@.service. Quadlet later replaced it, separating rootful and rootless services.

  /opt/containers/systemd/
├── rootful/
│   ├── node-exporter.container
│   ├── promtail.container
│   └── smartctl-exporter.container
└── rootless/
    ├── loki.container
    ├── minio.container
    ├── mktxp.container
    └── prometheus.container

Rootful / Rootless Split Criteria

Scope	Services	Reason
rootful	node-exporter, promtail, smartctl	Need access to host `/`, `/var/log/journal`, or raw disk devices
rootless	prometheus, loki, mktxp, minio	Network-only data collection and storage. No direct host device access needed

node_exporter requires Network=host and Volume=/:/host:ro,rslave for full host metrics. Promtail needs /var/log/journal access and UDP 1514 listening. smartctl_exporter needs raw disk device access.

For Quadlet verification details and UID mapping, see Building a Container Platform with Rootless Podman and Quadlet.

Prometheus Configuration

Seven jobs covering 3 hosts, RouterOS, and MinIO.

  global:
  scrape_interval: 30s
  evaluation_interval: 30s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets:
          - storage.home.arpa:9090

  - job_name: node-exporter
    static_configs:
      - targets:
          - storage.home.arpa:9100
          - desktop.home.arpa:9100
          - compute.home.arpa:9100

  - job_name: loki
    static_configs:
      - targets:
          - storage.home.arpa:3100

  - job_name: promtail
    static_configs:
      - targets:
          - storage.home.arpa:9080

  - job_name: smartctl
    scrape_interval: 5m
    static_configs:
      - targets:
          - storage.home.arpa:9633
          - compute.home.arpa:9633

  - job_name: mikrotik
    static_configs:
      - targets:
          - storage.home.arpa:49090

  - job_name: minio
    metrics_path: /minio/v2/metrics/cluster
    static_configs:
      - targets:
          - storage.home.arpa:9000

Port Map

Port	Service	Host(s)
9090	Prometheus	storage
9100	node_exporter	storage, desktop, compute
3100	Loki	storage
9080	Promtail	storage
9633	smartctl_exporter	storage, compute
49090	MKTXP (RouterOS)	storage
9000	MinIO	storage

node_exporter Across 3 Hosts

Storage Server and Compute Server run node_exporter through rootful Quadlet. The Mac uses Homebrew with a listen address configured for remote scraping.

  # Mac (Homebrew)
brew install node_exporter
sudo brew services start node_exporter
# Change listen address for external scraping
sudo sed -i '' 's/--web.listen-address=127\.0\.0\.1:9100/--web.listen-address=0.0.0.0:9100/' \
  /Library/LaunchDaemons/homebrew.mxcl.node_exporter.plist

Promtail Configuration (Log Collection)

Rootful Promtail collects Linux logs and network syslog, then sends them to Loki.

  server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /promtail/positions.yaml

clients:
  - url: http://storage.home.arpa:3100/loki/api/v1/push

scrape_configs:
  # --- Linux file logs ---
  - job_name: security-files
    static_configs:
      - labels:
          job: security
          host: storage.home.arpa
          __path__: /var/log/alternatives.log*
      - labels:
          job: security
          host: storage.home.arpa
          __path__: /var/log/apport.log*

  - job_name: apt-files
    static_configs:
      - labels:
          job: apt
          host: storage.home.arpa
          __path__: /var/log/apt/history.log
      - labels:
          job: apt
          host: storage.home.arpa
          __path__: /var/log/apt/term.log
      - labels:
          job: apt
          host: storage.home.arpa
          __path__: /var/log/dpkg.log

  # --- systemd journal ---
  - job_name: kernel-journal
    journal:
      path: /var/log/journal
      max_age: 168h
      labels:
        job: kernel
        host: storage.home.arpa
    relabel_configs:
      - source_labels: ["__journal__hostname"]
        target_label: host
      - source_labels: ["__journal__syslog_identifier"]
        target_label: ident
      - source_labels: ["__journal_priority"]
        target_label: priority

  - job_name: systemd-units
    journal:
      path: /var/log/journal
      max_age: 168h
      labels:
        job: systemd
        host: storage.home.arpa
    relabel_configs:
      - source_labels: ["__journal__systemd_unit"]
        target_label: unit
      - source_labels: ["__journal__hostname"]
        target_label: host
      - source_labels: ["__journal__uid"]
        target_label: uid

  # --- Network device syslog (RouterOS / Netgear WiFi) ---
  - job_name: network-syslog
    syslog:
      listen_address: 0.0.0.0:1514
      listen_protocol: udp
      syslog_format: rfc3164
      use_incoming_timestamp: false
      label_structured_data: true
    relabel_configs:
      - source_labels: ["__syslog_message_hostname"]
        target_label: host
      - source_labels: ["__syslog_message_app_name"]
        target_label: app
      - source_labels: ["__syslog_severity"]
        target_label: severity
      - source_labels: ["__syslog_facility"]
        target_label: facility

Network Device Syslog Reception

RouterOS and Netgear WiFi AP send syslog over UDP 1514 in rfc3164 format.

RouterOS configuration:

  /system logging action add name=promtail target=remote remote=10.10.10.3 remote-port=1514 bsd-syslog=yes
/system logging add topics=firewall action=promtail
/system logging add topics=system,info action=promtail

Bulk firewall logging:

  /ip firewall filter set [find builtin=no && dynamic=no] log=yes log-prefix="FW: "

MKTXP: RouterOS API Metrics

We chose API-based MKTXP for RouterOS. Connectivity was easier to check, and it avoided SNMP Exporter network-mode issues.

  [edge.home.arpa]
hostname = edge.home.arpa
username = metrics
password = (API user password)
port = 8728
use_ssl = False
verify_ssl = False
timeout = 5
allow_duplicates = False

http://storage.home.arpa:49090/metrics exposes CPU, memory, temperature, interface, and DHCP data. Grafana displays it in the Mikrotik MKTXP Exporter dashboard.

Grafana Integration (Desktop PC)

Grafana on the Mac Studio reads the Storage Server. This infrastructure setup separates collection and storage from visualization.

Data Sources

Prometheus → http://storage.home.arpa:9090
Loki → http://storage.home.arpa:3100

RouterOS Log Table Panel

Item	Setting
Data source	Loki
Query	`{job="network-syslog"}`
Transformations	Extract fields → Organize fields
Columns kept	`Time`, `host`, `app`, `severity`, `line`

LogQL query example:

  {job="network-syslog"} | line_format "{{.time}} | {{.host}} | {{.severity}} | {{.line}}"

Grafana shows RouterOS load spikes beside firewall logs from the same time.

Vector Migration (In Progress)

Promtail reached EOL in March 2026, prompting migration to Vector (Datadog Telemetry). Vector handles log collection, metric transformation, and routing.

The Go + NATS + Dagster platform has a design for Vector to consume TELEMETRY (telemetry.>) and forward it to Prometheus and Loki. Promtail’s work is planned to move there.

Caveats

Verify CRS304 L3 / filter rules do not block monitoring ports (9100, 9633, etc.)
Mac’s node_exporter defaults to localhost bind — plist modification needed for external scraping
Restrict monitoring ports to internal segments only
smartctl_exporter at 5-minute intervals is sufficient; over-scraping adds disk load
Promtail’s journal collection requires /var/log/journal access, which necessitates rootful Quadlet

Podman and Quadlet: UIDs and DNS

A development infrastructure …

smartctl-exporter and rootful / rootless operating rules

An NVMe workaround for …