Podman and Quadlet: UIDs and DNS

Quadlet on Linux, Podman Desktop on Mac

I selected container management per host for the local dev platform. Infrastructure setup also required matching UID behavior and name resolution.

Two Linux servers use rootless Podman and Quadlet under systemd. Mac uses Podman Desktop because CLI podman machine could not resolve storage.home.arpa inside containers.

The following sections cover Quadlet behavior, UID mapping, Loki permissions, and podman unshare versus sudo.

Quadlet Verification: Does It Work on Podman 4.9.x?

Ubuntu 24.04 ships Podman v4.9.3 without podman quadlet. Quadlet still worked through systemd-generator.

How It Works

A .container file in the expected directory generates a .service unit.

  systemctl cat node-exporter.service
  

  # /run/systemd/generator/node-exporter.service
# Automatically generated by /usr/lib/systemd/system-generators/podman-system-generator
[Unit]
Description=Prometheus Node Exporter (Quadlet)
After=network-online.target
Wants=network-online.target
SourcePath=/etc/containers/systemd/node-exporter.container
...
  

Node Exporter Quadlet File Example

File path: /etc/containers/systemd/node-exporter.container

  [Unit]
Description=Prometheus Node Exporter (Quadlet)
After=network-online.target
Wants=network-online.target

[Container]
Image=docker.io/prom/node-exporter:v1.10.1
ContainerName=node-exporter
Network=host
ReadOnly=true
DropCapability=ALL
Volume=/:/host:ro,rslave
Args=--path.rootfs=/host

[Install]
WantedBy=multi-user.target
  

After placement:

  systemctl daemon-reload
systemctl start node-exporter.service
systemctl status node-exporter.service
  

ENTRYPOINT and arguments

The Exec setting Exec=/bin/node_exporter --path.rootfs=/host failed because prom/node-exporter already sets /bin/node_exporter as ENTRYPOINT. Pass arguments through Args instead of repeating the command.

systemd Integration Notes

• Dynamically generated units under /run/systemd/generator cannot be enabled via systemctl enable
• The WantedBy=multi-user.target directive in the [Install] section controls auto-start on boot

Understanding Rootless UID Mapping

subuid and ownership

Rootless mapping separates the host user at UID 1000 from subordinate UIDs starting at 100000. Ordinary host permissions may not allow access to their files.

User ksh3 (UID 1000) cannot directly manipulate 100999:100999 files. podman unshare handles them as virtual root in the user namespace.

Loki permission denied: A Case Study

Rootless Grafana Loki (grafana/loki:3.5) produced this error:

  open /loki/index_cache/... .tsdb-tmp: permission denied
  

/opt/containers/runtime/loki/data was owned by ksh3:1000 with mode 0755. The image runs as loki at UID=10005, whose mapped host UID does not match the directory owner.

UID Mapping Flow

UID mappings by case:

  Host: ksh3 (UID=1000)

Case A: no keep-id (image loki=10005)
  container root(0) -> host 100000 (subuid start example)
  container loki(10005) -> host 110005
  effective UID = 110005 != directory owner 1000 -> cannot write

Case B: keep-id + loki (the trap)
  container root(0) -> host 1000
  container loki(10005) -> host 110005
  effective UID = 110005 != 1000 -> cannot write

Case C: keep-id + loki + :U
  :U chowns host /data owner -> 110005
  effective UID = 110005 -> can write
  but host user 1000 is no longer owner -> hard to edit

Case D: keep-id + User=0 <- adopted
  container root(0) -> host 1000
  effective UID = 1000 -> owner match -> can write (host operability preserved)
  

In Case B, keep-id leaves the non-root Loki user in the subordinate range. Root mapping and process user selection are separate.

Option Comparison

Recommended Patterns

1. Fast path / development → UserNS=keep-id + User=0
2. Preserve non-root execution → ACL-based approach (no UserNS=keep-id, no User=)

Loki Quadlet Configuration (User=0 + keep-id)

  [Unit]
Description=Grafana Loki (Quadlet - rootless)
After=network-online.target
Wants=network-online.target

[Container]
Image=docker.io/grafana/loki:3.5
ContainerName=loki
Exec=-config.file=/etc/loki/config.yaml
UserNS=keep-id
User=0
Network=host
PublishPort=3100:3100
Volume=/opt/containers/etc/loki:/etc/loki:ro
Volume=/opt/containers/runtime/loki/data:/loki:rw
Volume=/opt/containers/etc/loki/certs:/etc/certs:ro

[Service]
Restart=always
TimeoutStartSec=900s

[Install]
WantedBy=multi-user.target
  

  systemctl --user daemon-reload
systemctl --user restart loki.service
podman logs loki | grep -i permission || echo "permission errors none"
  

ACL-Based Approach (Keep Non-Root)

  LOKI_UID=$(podman run --rm docker.io/grafana/loki:3.5 id -u)
LOKI_GID=$(podman run --rm docker.io/grafana/loki:3.5 id -g)

setfacl -m u:${LOKI_UID}:rwx /opt/containers/runtime/loki/data
setfacl -R -m u:${LOKI_UID}:rwx /opt/containers/runtime/loki/data
setfacl -d -m u:${LOKI_UID}:rwx /opt/containers/runtime/loki/data

systemctl --user restart loki.service
podman logs loki | grep -i permission || echo "permission errors none"
  

Operational Model: Three Tiers of podman unshare and sudo

sudo can bypass ownership restrictions. Treat it as an administrative path separate from normal rootless operation.

For read-only backups, sudo rsync -a can be simpler than podman unshare. I copied MinIO .minio.sys using -a to preserve ownership.

macOS DNS Issue: From CLI to Podman Desktop

With CLI podman machine, Mac Grafana could not reach Prometheus or Loki on storage.home.arpa.

Symptoms

• curl https://storage.home.arpa:9090 worked from the Mac host
• Inside the Grafana container: name resolution error or connection refused
• DNS logs: storage.home.arpa AAAA -> name does not exist

Cause: DNS Resolution Differences Between Desktop and CLI

This VM did not share host DNS/hosts, so it could not resolve home.arpa. Compose dns: also failed to propagate because the VM generated /etc/resolv.conf.

Resolution and Decision

Linux retained Quadlet + rootless. Mac was the development hub for Grafana UI, browser, and VSCode, so internal LAN access took priority.

Podman Desktop passed host resolution into the VM and made storage.home.arpa reachable. I also added explicit extra_hosts mappings.

  services:
  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
      - NO_PROXY=localhost,127.0.0.1,10.0.0.0/8,*.home.arpa
    volumes:
      - /Users/ksh3/containers/iac/grafana/data:/var/lib/grafana
      - /Users/ksh3/containers/iac/grafana/provisioning:/etc/grafana/provisioning:ro
      - /Users/ksh3/containers/iac/grafana/certs:/etc/ssl:ro
    extra_hosts:
      - "storage.home.arpa:10.10.10.3"
    depends_on:
      - prometheus
  

Caveats

• Podman 4.9.x lacks the podman quadlet CLI but the systemd-generator mechanism works. Podman 5.x adds CLI support
• User=0 + keep-id is the fastest path for development but runs as root inside the container. For production-oriented setups, consider the ACL approach
• :U changes host-side ownership to the container UID, significantly reducing development-time operability
• If SELinux is enabled, consider adding :Z or :z to volumes
• On NFS / root_squash environments, :U and chown may fail — use ACLs or keep-id instead
• Podman Desktop is a convenient bridge, not a secure sandbox. Explicit name resolution (extra_hosts or internal DNS) is more reliable than relying on transparent host carry-over