Building a Container Platform with Rootless Podman and Quadlet: UID Mapping, Permission Design, and macOS DNS Resolution

Conclusion

Container operations for the local dev platform settled on different approaches per host.

The two Linux servers use rootless Podman with Quadlet for systemd service management. Mac was the exception — CLI podman machine could not resolve storage.home.arpa from inside containers, so Podman Desktop was adopted instead.

During construction, I worked through Quadlet verification on Podman 4.9.x, UID mapping mechanics, Loki permission resolution, and the operational boundary between podman unshare and sudo. This article records the full process.

Quadlet Verification: Does It Work on Podman 4.9.x?

Ubuntu 24.04’s standard package (Podman v4.9.3) does not include the podman quadlet subcommand. However, the Quadlet feature itself works fully through the systemd-generator mechanism.

How It Works

Place a .container file in the designated directory, and a .service is auto-generated.

  systemctl cat node-exporter.service
  

  # /run/systemd/generator/node-exporter.service
# Automatically generated by /usr/lib/systemd/system-generators/podman-system-generator
[Unit]
Description=Prometheus Node Exporter (Quadlet)
After=network-online.target
Wants=network-online.target
SourcePath=/etc/containers/systemd/node-exporter.container
...
  

Node Exporter Quadlet File Example

File path: /etc/containers/systemd/node-exporter.container

  [Unit]
Description=Prometheus Node Exporter (Quadlet)
After=network-online.target
Wants=network-online.target

[Container]
Image=docker.io/prom/node-exporter:v1.10.1
ContainerName=node-exporter
Network=host
ReadOnly=true
DropCapability=ALL
Volume=/:/host:ro,rslave
Args=--path.rootfs=/host

[Install]
WantedBy=multi-user.target
  

After placement:

  systemctl daemon-reload
systemctl start node-exporter.service
systemctl status node-exporter.service
  

The ENTRYPOINT Pitfall

The first attempt used Exec=/bin/node_exporter --path.rootfs=/host, which conflicted with the image’s built-in ENTRYPOINT. The prom/node-exporter image already defines /bin/node_exporter as its ENTRYPOINT. When passing arguments to images with an established ENTRYPOINT, use Args instead of Exec.

systemd Integration Notes

• Dynamically generated units under /run/systemd/generator cannot be enabled via systemctl enable
• The WantedBy=multi-user.target directive in the [Install] section controls auto-start on boot

Understanding Rootless UID Mapping

The Ownership Wall via subuid

Rootless Podman’s security rests on the fact that a host user (UID 1000) cannot access files owned by subordinate UIDs (100000+) used inside containers. This is not a bug — it is the isolation mechanism working as designed.

Files that appear as 100999:100999 on the host are completely inaccessible to user ksh3 (UID 1000). The standard way to handle them is podman unshare, which enters the user namespace as a virtual root.

Loki permission denied: A Case Study

When launching Grafana Loki (grafana/loki:3.5) as a rootless container, this error appeared:

  open /loki/index_cache/... .tsdb-tmp: permission denied
  

The host data directory /opt/containers/runtime/loki/data was owned by ksh3:1000 with mode 0755. Loki runs as the image-defined loki user (UID=10005). After rootless UID mapping, the container’s effective UID does not match the host-side owner, so write access is denied.

UID Mapping Flow

The behavior becomes clear when laid out case by case.

  Host: ksh3 (UID=1000)

Case A: no keep-id (image loki=10005)
  container root(0) -> host 100000 (subuid start example)
  container loki(10005) -> host 110005
  effective UID = 110005 != directory owner 1000 -> cannot write

Case B: keep-id + loki (the trap)
  container root(0) -> host 1000
  container loki(10005) -> host 110005
  effective UID = 110005 != 1000 -> cannot write

Case C: keep-id + loki + :U
  :U chowns host /data owner -> 110005
  effective UID = 110005 -> can write
  but host user 1000 is no longer owner -> hard to edit

Case D: keep-id + User=0 <- adopted
  container root(0) -> host 1000
  effective UID = 1000 -> owner match -> can write (host operability preserved)
  

Case B is the trap. Adding keep-id only changes how container root is mapped — it does not change who the process runs as. If the image runs as a non-root user, that user stays in the subordinate UID range.

Option Comparison

Recommended Patterns

1. Fast path / development → UserNS=keep-id + User=0
2. Preserve non-root execution → ACL-based approach (no UserNS=keep-id, no User=)

Loki Quadlet Configuration (User=0 + keep-id)

  [Unit]
Description=Grafana Loki (Quadlet - rootless)
After=network-online.target
Wants=network-online.target

[Container]
Image=docker.io/grafana/loki:3.5
ContainerName=loki
Exec=-config.file=/etc/loki/config.yaml
UserNS=keep-id
User=0
Network=host
PublishPort=3100:3100
Volume=/opt/containers/etc/loki:/etc/loki:ro
Volume=/opt/containers/runtime/loki/data:/loki:rw
Volume=/opt/containers/etc/loki/certs:/etc/certs:ro

[Service]
Restart=always
TimeoutStartSec=900s

[Install]
WantedBy=multi-user.target
  

  systemctl --user daemon-reload
systemctl --user restart loki.service
podman logs loki | grep -i permission || echo "permission errors none"
  

ACL-Based Approach (Keep Non-Root)

  LOKI_UID=$(podman run --rm docker.io/grafana/loki:3.5 id -u)
LOKI_GID=$(podman run --rm docker.io/grafana/loki:3.5 id -g)

setfacl -m u:${LOKI_UID}:rwx /opt/containers/runtime/loki/data
setfacl -R -m u:${LOKI_UID}:rwx /opt/containers/runtime/loki/data
setfacl -d -m u:${LOKI_UID}:rwx /opt/containers/runtime/loki/data

systemctl --user restart loki.service
podman logs loki | grep -i permission || echo "permission errors none"
  

Operational Model: Three Tiers of podman unshare and sudo

Rootless isolation is “ownership-based blocking” — sudo bypasses it at the kernel level. The key is treating this not as a security failure but as a well-defined exception path.

For pure read operations like backups, sudo rsync -a is often simpler and safer than podman unshare. When backing up MinIO metadata (.minio.sys) without risking corruption, preserving ownership via -a while reading from outside the namespace was the more reliable approach.

macOS DNS Issue: From CLI to Podman Desktop

On the Mac (Desktop PC), I initially used CLI podman machine. Grafana containers could not reach Prometheus and Loki on storage.home.arpa (the Storage Server).

Symptoms

• curl https://storage.home.arpa:9090 worked from the Mac host
• Inside the Grafana container: name resolution error or connection refused
• DNS logs: storage.home.arpa AAAA -> name does not exist

Cause: DNS Resolution Differences Between Desktop and CLI

CLI podman machine runs a fully independent VM that does not share the host’s DNS or hosts entries. Local domains like home.arpa that depend on host-side configuration are not resolvable from inside the container. Compose dns: settings also failed to propagate reliably because the VM generates its own /etc/resolv.conf.

Resolution and Decision

The Linux servers already had a well-established Quadlet + rootless model. Only the Mac had the DNS issue. While CLI’s isolation is more robust, the Mac serves as the development hub (Grafana UI, browser, VSCode) where reachability to internal LAN services is the top priority.

Podman Desktop was adopted for the Mac. Desktop transparently carries host DNS state into the VM, making storage.home.arpa directly resolvable from containers. An extra_hosts mapping was added as well to avoid depending on DNS ambiguity.

  services:
  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
      - NO_PROXY=localhost,127.0.0.1,10.0.0.0/8,*.home.arpa
    volumes:
      - /Users/ksh3/containers/iac/grafana/data:/var/lib/grafana
      - /Users/ksh3/containers/iac/grafana/provisioning:/etc/grafana/provisioning:ro
      - /Users/ksh3/containers/iac/grafana/certs:/etc/ssl:ro
    extra_hosts:
      - "storage.home.arpa:10.10.10.3"
    depends_on:
      - prometheus
  

Caveats

• Podman 4.9.x lacks the podman quadlet CLI but the systemd-generator mechanism works. Podman 5.x adds CLI support
• User=0 + keep-id is the fastest path for development but runs as root inside the container. For production-oriented setups, consider the ACL approach
• :U changes host-side ownership to the container UID, significantly reducing development-time operability
• If SELinux is enabled, consider adding :Z or :z to volumes
• On NFS / root_squash environments, :U and chown may fail — use ACLs or keep-id instead
• Podman Desktop is a convenient bridge, not a secure sandbox. Explicit name resolution (extra_hosts or internal DNS) is more reliable than relying on transparent host carry-over