Running Resident Services with Quadlet on Minimal Ubuntu

When I use a minimized Ubuntu server as an always-on node, I do not want its operational model to emerge accidentally over time. This note captures a cleaner approach: start with Ubuntu 24.04 minimized, then define resident services around systemd and Quadlet from the beginning.

The goal was straightforward. Keep the node lightweight while still having a complete operational baseline for monitoring, logging, exposure control, auditing, and backup. With Quadlet, container startup becomes part of systemd, which means log inspection, restart policy, autostart, and failure analysis all move into the same control plane through journalctl and systemctl.

Background and Design Intent

The main concern was to avoid leaving containers as loosely managed processes. The second concern was to maintain a clear boundary between rootless and rootful workloads and never let that boundary blur.

In this setup, Loki, Prometheus, and mktxp are grouped on the rootless side, while the exporter set remains rootful. The related Podman Pods design follows the same logic: rootful for GPU, networking, and infrastructure services; rootless for UI and development workloads. That consistency makes the overall operating model easier to maintain and reason about.

Base Layout Policy

The first decision was to separate responsibilities under /opt/containers. If shared assets, build files, compose definitions, runtime mounts, and systemd definitions all end up mixed together, backup and permission decisions become inconsistent very quickly.

  /opt/containers/
├─ _shared/                # shared assets (certs, secrets, config)
├─ build/                  # Dockerfiles per app
├─ compose/                # docker/podman-compose (rootful/rootless)
├─ runtime/                # runtime configs and mount targets
│  ├─ loki/
│  ├─ prometheus/
│  └─ ...
└─ systemd/
   ├─ rootless/
   └─ rootful/
  

With this structure, it stays obvious which paths contain definitions, which hold persistent data, and which hold shared configuration. Quadlet is then deployed into /home/ksh3/.config/containers/systemd and runs as systemd user units.

Base assumptions for this setup:

• OS: Ubuntu 24.04 (minimized)
• Management: systemd + Quadlet (/opt/containers/systemd/{rootless,rootful})
• Containers: rootless (Loki, Prometheus, mktxp) plus rootful (exporters)
• Storage layout centered on /opt/containers/runtime

The related Podman Pods note uses podman play kube as a different execution unit, but the responsibility split is effectively the same. Whether the runtime is pod-based or unit-based, infrastructure-facing workloads fit naturally on the rootful side.

Required Packages for a Minimized Install

A minimized install stays lean, but it also omits many tools that become basic requirements during daily operations. Treating these as a deliberate package baseline rather than an afterthought keeps the node operational from the start.

This is more than a convenience list. Each group supports later parts of the operating model. ufw and fail2ban support exposure control, while lynis and chkrootkit make periodic auditing part of the base system rather than something remembered only after an incident.

Firewall and Protection

External exposure gets reduced first with ufw. The rule is explicit: only allow ports published by rootful containers. Rootless services running through slirp4netns are generally not reachable directly from outside, which simplifies the attack surface considerably.

  sudo apt install ufw
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 22/tcp comment "SSH"
sudo ufw allow 9090/tcp comment "Prometheus"
sudo ufw allow 3100/tcp comment "Loki"
sudo ufw allow 9633/tcp comment "smartctl exporter"
sudo ufw enable
sudo systemctl enable --now ufw
  

This split matters because it keeps exposure decisions tied to a small, well-defined set of services. Once it is decided which Quadlet units own published ports, the firewall rules stay understandable and the reasons for each rule stay clear.

Security Auditing and Maintenance

If the node is meant to stay online continuously, auditing should be part of the starting condition rather than an occasional manual task. lynis runs weekly; chkrootkit runs monthly.

Lynis (system-wide configuration audit)

  sudo apt install lynis
sudo lynis audit system
  

Weekly example in /etc/cron.weekly/lynis:

  /usr/sbin/lynis audit system --quiet --no-colors \
  --logfile /var/log/lynis-$(date +%F).log
  

Chkrootkit (monthly is enough)

  sudo apt install chkrootkit
sudo chkrootkit
  

The important part is not only which tools are used, but that they produce log artifacts that can be compared over time. A minimized server can look clean right after setup but become harder to observe after a few weeks. Consistent log output makes drift detectable.

Virus and Malware Protection

Even on an infrastructure node, file scanning still matters if the machine handles shared data or incoming artifacts. clamav is paired with clamav-daemon, and chkrootkit serves as a complementary intrusion-oriented check.

ClamAV (file scanning)

  sudo apt install clamav clamav-daemon
sudo systemctl enable --now clamav-freshclam
  

Scheduled scan example

/etc/cron.weekly/clamav-scan:

  #!/bin/bash
LOG=/var/log/clamav/scan_$(date +%F).log
clamscan -r --bell -i /home /opt /srv > "$LOG"
  

Combined with rootkit detection

  sudo chkrootkit
  

The scan paths /home, /opt, and /srv align with the directories that are likely to accumulate configuration, application data, and backup assets over time — the same directories structured under /opt/containers and backed up under /srv.

journalctl / systemd Operational Tips

Once Quadlet is in place, it makes more sense to center routine operations on journalctl and systemctl than to keep switching back to podman logs. A unified operational view is the goal.

This table works well as an operational cheat sheet. Whether services are managed as Podman Pods or as Quadlet units, the real value is reducing fragmentation in how service health gets observed. The state that needs to be checked should not be scattered across multiple tools.

Why Quadlet Helps in Practice

A direct comparison between older compose / run based workflows and a Quadlet-based setup shows what actually improves in daily operations.

The biggest operational gain is that autostart, restart behavior, and logs all become first-class parts of the host service manager. At that point, resident container workloads stop feeling like a special case and start behaving like normal Linux services.

journald Tuning

Since the logging model is built around journald, storage limits need to be defined early.

Example /etc/systemd/journald.conf:

  SystemMaxUse=1G
SystemMaxFileSize=100M
RuntimeMaxUse=512M
Storage=persistent
Compress=yes
  

The intent is to keep logs persistent while preventing unbounded growth. logrotate is not needed here because journald already handles rotation and compression. This fits the goal of centralizing operational behavior under systemd.

Backup Operations

Backing up data and backing up definitions are separate concerns. Container processes can often be recreated, while persistent data and shared assets are much harder to recover if lost.

Podman volume backup

  podman run --rm -v loki-data:/data -v /srv/backup:/backup alpine \
  sh -lc 'tar czf /backup/loki-data_$(date +%F_%H%M).tar.gz -C /data .'
  

systemd definitions and _shared assets

  tar czf /srv/backup/containers_config_$(date +%F).tar.gz \
    -C /opt/containers systemd runtime/_shared
  

The first command captures the contents of the volume itself. The second captures the service definitions and shared assets needed to reconstruct the environment. Keeping those concerns separate makes recovery planning much clearer.

Conclusion

This is a solid baseline for building a lightweight, static, and secure always-on node on top of minimal Ubuntu. The core operational pillars are:

1. journald — centralized log handling
2. ufw + fail2ban — surface-level defense
3. clamav + chkrootkit — file and intrusion checks
4. lynis — scheduled configuration auditing (weekly)

Whether services are grouped as pods or managed as Quadlet units, the same structure holds: clear rootful versus rootless boundaries, explicit exposed ports, centralized logs, and distinct backup targets. Ubuntu minimized stays lightweight while Quadlet, journald, UFW, security auditing, and backups are treated as one integrated baseline from the start.

Next Steps

• Turn the rootless versus rootful split into an explicit service inventory: service names, published ports, and backup targets written down together. This makes UFW rules and recovery procedures easier to keep in sync.
• Move the weekly audit and scanning routines from cron.weekly into systemd timer jobs. Since service management is already centered on systemd, keeping recurring maintenance in the same control plane makes the environment easier to inspect.
• Make Quadlet naming conventions and directory structure reflect service responsibility more explicitly. That will matter once the number of resident services grows.