Quadlet operations on minimal Ubuntu

This setup uses Ubuntu 24.04 minimized with systemd and Quadlet for resident services. The infrastructure plan adds monitoring, logs, exposure control, audits, and backups to a small OS installation.

journalctl handles logs and systemctl handles startup and restart control.

Background and Design Intent

Manage containers as Linux services and include maintenance settings in the initial setup.

Loki, Prometheus, and mktxp are rootless; exporters are rootful. The related Podman Pods design likewise separates rootful GPU/network infrastructure from rootless UI/development workloads.

Base Layout Policy

Under /opt/containers, separate shared assets, builds, compose files, runtime mounts, and systemd definitions. This makes backup and permission targets explicit.

  /opt/containers/
├─ _shared/                # shared assets (certs, secrets, config)
├─ build/                  # Dockerfiles per app
├─ compose/                # docker/podman-compose (rootful/rootless)
├─ runtime/                # runtime configs and mount targets
│  ├─ loki/
│  ├─ prometheus/
│  └─ ...
└─ systemd/
   ├─ rootless/
   └─ rootful/
  

Deploy Quadlet to /home/ksh3/.config/containers/systemd as systemd user units.

Base configuration:

• OS: Ubuntu 24.04 (minimized)
• Management: systemd + Quadlet (/opt/containers/systemd/{rootless,rootful})
• Containers: rootless (Loki, Prometheus, mktxp) plus rootful (exporters)
• Storage layout centered on /opt/containers/runtime

The related Pods design uses podman play kube instead of units, with the same responsibility split.

Required Packages for a Minimized Install

Add the operational tools omitted by the minimized install.

Use ufw and fail2ban for exposure control, with lynis and chkrootkit for periodic checks.

Firewall and Protection

The proposed ufw policy permits only ports published by rootful containers. Rootless services use slirp4netns and are assumed not to be published directly.

  sudo apt install ufw
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 22/tcp comment "SSH"
sudo ufw allow 9090/tcp comment "Prometheus"
sudo ufw allow 3100/tcp comment "Loki"
sudo ufw allow 9633/tcp comment "smartctl exporter"
sudo ufw enable
sudo systemctl enable --now ufw
  

Keep firewall rules tied to the units owning published ports.

Security Auditing and Maintenance

Schedule lynis weekly and chkrootkit monthly.

Lynis (system-wide configuration audit)

  sudo apt install lynis
sudo lynis audit system
  

Weekly example in /etc/cron.weekly/lynis:

  /usr/sbin/lynis audit system --quiet --no-colors \
  --logfile /var/log/lynis-$(date +%F).log
  

Chkrootkit (monthly is enough)

  sudo apt install chkrootkit
sudo chkrootkit
  

Preserve audit logs so changes can be compared over time.

Virus and Malware Protection

Scan shared and incoming files with clamav and clamav-daemon. Use chkrootkit as a complementary intrusion check.

ClamAV (file scanning)

  sudo apt install clamav clamav-daemon
sudo systemctl enable --now clamav-freshclam
  

Scheduled scan example

/etc/cron.weekly/clamav-scan:

  #!/bin/bash
LOG=/var/log/clamav/scan_$(date +%F).log
clamscan -r --bell -i /home /opt /srv > "$LOG"
  

Combined with rootkit detection

  sudo chkrootkit
  

Scan /home, /opt, and /srv, including /opt/containers assets and backup locations.

Operations through journalctl and systemctl

Use journalctl and systemctl for Quadlet inspection rather than spreading checks across podman logs.

Pods align lifecycle by pod; Quadlet aligns it by systemd unit.

Compared with compose and run

The differences from compose / run are:

Autostart, restarts, and logs become part of Linux service management.

journald Tuning

Set storage limits for journald.

Example /etc/systemd/journald.conf:

  SystemMaxUse=1G
SystemMaxFileSize=100M
RuntimeMaxUse=512M
Storage=persistent
Compress=yes
  

Keep logs persistent but bounded. journald handles rotation and compression; these logs need no separate logrotate.

Backup Operations

Back up persistent data and shared assets separately from reproducible runtime definitions.

Podman volume backup

  podman run --rm -v loki-data:/data -v /srv/backup:/backup alpine \
  sh -lc 'tar czf /backup/loki-data_$(date +%F_%H%M).tar.gz -C /data .'
  

systemd definitions and _shared assets

  tar czf /srv/backup/containers_config_$(date +%F).tar.gz \
    -C /opt/containers systemd runtime/_shared
  

The first command captures volume data; the second captures systemd definitions and _shared assets.

Operational components

The four components are:

1. journald — centralized log handling
2. ufw + fail2ban — surface-level defense
3. clamav + chkrootkit — file and intrusion checks
4. lynis — scheduled configuration auditing (weekly)

The setup makes rootful/rootless boundaries, published ports, logs, and backup targets explicit.

Next Steps

• Inventory rootless/rootful service names, published ports, and backup targets
• Move cron.weekly audits and scans to systemd timer jobs
• Reflect responsibility in Quadlet names and directory layout