💫 Podman as a service

Introduction

It’s nice to run everything on k8s, but as Yaakov was underling it in his blog

My personal experience on Azure Kubernetes Service was that I immediately lost a massive chunk of RAM to their Kubernetes implementation, and it used about 7–10% idle CPU on worker nodes.
Even with a single-instance MicroK8s on a small VPS, I observed an idle CPU load hovering around 12% on a 2× vCPU x86_64 box.
K3s, which is supposed to be leaner, still shows about 6% constant CPU consumption on a 2× vCPU Ampere A1 machine.
— Yaakov Blog, Feb 04, 2024

Podman brings several important advantages:

It can run rootless, which improves security.
It integrates well with systemd, making containers easy to manage as standard services.
It offers useful features such as auto-update with
--label "io.containers.autoupdate=registry".

Instead of running containers manually, we let systemd:

Start containers at boot
Restart them on failure
Stop them cleanly
Track logs and status
Seamless integration with monitoring tools

The basic example

Here, we run a container then init the systemd config file from it:

Let’s start a container normally:

1podman run -d \
2  --name my-nginx \
3  -p 8080:80 \
4  nginx:latest

Podman can generate a systemd unit automatically:

1podman generate systemd my-nginx

Here is more or less what you should get:

 1[Unit]
 2Description=Podman container-my-nginx.service
 3After=network.target
 4
 5[Service]
 6Restart=on-failure
 7ExecStart=/usr/bin/podman start my-nginx
 8ExecStop=/usr/bin/podman stop -t 10 my-nginx
 9ExecStopPost=/usr/bin/podman rm -f my-nginx
10
11[Install]
12WantedBy=multi-user.target

Check it:

1systemctl --user status container-my-nginx
2systemctl --user restart container-my-nginx
3journalctl --user -u container-my-nginx

One interesting example… Gitea

A few important notes:

If you run podman as a user, use the rootless container image.
In this example: docker.io/gitea/gitea:1-rootless

We let systemd download and run the container for us.

Do NOT use Type=forking with podman run, as Podman does not fork in the way systemd expects.

The dependency chain is as follows:

1network-online.target
2  |→ podman-network-gitea-net.service
3      |→ container-gitea-db.service
4            |→ container-gitea-app.service

Create network systemd unit

 1# /etc/systemd/system/podman-network-gitea-net.service
 2[Unit]
 3Description=Podman network gitea-net
 4Wants=network-online.target
 5After=network-online.target
 6
 7[Service]
 8Type=oneshot
 9RemainAfterExit=yes
10
11# Create network
12ExecStart=/usr/bin/podman network create gitea-net
13
14# Optional: remove network on stop (only if you really want ownership)
15ExecStop=/usr/bin/podman network rm gitea-net
16
17[Install]
18WantedBy=multi-user.target

This is the manual equivalent approach, but be careful:
if you create the network manually, the podman-network-gitea-net.service may fail due to a conflict.
This service is a dependency for the following units.

 1# Create network manually
 2podman network create gitea-net
 3
 4# Check
 5podman network ls
 6NETWORK ID    NAME        DRIVER
 7546e4f544220  gitea-net   bridge
 82f259bab93aa  podman      bridge
 9
10# in case, you need to reset
11sudo systemctl stop podman-network-gitea-net.service
12sudo systemctl disable podman-network-gitea-net.service
13sudo systemctl reset-failed podman-network-gitea-net.service
14sudo systemctl daemon-reload
15sudo systemctl enable podman-network-gitea-net.service
16sudo systemctl start podman-network-gitea-net.service
17sudo systemctl status podman-network-gitea-net.service

Create the MariaDB service

 1# /etc/systemd/system/container-gitea-db.service
 2[Unit]
 3Description=Gitea Database (MariaDB, Podman)
 4Wants=network-online.target
 5After=network-online.target
 6RequiresMountsFor=/var/lib/containers/storage
 7
 8# If you manage the network via systemd
 9Requires=podman-network-gitea-net.service
10After=podman-network-gitea-net.service
11
12[Service]
13Environment=PODMAN_SYSTEMD_UNIT=%n
14Restart=on-failure
15RestartSec=5
16TimeoutStartSec=120
17TimeoutStopSec=70
18
19Type=notify
20NotifyAccess=all
21
22# Ensure network exists (safe if already created or not)
23ExecStartPre=/bin/sh -c '/usr/bin/podman network exists gitea-net || /usr/bin/podman network create gitea-net'
24
25ExecStart=/usr/bin/podman run \
26  --name gitea-db \
27  --replace \
28  --detach \
29  --sdnotify=conmon \
30  --network gitea-net \
31  --env MYSQL_ROOT_PASSWORD=strong-root-password \
32  --env MYSQL_DATABASE=gitea \
33  --env MYSQL_USER=gitea \
34  --env MYSQL_PASSWORD=password \
35  --volume gitea-db-volume:/var/lib/mysql:Z \
36  --health-cmd='mysqladmin ping -h 127.0.0.1 --silent' \
37  --health-interval=5s \
38  --health-retries=10 \
39  docker.io/library/mariadb:11
40
41ExecStop=/usr/bin/podman stop \
42  --ignore \
43  --time 10 \
44  gitea-db
45
46ExecStopPost=/usr/bin/podman rm \
47  --ignore \
48  gitea-db
49
50[Install]
51WantedBy=multi-user.target

Create gitea systemd unit

 1# /etc/systemd/system/container-gitea-app.service
 2[Unit]
 3Description=Gitea (Podman container)
 4Wants=network-online.target
 5After=network-online.target
 6RequiresMountsFor=/var/lib/containers/storage
 7
 8# If you manage the db via systemd
 9Requires=container-gitea-db.service
10After=container-gitea-db.service
11
12[Service]
13Environment=PODMAN_SYSTEMD_UNIT=%n
14Restart=on-failure
15RestartSec=5
16TimeoutStartSec=120
17TimeoutStopSec=70
18
19Type=notify
20NotifyAccess=all
21
22# Ensure db exists and healthy
23ExecStartPre=/bin/sh -c 'until podman inspect --format "{{.State.Health.Status}}" gitea-db | grep -q healthy; do sleep 2; done'
24
25ExecStart=/usr/bin/podman run \
26  --name gitea-app \
27  --replace \
28  --detach \
29  --sdnotify=conmon \
30  --env DB_TYPE=mysql \
31  --env DB_HOST=gitea-db:3306 \
32  --env DB_NAME=gitea \
33  --env DB_USER=gitea \
34  --env DB_PASSWD=password \
35  --volume gitea-data-volume:/var/lib/gitea:Z \
36  --volume gitea-config-volume:/etc/gitea:Z \
37  --network gitea-net \
38  --publish 2222:2222 \
39  --publish 3000:3000 \
40  --label io.containers.autoupdate=registry \
41  docker.io/gitea/gitea:1-rootless
42
43ExecStop=/usr/bin/podman stop \
44  --ignore \
45  --time 10 \
46  gitea-app
47
48ExecStopPost=/usr/bin/podman rm \
49  --ignore \
50  gitea-app
51
52[Install]
53WantedBy=multi-user.target

The data and configuration are stored in /var/lib/containers/storage/volumes/
as persistent volumes: gitea-config-volume and gitea-data-volume.

Run the containers

Services run as root:

 1# Re-read and enable systemd service 
 2sudo systemctl daemon-reload
 3sudo systemctl enable podman-network-gitea-net.service
 4sudo systemctl enable container-gitea-db.service
 5
 6# Enable and start Gitea service
 7sudo systemctl enable --now container-gitea-app
 8
 9# Checks
10sudo podman ps
11systemctl status container-gitea-app.service
12journalctl -u container-gitea-app.service -b
13journalctl -u container-gitea-app.service -o cat

Important: services running in a user session are created in
$HOME/.config/containers/systemd/.

and executed with:

 1# Re-read and enable systemd service 
 2systemctl --user daemon-reload
 3systemctl --user enable podman-network-gitea-net.service
 4systemctl --user enable container-gitea-db.service
 5
 6# Enable and start Gitea service
 7systemctl --user enable --now container-gitea-app
 8
 9# Checkings
10podman ps
11systemctl status container-gitea-app.service
12journalctl -u container-gitea-app.service -b
13
14systemctl list-unit-files | grep gitea
15container-gitea-app.service                                                   enabled         enabled
16container-gitea-db.service                                                    enabled         enabled
17podman-network-gitea-net.service                                              enabled         enabled

Why not Docker Compose or Kubernetes?

Why not docker-compose, which does roughly the same thing?

Because services come with important advantages. docker-compose is not a service: it does not start at boot, it is not monitored by the operating system, and it does not integrate directly with systemd. It is excellent for development and testing, but less suitable for production.

Why not Kubernetes manifests, which do the same thing — and more?

Because even the most basic single-node k3s setup consumes resources. In cloud environments, everything is billable, and Kubernetes also introduces additional complexity that may be unnecessary for small or simple deployments.

In conclusion, running containers as first-class systemd services with Podman offers a pragmatic alternative to Kubernetes for many single-node or small-scale deployments. You get predictable startup behavior, proper dependency management, clean shutdowns, logging, and monitoring — all without the overhead and complexity of a full orchestration platform.

What’s next?

At this point in the post, I want to look a bit ahead and suggest some directions that could be explored without going into too much detail.

Quadlet

Quadlet was originally developed to simplify the process. You can think of it as a Docker Compose or a Kubernetes manifest for systemd. The project has since been archived because it is now directly integrated into Podman itself.

I won’t go into more detail on this topic here, as I haven’t used it yet to form a solid opinion. I mainly want to underline its existence.

A service with docker-compose

To get the best of both worlds, docker-compose can be launched from systemd. In this case, the app, database, and network are no longer split into multiple services but merged into a single one.

Cloud-init for further automation

systemd can be easily configured during first boot with cloud-init. A common automation scenario on cloud providers could be:

OpenTofu → Cloud-init → systemd → Podman containers