Auto-updating podman containers with systemd

Auto-Updating containers can be very useful in some cases. Podman provides mechanisms to take care of container updates automatically. This article demonstrates how to use Podman Auto-Updates for your setups.

Podman

Podman is a daemonless Docker replacement that can handle rootfull and rootless containers. It is fully aware of SELinux and Firewalld. Furthermore, it comes pre-installed with Fedora Linux so you can start using it right away.

If Podman is not installed on your machine, use one of the following commands to install it. Select the appropriate command for your environment.

# Fedora Workstation / Server / Spins
$ sudo dnf install -y podman

# Fedora Silverblue, IoT, CoreOS
$ rpm-ostree install podman

Podman is also available for many other Linux distributions like CentOS, Debian or Ubuntu. Please have a look at the Podman Install Instructions.

Auto-Updating Containers

Updating the Operating System on a regular basis is somewhat mandatory to get the newest features, bug fixes, and security updates. But what about containers? These are not part of the Operating System.

Why Auto-Updating?

If you want to update your Operating System, it can be as easy as:

$ sudo dnf update

This will not take care of the deployed containers. But why should you take care of these? If you check the content of containers, you will find the application (for example MariaDB in the docker.io/library/mariadb container) and some dependencies, including basic utilities.

Running updates for containers can be tedious and time-consuming, since you have to:

  1. pull the new image
  2. stop and remove the running container
  3. start the container with the new image

This procedure must be done for every container. Updating 10 containers can easily end up taking 30-40 commands that must be run.

Automating these steps will save time and ensure, that everything is up-to-date.

Podman and systemd

Podman has built-in support for systemd. This means you can start/stop/restart containers via systemd without the need of a separate daemon. The Podman Auto-Update feature requires you to have containers running via systemd. This is the only way to automatically ensure that all desired containers are running properly. Some articles like these for Bitwarden and Matrix Server already had a look at this feature. For this article, I will use an even simpler Apache httpd container.

First, start the container with the desired settings.

# Run httpd container with some custom settings
$ sudo podman container run -d -t -p 80:80 --name web -v web-volume:/usr/local/apache2/htdocs/:Z docker.io/library/httpd:2.4

# Just a quick check of the container
$ sudo podman container ls
CONTAINER ID  IMAGE                        COMMAND           CREATED        STATUS            PORTS               NAMES
58e5b07febdf  docker.io/library/httpd:2.4  httpd-foreground  4 seconds ago  Up 5 seconds ago  0.0.0.0:80->80/tcp  web

# Also check the named volume
$ sudo podman volume ls
DRIVER      VOLUME NAME
local       web-volume

Now, set up systemd to handle the deployment. Podman will generate the necessary file.

# Generate systemd service file
$ sudo podman generate systemd --new --name --files web

/home/USER/container-web.service

This will generate the file container-web.service in your current directory. Review and edit the file to your liking. Here is the file contents with added newlines and formatting to improve readability.

# container-web.service

[Unit]
Description=Podman container-web.service
Documentation=man:podman-generate-systemd(1)
Wants=network.target
After=network-online.target
RequiresMountsFor=%t/containers

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
TimeoutStopSec=70
ExecStartPre=/bin/rm -f %t/container-web.pid %t/container-web.ctr-id

ExecStart=/usr/bin/podman container run \
          --conmon-pidfile %t/container-web.pid \
          --cidfile %t/container-web.ctr-id \
          --cgroups=no-conmon \
          --replace \
          -d \
          -t \
          -p 80:80 \
          --name web \
          -v web-volume:/usr/local/apache2/htdocs/ \
          docker.io/library/httpd:2.4

ExecStop=/usr/bin/podman container stop \
          --ignore \
          --cidfile %t/container-web.ctr-id \
          -t 10

ExecStopPost=/usr/bin/podman container rm \
          --ignore \
          -f \
          --cidfile %t/container-web.ctr-id

PIDFile=%t/container-web.pid
Type=forking

[Install]
WantedBy=multi-user.target default.target

Now, remove the current container, copy the file to the proper systemd directory, and start/enable the service.

# Remove the temporary container
$ sudo podman container rm -f web

# Copy the service file
$ sudo cp container-web.service /etc/systemd/system/container-web.service

# Reload systemd
$ sudo systemctl daemon-reload

# Enable and start the service
$ sudo systemctl enable --now container-web

# Another quick check
$ sudo podman container ls
$ sudo systemctl status container-web

Please be aware, that the container can now only be managed via systemd. Starting and stopping the container with the “podman” command may interfere with systemd.

Now that the general setup is out of the way, have a look at auto-updating this container.

Manual Auto-Updates

The first thing to look at is manual auto-updates. Sounds weird? This feature allows you to avoid the 3 steps per container, but you will have full control over the update time and date. This is very useful if you only want to update containers in a maintenance window or on the weekend.

Edit the /etc/systemd/system/container-web.service file and add the label shown below to it.

--label "io.containers.autoupdate=registry"

The changed file will have a section appearing like this:

...snip...

ExecStart=/usr/bin/podman container run \
          --conmon-pidfile %t/container-web.pid \
          --cidfile %t/container-web.ctr-id \
          --cgroups=no-conmon \
          --replace \
          -d \
          -t \
          -p 80:80 \
          --name web \
          -v web-volume:/usr/local/apache2/htdocs/ \
          --label "io.containers.autoupdate=registry" \
          docker.io/library/httpd:2.4

...snip...

Now reload systemd and restart the container service to apply the changes.

# Reload systemd
$ sudo systemctl daemon-reload

# Restart container-web service
$ sudo systemctl restart container-web

After this setup you can run a simple command to update a running instance to the latest available image for the used tag. In this example case, if a new 2.4 image is available in the registry, Podman will download the image and restart the container automatically with a single command.

# Update containers
$ sudo podman auto-update

Scheduled Auto-Updates

Podman also provides a systemd timer unit that enables container updates on a schedule. This can be very useful if you don’t want to handle the updates on your own. If you are running a small home server, this might be the right thing for you, so you are getting the latest updates every week or so.

Enable the systemd timer for podman as follows:

# Enable podman auto update timer unit
$ sudo systemctl enable --now podman-auto-update.timer 

Created symlink /etc/systemd/system/timers.target.wants/podman-auto-update.timer → /usr/lib/systemd/system/podman-auto-update.timer.

Optionally, you can edit the schedule of the timer. By default, the update will run every Monday morning, which is ok for me. Edit the timer module using this command:

$ sudo systemctl edit podman-auto-update.timer

This will bring up your default editor. Changing the schedule is beyond the scope of this article but the link to systemd.timer below will help. The Demo section of Systemd Timers for Scheduling Tasks contains details as well.

That’s it. Nothing more to do. Podman will now take care of image updates and also prune old images on a schedule.

Hints & Tips

Auto-Updating seems like the perfect solution for container updates, but you should consider some things, before doing so.

  • avoid using the “latest” tag, since it can include major updates
  • consider using tags like “2” or “2.4”, if the image provider has them
  • test auto-updates beforehand (does the container support updates without additional steps?)
  • consider having backups of your Podman volumes, in case something goes sideways
  • auto-updates might not be very useful for highly productive setups, where you need full control over the image version in use
  • updating a container also restarts the container and prunes the old image
  • occasionally check if the updates are being applied

If you take care of the above hints, you should be good to go.

Docs & Links

If you want to learn more about this topic, please check out the links below. There is a lot of useful information in the official documentation and some blogs.

Conclusion

As you can see, without the use of additional tools, you can easily run auto-updates on Podman containers manually or on a schedule. Scheduling allows unattended updates overnight, and you will get all the latest security updates, features, and bug fixes. Some setups I have tested successfully are: MariaDB, Ghost Blog, WordPress, Gitea, Redis, and PostgreSQL.

Fedora Project community

12 Comments

  1. For the auto-update you don’t have to edit the systemd unit file. Instead you just pass the label as an argument when you create the container:

    sudo podman container run -d -t -p 80:80 --name web --label "io.containers.autoupdate=registry" -v web-volume:/usr/local/apache2/htdocs/:Z docker.io/library/httpd:2.4

    and that’s it. Generate your unit file and do the rest, but you don’t need to edit anything. This is how podman know what containers to update and which ones not to.

    • Daniel Schier

      Hi,

      thanks for the addition. You are 100% correct. For the sake of this article, I opted to showcase the difference between the two different systemd files. đŸ™‚

    • tizzef

      Hi,
      I am new to podman and containers, and I wonder, in the web-volume syntax, what “Z:” (-v web-volume:/usr/local/apache2/htdocs/:Z) stand for ?
      I cannot see this parameter in the container-web.service file
      I must miss something ..
      Thanks for your response.

  2. laolux

    How convenient, I did not know that podman was shipping with an auto-update systemd unit. I always created my own.
    One tip: if using the btrfs filesystem, then one can easily create a snapshot before updating the containers.
    Another thing to note: This auto update can also be used by users running podman containers, so it is not restricted to root containers.

  3. James

    Really useful to know, thanks. In the case where I have a custom image that I build using the podman build command, could you suggest a workflow that will achieve a similar result?
    The run command is from the custom image, so unless I provide the run command with a server provided custom image that autoupdates using some sort CI system, I don’t think it is possible.

    • Daniel Schier

      If you refer to “have a image only on the local machine” – Yes, the label supports another option “local” (–label “io.containers.autoupdate=local \”). This way, Podman will look for a local, matching image.

      https://docs.podman.io/en/latest/markdown/podman-auto-update.1.html

      Alternatively, if the autoupdate label is set to local, Podman will compare the image a container is using to the image with its raw name in local storage. If an image is updated locally, Podman simply restarts the systemd unit executing the container.

  4. mediaklan

    How do you find out the url for the ‘docker.io’ part ? As I undertand it, here you’re providing the url for Apache (docker.io/library/httpd:2.4) ?
    What if I want to auto-update navidrome : https://hub.docker.com/r/deluan/navidrome ?
    Thank you for youur help

    • Daniel Schier

      If you search for the container image (ex.: podman search navidrome), you will get the url. In most cases, the format is “docker.io//”. For official images like httpd, nginx, mariadb, you can use “library” as the organization.

  5. Mehdi

    Very useful article. Thanks for sharing.

  6. René Genz

    I noticed 2 typos:
    At “Podman will generated the” I would remove the trailing d, so it reads “generate”.
    At “the file container-web service” I would add a dot, so it reads “container-web.service”.

Comments are Closed

The opinions expressed on this website are those of each author, not of the author's employer or of Red Hat. Fedora Magazine aspires to publish all content under a Creative Commons license but may not be able to do so in all cases. You are responsible for ensuring that you have the necessary permission to reuse any work on this site. The Fedora logo is a trademark of Red Hat, Inc. Terms and Conditions