Skip to main content

Building images using Podman and cron

How to combine Podman, cron, tea, and a shower to update images
Image
Camera on tripod with sun landscape

Image by jurapolle from Pixabay

The old saying that necessity is the mother of invention rang so very true for me the other day. As you may be aware, we have a number of container images on quay.io for each of our headline container projects. For Buildah, Podman, and Skopeo, we have four container images that live out on quay.io. It's very handy to pull one of these images down and then use it to test out the latest bits or to avoid going through an installation process.

I recently experienced a problem where image version information wasn't updating after a build trigger event. This issue left us with out-of-date images. Clearly, not acceptable.

Before I delve into the problem (and my solution), let's get a little bit of background on these container images.

Skopeo container images

For instance, let's take a look at the Skopeo container images. There are four "flavors" of this image.

They are:

  • quay.io/containers/skopeo:latest
  • quay.io/skopeo/stable:latest
  • quay.io/skopeo/testing:latest
  • quay.io/skopeo/upstream:latest

The first two images, container/skopeo and skopeo/stable, are in effect the same image. They're built with the same Dockerfile and are just kept in separate locations in case somebody has a preference.

Both of these images have the latest stable version of Skopeo on board. The skopeo/testing image has the latest version of Skopeo that's in updates testing in the Fedora Project's Bodhi system. Quite often, this is the same as the version in the stable image, but when it's not, it's a great image to test out the upcoming release.

The last flavor is skopeo/upstream. That image contains all of the bits that are currently in the upstream development repository on GitHub. So if you've made a commit there, you can grab this image to double-check it after it's been rebuilt.

Use Skopeo container images

Using these images is a breeze. Take Skopeo's container image, for instance. Skopeo is especially handy for copying an image from a public to a private repository, or for looking at the manifest of an image. Here are a few quick examples using Podman to run the stable Skopeo container image:

# Get Help on Skopeo
podman run quay.io/skopeo/stable:latest --help

# Get help on the Skopeo Copy command
podman run quay.io/skopeo/stable:latest copy --help

# Copy the Skopeo container image from quay.io to
# a private registry
podman run quay.io/skopeo/stable:latest copy docker://quay.io/skopeo/stable docker://registry.internal.company.com/skopeo

# Inspect the fedora:latest image
podman run quay.io/skopeo/stable:latest inspect --config docker://registry.fedoraproject.org/fedora:latest  | jq

For a really in-depth look at how to use the Skopeo container images, check out this terrific and recent post by Valentin Rothberg.

Podman and Buildah have the same corresponding images and are used in similar ways. If you're interested in more details, each set of these container images has a README.md in their repository. Buildah’s README.md is here, Podman’s README.md is here, and Skopeo’s README.md is here.

Back to my tale of woe

In the quay.io repositories, there's functionality called a build trigger. You can set a build trigger to run whenever a merge is completed on a GitHub repository. It automatically rebuilds and replaces the container image in your repository. We set up build triggers for each of our repositories, and recently added build triggers for Skopeo. At that time, Skopeo was at version v0.1.42-dev, and our initial images worked great.

A few weeks after that, we bumped Skopeo up to v1.0.0. I kept checking the version for the Skopeo stable image. I expected it to change, but it never did. I waited for the next merge to go through and rechecked—still the older version. In the quay.io web interface, you can trigger a build "by hand." After I did that, the same older version was still reported. I could see that the image had been rebuilt. What the heck!?

Digging around a bit more in the web interface at quay.io, I was able to determine that the container runtime engine used by quay.io was building with the --cache flag turned on. Even though there was a new RPM package, the build process wasn't using it as the Dockerfile, and the Fedora image specified in the FROM statement had not changed. In effect, the build trigger was just re-copying the existing image.

Re-copying the image is not only a problem because it does not receive our latest executable, but also because the image does not get updated executables. Even if our executable does not change, we would like to receive updates from Fedora periodically. That way we know that our image has the latest security fixes. We've since filed an RFE with quay.io in hopes that images can be built there with --no-cache, but if it happens, it won't happen right away. How can I solve this problem?

My shower solution

Short personal confession time. I don't know why, but a huge chunk of my ideas come to me in the shower. While showering the other day, I thought, "why rely on quay.io's build trigger? I've got Podman, and it builds container images really nicely. I can probably automate it using cron." So I put together this little bash script. It calls a few Podman commands to build and push each of the flavors of container images. Let's take a look at a few of the fun snippets in the script.

The script

The script accepts two arguments passed into it: The name of the project (buildah, podman or skopeo), and the directory location of the authfile created by podman login quay.io. That file must be created by someone with write privileges to the container repositories on quay.io. If either of these values is not passed in, the script queries the user for them. So far, the script is just your normal bash hackery.

Now for the fun part, and it was actually quick and straightforward. The script uses just three Podman commands for each image. Arguably it could drop the third one, which is just a cleanup step. The commands to build and push the quay.io/containers/${PROJECT}:latest image are:

podman build --no-cache -t quay.io/containers/${PROJECT}:latest -f https://raw.githubusercontent.com/containers/${GITHUBPROJECT}/master/contrib/${PROJECT}image/stable/Dockerfile .

podman push --creds $QUAYUSER:$QUAYPWD quay.io/containers/${PROJECT}:latest

podman rmi -f quay.io/containers/${PROJECT}:latest

The podman build command creates the image with the --no-cache option, which keeps the image from using any pre-existing layers in the environment. The command builds it "fresh." It tags the image with the name specified with the -t option and then uses the Dockerfile for the stable image in Skopeo's GitHub repository. After it's created, the script uses a simple podman push to the repository on quay.io. The final step removes the local copy of the image by using the podman rmi command. Repeat the same process for the other three flavors of images for the project.

It's an easy procedure, if I've consumed enough tea to remember to do it!

What about automation?

That's all good, but I want to run this automatically. I'm a very forgetful person, especially when not loaded up with tea, and I know I'll forget to run the script one day when I'm supposed to. Here's where our old friend cron comes in to play.

First, I logged into my development VM as root, created the /root/quay.io directory, and then copied the script into it. I set its permissions with chmod 755 so it could be run. Next, I ran podman login quay.io to log in to the repository. Finally, I ran crontab -e to open up the editor for cron's configuration file. If you have not used cron for a while, here's a terrific blog post. Once the vi editor popped up, I just dropped these three lines in place:

0 8 * * * /root/quay.io/rebuild_quay_ctr_images.sh buildah /run/user/0/containers/auth.json
0 9 * * * /root/quay.io/rebuild_quay_ctr_images.sh skopeo /run/user/0/containers/auth.json
0 10 * * * /root/quay.io/rebuild_quay_ctr_images.sh podman /run/user/0/containers/auth.json

One quick thing to note in those three crontab lines. They each pass in the authfile location /run/user/0/containers/auth.json, which is created by the podman login quay.io command when called as a root user. If you decide to run this as a non-root user, that location would have to be updated to the file location of the user, replacing 0 with the user's UID. If you wanted to, you could copy the /run/user/0/containers/auth.json file to a permanent location, such as /root/auth.json. That way, the file survives any system reboots, as the /run directory is cleared with each restart.

Back to the crontab

Now let's return to the crontab file. After saving and exiting the crontab configuration file, cron scheduled the build for Buildah at 8:00 a.m. every day, the build for Skopeo at 9:00 a.m. every day, and the build for Podman at 10:00 a.m. every day. Each project takes about 20 minutes to build all four flavors of the images, so I could have scheduled them more closely together, but this spacing is sufficient. So as long as my development VM is up, the four flavors of container images will be built and placed out on quay.io, regardless of my tea content.

Wrap up

All in all, my little invention—built by the necessity to update these images more frequently—was much simpler than I thought it would be going into this process. I don't think I'll be running down to the patent office with this idea, but it's nice to be doing a little dogfooding with Podman. In a way, it keeps itself updated all on its own. Hopefully, this example will spark some ideas for your own environment, and you'll create your own simple invention that fills a need in your shop—that you can then share on Enable Sysadmin.

[ Getting started with containers? Check out this free course. Deploying containerized applications: A technical overview. ]

Topics:   Containers   Podman  
Author’s photo

Tom Sweeney

Software engineer at Red Hat working on containers focusing on the Buildah and Podman projects. Manages the buildah.io and podman.io websites and can be found on freenode at #buildah and #podman. More about me

Try Red Hat Enterprise Linux

Download it at no charge from the Red Hat Developer program.