How to move Request Tracker into a Linux container

Even though it took me a long time to get motivated, I finally containerized several personal Linux services. I've documented the project in this series. In this article, we'll take you through a final example, Request Tracker.

To kick off, we looked at some general principles for migrating applications to containers. Then we looked at the containerization of WordPress, and next, we discussed moving MediaWiki into a container. That project was a bit more involved than the first, with the addition of task scheduling. In this final article, we're going to consider a much more complex migration. Specifically, we'll look at Request Tracker. This service might be the most tricky because both the build and run are fairly sophisticated.

Editor's Note: For the purpose of this article, we assume you'll be building your containers on Red Hat Enterprise Linux 8 using podman build. You may be able to use the instructions on other distributions or with other toolchains, however, some modifications may be required.

Moving Request Tracker

Build

Unlike WordPress and MediaWiki, which run on a single-layered image on top of a base image, Request Tracker uses two layers on top of a base image. Let's look at each layer and see why we did it this way.

The first layer is built quite similar to the httpd-php image. This image adds the necessary services for a Perl-based web application. We include Apache, the FastCGI module, Perl, MariaDB, cron, and some basic utilities for troubleshooting:

FROM registry.access.redhat.com/ubi8/ubi-init
MAINTAINER fatherlinux <scott.mccarty@gmail.com>
RUN yum install -y httpd mod_fcgid perl mariadb-server mariadb crontabs cronie iputils net-tools; yum clean all
RUN systemctl enable mariadb
RUN systemctl enable httpd
RUN systemctl enable postfix
RUN systemctl disable systemd-update-utmp.service
ENTRYPOINT ["/sbin/init"]
CMD ["/sbin/init"]

The second layer is where things get pretty sophisticated. Request Tracker uses a lot of Perl modules from CPAN. Many of these modules are compiled with gcc and take a long time to install. It also took a lot of work to nail down all of these dependencies to get Request Tracker to install successfully. Historically, we would have captured this in a script somewhere, but with containers, we can have it all in one Containerfile. It's very convenient.

[ You might also enjoy: 6 guides on making containers secure ]

The next thing you should notice about this file is that it's a multi-stage build. Podman and Buildah can absolutely do multi-stage builds, and they can be extremely useful for applications like Request Tracker. We could have bind-mounted in directories, as we did with WordPress and MediaWiki, but we chose a multi-stage build instead. This will give us portability and speed if we need to rebuild the application somewhere else.

Multi-stage builds can be thought of as capturing the development server and the production server in a single build file. Historically, development servers were actually the hardest to automate. Since the early days of CFEngine in the mid-1990s, developers refused to use version control and added anything they wanted to development servers to make them work. Often, they didn't even know what they added to make a build complete. This was actually rational when you had long-lived servers that were well backed up, but it always caused pain when systems administrators had to "upgrade the dev server." It was a nightmare to get builds to function on a brand new server with a fresh operating system.

With multi-stage builds, we capture all of the build instructions and even cache layers that are constructed. We can rebuild this development virtual server anywhere we like.

FROM registry.access.redhat.com/ubi8/ubi-init
FROM localhost/httpd-perl AS localhost/rt4-build
MAINTAINER fatherlinux <scott.mccarty@gmail.com>
RUN yum install -y expat-devel gcc; yum clean all
RUN cpan -i CPAN
RUN cpan -i -f GnuPG::Interface
RUN cpan -i DBIx::SearchBuilder \
ExtUtils::Command::MM \
Text::WikiFormat \
Devel::StackTrace \
Apache::Session \
Module::Refresh \
HTML::TreeBuilder \
HTML::FormatText::WithLinks \
HTML::FormatText::WithLinks::AndTables \
Data::GUID \
CGI::Cookie \
DateTime::Format::Natural \
Text::Password::Pronounceable \
UNIVERSAL::require \
JSON \
DateTime \
Net::CIDR \
CSS::Minifier::XS \
CGI \
Devel::GlobalDestruction \
Text::Wrapper \
Net::IP \
HTML::RewriteAttributes \
Log::Dispatch \
Plack \
Regexp::Common::net::CIDR \
Scope::Upper \
CGI::Emulate::PSGI \
HTML::Mason::PSGIHandler \
HTML::Scrubber \
HTML::Entities \
HTML::Mason \
File::ShareDir \
Mail::Header \
XML::RSS \
List::MoreUtils \
Plack::Handler::Starlet \
IPC::Run3 \
Email::Address \
Role::Basic \
MIME::Entity \
Regexp::IPv6 \
Convert::Color \
Business::Hours \
Symbol::Global::Name \
MIME::Types \
Locale::Maketext::Fuzzy \
Tree::Simple \
Clone \
HTML::Quoted \
Data::Page::Pageset \
Text::Quoted \
DateTime::Locale \
HTTP::Message \
Crypt::Eksblowfish \
Data::ICal \
Locale::Maketext::Lexicon \
Time::ParseDate \
Mail::Mailer \
Email::Address::List \
Date::Extract \
CSS::Squish \
Class::Accessor::Fast \
LWP::Simple \
Module::Versions::Report \
Regexp::Common \
Date::Manip \
CGI::PSGI \
JavaScript::Minifier::XS \
FCGI \
PerlIO::eol \
GnuPG::Interface \
LWP::UserAgent >= 6.02 \
LWP::Protocol::https \
String::ShellQuote \
Crypt::X509
RUN cd /root/rt-4.4.4;make testdeps;make install

# Deploy
FROM localhost/httpd-perl AS localhost/rt:4.4.4
RUN yum install -y postfix mailx;yum clean all
COPY --from=localhost/rt4-build /opt/rt4 /opt/rt4
COPY --from=localhost/rt4-build /usr/lib64/perl5 /usr/lib64/perl5
COPY --from=localhost/rt4-build /usr/share/perl5 /usr/share/perl5
COPY --from=localhost/rt4-build /usr/local/share/perl5 /usr/local/share/perl5
COPY --from=localhost/rt4-build /usr/local/lib64/perl5/ /usr/local/lib64/perl5/
RUN chown -R root.bin /opt/rt4/lib;chown -R root.apache /opt/rt4/etc
ENTRYPOINT ["/sbin/init"]
CMD ["/sbin/init"]

The second stage in this multi-stage build constructs the virtual production server. By splitting this into a second stage, we don't have to install development tools like gcc or expat-devel in the final production image. This reduces our image's size and reduces the size of the software supply chain in network-exposed services. This also potentially reduces the chances of somebody doing something nasty with our container, should they hack in.

We only install the mail utilities in this second stage, which defines the second layer of our production image for Request Tracker. We could have installed these utilities in the httpd-perl layer, but many other Perl applications won't need mail utilities.

Another convenience of multi-stage builds is that we don't have to rebuild all of those Perl modules every time we want to update the Perl interpreter, Apache, or MariaDB for security patches.

Run

Now, like WordPress and MediaWiki, let's take a look at some of the tricks we use at runtime:

[Unit]
Description=Podman container – rt.fatherlinux.com
Documentation=man:podman-generate-systemd(1)

[Service]
Type=simple
ExecStart=/usr/bin/podman run -i --rm --read-only -p 8081:8081 --name rt.fatherlinux.com \
-v /srv/rt.fatherlinux.com/code/reminders:/root/reminders:ro \
-v /srv/rt.fatherlinux.com/config/rt.fatherlinux.com.conf:/etc/httpd/conf.d/rt.fatherlinux.com.conf:ro \
-v /srv/rt.fatherlinux.com/config/MyConfig.pm:/root/.cpan/CPAN/MyConfig.pm:ro \
-v /srv/rt.fatherlinux.com/config/RT_SiteConfig.pm:/opt/rt4/etc/RT_SiteConfig.pm:ro \
-v /srv/rt.fatherlinux.com/config/root-crontab:/var/spool/cron/root:ro \
-v /srv/rt.fatherlinux.com/config/aliases:/etc/aliases:ro \
-v /srv/rt.fatherlinux.com/config/main.cf:/etc/postfix/main.cf:ro \
-v /srv/rt.fatherlinux.com/data/mariadb:/var/lib/mysql:Z \
-v /srv/rt.fatherlinux.com/data/logs/httpd:/var/log/httpd:Z \
-v /srv/rt.fatherlinux.com/data/logs/rt4:/opt/rt4/var:Z \
-v /srv/rt.fatherlinux.com/data/backups:/root/.backups:Z \
--tmpfs /etc \
--tmpfs /var/log/ \
--tmpfs /var/tmp \
--tmpfs /var/spool \
--tmpfs /var/lib \
localhost/rt:latest
ExecStop=/usr/bin/podman stop -t 3 rt.fatherlinux.com
ExecStopAfter=/usr/bin/podman rm -f rt.fatherlinux.com
Restart=always

[Install]
WantedBy=multi-user.target

Like MediaWiki, all of the config files are bind-mounted in read-only, giving us a solid security upgrade. Finally, the data directories are read-write, just like our other containers. One simple observation: We did still bind mount some code into the image for Reminders, which is a small, home-grown set of scripts that send emails and generate tickets for weekly, monthly, and annual entries.

Further analysis

Let's tackle a few last subjects that aren't specific to any one of our containerized Linux services.

Recoverability

Recoverability is something we have to consider carefully. By using systemd, we get solid recoverability, on par with regular Linux services. Notice systemd restarts my services without blinking an eye:

podman kill -a
55299bdfebea23db81f0277d45ccd967e891ab939ae3530dde155f550c18bda9
87a34fb86f854ccb86d9be46b5fe94f6e0e15322f5301e5e66c396195480047a
C8092df3249e5b01dc11fa4372a8204c120d91ab5425eb1577eb5f786c64a34b

Look at that! Restarted services:

podman ps
CONTAINER ID  IMAGE                       COMMAND     CREATED       STATUS                     PORTS                   NAMES
33a8f9286cee  localhost/httpd-php:latest  /sbin/init  1 second ago  Up Less than a second ago  0.0.0.0:80->80/tcp      wordpress.crunchtools.com
37dd6d4393af  localhost/rt:4.4.4          /sbin/init  1 second ago  Up Less than a second ago  0.0.0.0:8081->8081/tcp  rt.fatherlinux.com
e4cc410680b1  localhost/httpd-php:latest  /sbin/init  1 second ago  Up Less than a second ago  0.0.0.0:8080->80/tcp    learn.fatherlinux.com

Tips and tricks

This is quite useful for making config file changes. We can simply edit the config file on the container host or use something like Ansible and kill all of the containers with the podman kill -a command. Because we are using systemd, it will gracefully handle restarting the services. This is very convenient.

It can be tricky to get software to run within a container, especially when you want it to run read-only. You are constraining the process in ways in which it wasn't necessarily designed. As such, here are some tips and tricks.

First, it's useful to install some standard utilities in your containers. In this guide, we installed ip-utils and net-tools so that we could troubleshoot our containers. For example, with Request Tracker, I had to troubleshoot the following entry in /etc/aliases, which generates tickets from emails:

professional:         "|/opt/rt4/bin/rt-mailgate --queue 'Professional' --action correspond --url http://localhost:8081/"

The tools curl, ping, and netstat were all extremely useful because we are also using external DNS and Cloudflare.

Next up is podman diff, which I used extensively for running containers as read-only. You can run the container in read-write mode and constantly check podman diff to see what files have changed. Here's an example:

podman diff learn.fatherlinux.com
C /var
C /var/spool
C /var/spool/cron
A /var/spool/cron/root
C /var/www
C /var/www/html
A /var/www/html/learn.fatherlinux.com
C /root
A /root/.backups

Moving to Kubernetes

Notice that Podman will tell us which files have changed since the container started. In this case, every file that we care about is either on a tmpfs or a bind mount. This enables us to run this container as read-only.

Taking a hard look at Kubernetes is a natural next step. Using a command like podman generate kube will get us part of the way there, but we still need to figure out how to manage persistent volumes and backups on those persistent volumes. For now, we've decided that Podman + systemd provides a nice foundation. All of the work that we have done with splitting up the code, configuration, and data is requisite to getting us to Kubernetes.

Notes on environment

My environment is a single virtual machine running at Linode.com with 4GB of RAM, two CPUs, and 80GB of storage. I was able to upload my own custom image of RHEL 8 to serve as the container host. Other than setting the hostname and pointing DNS through Cloudflare, I really didn't have to make any other changes to the host. All of the important data is in /srv, which would make it extremely easy to replace if it were to fail. Finally, the /srv directory on the container host is completely backed up.

If you are interested in looking at the configuration files and directory structure of /srv, I have saved the code here in my GitHub.

Biases

Like everyone, I have biases, and I think it's fair to disclose them. I served as a Linux Systems Administrator for much of my career before coming to Red Hat. I have a bias towards Linux, and towards Red Hat Enterprise Linux in particular. I also tend toward automation and the psychology of how to make that automaton accessible to regular contributors.

One of my earliest frustrations as a sysadmin was working on a team with 1000 Linux web servers (doing eCards in web 1.0) where documentation for how to contribute to the automation was completely opaque and had no reasoning documented for why things were the way they were. We had great automation, but nobody considered the psychology of how to introduce new people to it. It was sink-or-swim.

This blog aims to help people get over that hump, while at the same time, making it almost self-documenting. I think it's critically important to consider the human inputs and robot outputs of automation. See also: Bootstrapping And Rooting Documentation: Part 1

[ Free cheat sheet: Kubernetes glossary ] 

Conclusion

It seems so easy to move a common service like WordPress into containers, but it's really not. The flexible and secure architecture outlined in this article marshals a senior Linux Administrator or Architect's skills to move from a regular LAMP server to OCI-compliant containers. This guide leveraged a container engine called Podman while also preparing your services for Kubernetes. Separating your code, configuration, and data is a requisite step for moving on to Kubernetes. It all starts with solid, foundational Linux skills.

Some decisions highlighted in this article purposefully challenge various misconceptions within the container community—things like using systemd in a container or only focusing on the smallest base image you can find without paying attention to the entire software supply chain. However, the end product is simple to use. It provides a workflow quite similar to a traditional LAMP server, requiring a minimal cognitive load for traditional Linux systems administrators.

Some of the design decisions made in this article are a compromise and imperfect. Still, I made them because I understand both the pressures of a modern DevOps culture and the psychology of operations and development teams. I wanted to provide the flexibility to get more value out of containers. This set of services should be useful as a model for migrating many of your own services into containers. This will simplify their management, upgrade, and recovery. This not only helps existing Linux admins but future cohorts who will inherit these services, including the future version of me who will have forgotten all of the details. These containerized services are essentially self-documenting in a style that is conducive to a successful DevOps culture.

This series is based on "A Hacker's Guide to Moving Linux Services into Containers" on CrunchTools.com and is republished with permission.