Preparing for applying updates on Red Hat Linux

installonly_limit=3

After examining the results, we can decide if we start the update, or cancel it. As yum will update everything it can find updates for, we may want to remove unneeded packages beforehand. We may also notice a package marked for update that we are version-locked with that need to be excluded from the upgrade.

After approval yum will download all new packages, and install/update them one by one. When completed, it will check the integrity of the installed/updated packages, clean up unneeded files. It also provides feedback during the process, providing a line of text for each step, as well as an exit code that hints if the upgrade was successful, or if some problem arose. It will also cancel the update process if a problem rises that seems critical from the consistent system perspective – but there are times when it is too late already, so preventing update problems from happening is a better approach.

Disk space

yum cache

From the process described above we could guess that we need some disk space for the update process:

• The metadata of all configured repositories need to be stored until the calculation of all packages (and their dependencies) to be updated finishes.
• rpm packages that constitute the update itself need to be stored locally until installed properly.

This data, called yum cache is only needed during the update, but can take up substantial disk space. The default location for this cache is in the /var/cache/yum directory. Needless to say that if there isn’t enough space to store all data needed, the update process will fail. Some unfinished downloads will be dropped, but not all space may be freed, which ends up having a system failed the update and have it’s volume containing /var/cache almost full.

Many installations store their /var directory on a volume dedicated to logging, as the default place for logfiles is /var/log on most distros, and most well-behaving applications will stop working or even crash if they can’t write their logfiles. So filling up the volume they are writing to is a bad thing.

The more packages need to be upgraded, and the more repositories we have, the more space the update will occupy temporary. To calculate this space from update to update is hard, but can be tested with the dry run solution described later if we have a test machine with the exact software content. For a real time example, updating from RHEL 7.1 to 7.5 (desktop installation with Gnome) may take up 4 GB of cache space, but installation of a few fixes to a system that is only one or two months out of date will take up only a few MB.

To check how much space we have, we can use the df command:

# df -h /var/
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/vg_sys-var  6.0G  1.7G  4.4G  28% /var




In the above example we have 4.4 GB of free space, which will be enough given that the server was updated only a few months ago. To free up space a trivial step would be to clear the yum cache already stored (maybe at the last update). To check how much space the a cache occupies at the moment, we can use du:

# du -mcd 1 /var/cache/yum
1103    /var/cache/yum/x86_64
1103    /var/cache/yum
1103    total




The above numbers are in MB, so the yum cache in this example takes up about 1 GB of disk space and occupies most of the space on the /var volume.

---

---

Clearing the cache

We can clear the whole cache with the following command:

yum clean all

But as yum notifies us in the above command’s output on RHEL 7 versions, there may be orphaned data from removed or disabled repositories, which will most likely happen after minor release ugrades, in which case we can safely clear the data by hand:

rm -rf /var/cache/yum/*

We may get more space for the update by clearing other data stored on the volume, like compressing/deleting old logfiles, moving large files to other volumes, or extending the volume size.

Moving the cache

To work on with the possibilities of yum, if we are really low on disk space, can’t clear anything further, and can’t add more space to the volume, we can move the location of the yum cache to another volume with more free space. We can configure the cache location in the yum.conf configuration file mentioned above. Consider the default setting:

cachedir=/var/cache/yum/$basearch/$releasever

By changing the path before $basearch the next yum operation will work with the same directory structure, but on a different path – hopefully with more free space for the upgrade. We can also move the cache to another volume by moving the whole directory:

mv /var/cache/yum /extended_data_volume/




And creating a symlink at the original location that points to the new place:

ln -s /extended_data_volume/yum /var/cache/yum




It is wise to know that the update will not fail on a trivial error such as low disk space. On a large system sysadmins deploy monitoring tools like Nagios that can report low disk space on all machines, making this step much less time consuming and error prone.

Network errors

If there are problems with connectivity between the repositories and the machine performing the update, the update may fail. This can only happen at the metadata, or the new rpms download stage, and will not break the system. You can start the update process again when the network issue is solved.

On the other hand, if the update is initialized from an interactive session, on network outage the connection may break, leaving the updating machine without admin to answer the questions yum may ask. If the package install/update stage already started, it will continue unattended, and may fail or complete if it would otherwise do. After reconnection the process can be followed in the /var/log/yum.log.

---

---

Yum dry run

Aside insufficient disk space and network problems, the update in many cases can fail on unresolved package dependencies. These need to be solved with tools that can calculate and handle package dependencies, but it would be useful to know there will be problems before the actual update (and therefore not wasting the always too short downtime of the system). To get this valuable information we can run the update process as it would run the actual update, but stop before any actual package downloading, installing or updating have taken place.

Around Redhat 6.6 a new option was introduced that will cause yum to assume “No” to every question that comes up during update – including the approval before the actual package manipulation stage, and as a consequence no actual interaction is needed execute a dry run:

yum update --assumeno

This can be the ideal tool to provide a dry run of the coming update, including packages to be upgraded, and any errors what may occur. Consider the following simple bash script:

#!/bin/bash
yum update --assumeno &> $(hostname).yum.dryrun.$(date '+%Y-%m-%d').out
exit $?




The above script can be executed automatically and will provide a text report of the dry run, as well as an overall exit code indicating any problems. The output does not need to be saved on the local file system. The target of the output redirection can be a network file system, or the report can be posted to some central reporting server, may be gathered by other scripts or applications. The reports can be published and distributed among other IT departments for approval, this way everyone involved can see exactly what packages will be updated, and to what version.

The dry run can be scheduled to run on a given time frame (maybe at night to impact the system’s performance less) with cron, or executed from a central source with a puppet setup. The exit code can also be stored and processed by monitoring or facter, to aggregate the possible results of the upcoming upgrade before proceeding.

Conclusion

Even with one or a few computers, we should gather information before starting an update of the whole operating system, just to be on the safe side. One day there will be a problem, and it is much less stressing if you can solve it before it has impact on the actual job of a given machine. On a larger scale, it is simply not possible to sit next to each server or desktop and support it with your presence in the hopes that this will help the update to run flawlessly.

By knowing the stages of the update process, the pitfalls, as well as the solution to them is essential for successful updates. Starting your whole infrastructure’s next update stage with the confidence that there will be no problems is to do it with style.