Upgrading and updating your Red Hat Satellite server

Editor's note: This article was written while James Brigman was a member of the Red Hat Accelerator program.

Red Hat Satellite provides a means for updating Red Hat Linux systems with new or updated RPMs. Satellite hosts repositories locally so we can get those updates faster and for a larger number of clients. This feature becomes important for our software development process, because we need to quickly create and destroy virtual machines in service to the development process.

Our Satellite server is often used during the development cycle, so we have to know it’s working correctly. If any problems occur, we need to spot and solve them quickly so that the developers can continue their work uninterrupted.

This article describes the high points of the process and includes lessons learned from updating and upgrading my Red Hat Satellite server.

An important warning

If you run Satellite as a virtual machine (in whatever VM system you use) take a backup or a snapshot. If you run Satellite on Red Hat Enterprise Linux running bare metal, take a full backup. You want to be able to return to the original state if you run into problems.

[Want to try out Red Hat Enterprise Linux? Download it now for free.]

Updating vs. upgrading

Time marches on. This fact means that for those of us in the system administration profession, upgrading software is a constant activity. We upgrade for various reasons, whether being able to get support for the products we run, getting (and keeping) needed functionality, or improving performance.

For Satellite, the terms updating and upgrading have very specific meanings. Updating is when you make a dot update to the existing software. For this discussion, updating took us from Red Hat Satellite 6.4.2 to 6.4.4. Both are still Satellite 6.4 (major version), but the second changed from .2 to .4 (minor version).

Upgrading is when you take the Satellite server from an earlier major version to a more current major version. After the update, I then worked to upgrade the server from the 6.4 major version to the 6.5 major version.

By the end of the process, our satellite server was taken from 6.4.2 to 6.5.2.1.

Upgrading and updating

The process to upgrade or update involves four major steps. Let's take a look at each.

A tidy satellite

Before either updating or upgrading, you have a utility that can check the current state of your satellite and advise you on how ready you are to perform the update or upgrade. That utility is foreman-maintain and is described in the Satellite documentation.

You can search access.redhat.com with the term "upgrade satellite 6.5" to find this link. Under Content Type Filters, select Documentation to limit your search. You will find important definitions, prerequisites, and upgrade paths at this URL.

The steps listed in this document culminate in running the foreman-maintain command in upgrade mode. This command goes through your Satellite server, checking its fitness to be updated before you do the actual update. The command I used for both update and upgrade was the same, with different target versions:

# /usr/bin/foreman-maintain upgrade check --target-version 6.4.z

Of course, the correct syntax for your Satellite will vary.

Being tasked by tasks

My upgrade check failed several times. The most important reason is that for the version I was using, I had tasks left over from long ago that were not concluding in the needed state. I was able to clear those tasks using a foreman-rake command:

# /usr/bin/foreman-rake foreman_tasks:cleanup TASK_SEARCH='label = Actions::Katello::Repository::MetadataGenerate' STATES='running,pending,stopped' VERBOSE=true

Like the foreman-maintain command above, your foreman-rake command may differ depending on the output of your upgrade check.

The point of no return

Eventually, I reached the point where I was ready to do the upgrade itself. In a wise move, you will use the same command to do the actual upgrade, but with one change. The word check becomes run:

# /usr/bin/foreman-maintain upgrade run --target-version 6.4.z

My upgrade was successful. This result was due in no small part to the pre-upgrade testing and checking available to the system administrator. This step is an important consideration for a mission-critical system such as Satellite.

Post-upgrade checking

The upgrade command itself gives you useful output. This output is found in the log produced by foreman-maintain:

=============================================
Upgrade Step 3/10: katello:clean_backend_objects. This may take a long while.
0 orphaned consumer id(s) found in candlepin.
Candlepin orphaned consumers: []
0 orphaned consumer id(s) found in pulp.
.....
=============================================
Upgrade Step 10/10: katello:upgrades:3.11:clear_checksum_type.
foreman-rake upgrade:run finished successfully!
Upgrade completed!

The log is important and useful, but there are other checks you can do to verify the health of your Satellite server, such as:

# /usr/bin/foreman-maintain health check

The output of this command looks like (edited to fit):

Running ForemanMaintain::Scenario::FilteredScenario
Check for verifying syntax for ISP DHCP configurations: [SKIPPED]
DHCP feature is not enabled
----------------------------------------------------------------
Check whether all services are running: [OK]
----------------------------------------------------------------
Check whether all services are running using hammer ping: [OK]
----------------------------------------------------------------
Check for paused tasks: [OK]
----------------------------------------------------------------

More post-upgrade health checks

Here are three more additional checks to verify the post-upgrade health of your Satellite server. The first checks to see if the Satellite APIs are available and responding:

# /usr/bin/hammer ping
candlepin:
    Status:          ok
    Server Response: Duration: 19ms
candlepin_auth:
    Status:          ok
    Server Response: Duration: 21ms
pulp:
    Status:          ok
    Server Response: Duration: 72ms
pulp_auth:
    Status:          ok
    Server Response: Duration: 36ms
foreman_tasks:
    Status:          ok
    Server Response: Duration: 5ms

The second checks service status with the foreman-maintain command, which checks the main services using systemctl commands:

# /usr/bin/foreman-maintain service status

The output is too verbose to include here, but you will recognize it from using systemctl.

The third command checks that you can synchronize a repository:

Content --> Product --> Repository --> <choose repo to sync>

Conclusion

Upgrading a Red Hat Satellite server is challenging enough when it’s a mission-critical component of your business. Take advantage of all the opportunities you can to verify that the server is in condition to update, and have backups so you can roll back if need be. In my case, I even opened cases with support to discuss the output of the pre- and post-checks and got great help.

My upgrade turned out to be helpful. The load on my Satellite server was slightly reduced, the performance was slightly improved and many old, stuck tasks were cleared. Possibly the most important benefit was gaining increased familiarity with Satellite and the important commands needed to accomplish the task.