How to set up automatic rebootless kernel updates in Linux server
Patching the kernel on a Linux server seems simple. It can be done using common tools such as dpkg, apt-get, or kexec. These methods, however, become complicated if an organization runs hundreds or thousands of servers. Many servers mean multiple distributions to patch, each of which requires personal attention from a sysadmin or engineer.
These manual-patching methods are also risky in that they require reboots. Reboots involve server downtime, which is always problematic, so they’re usually done in reboot cycles. Because manual patching is done during these cycles, it provides hackers with a “time window” in which they can attack the server infrastructure.
For organizations that run more than a few servers, live patching is a better option. It’s an automated way to patch a Linux kernel while the server is running, which enables it to be both more efficient and more secure than manual methods. Let’s learn how to set up four of the most popular live patching systems from Canonical, Oracle, Red Hat, and CloudLinux.
What is live patching and how it works
Ultimately, there are two methods of live patching for kernels and libraries: temporary and persistent. The temporary method applies a patch without a reboot but actually does require rebooting the server later on. Persistent live patching requires no reboot.
The Temporary Method
The temporary method (or “stack” patching) is executed with package management software (such as the YUM plugin). Patches are delivered to repositories and applied according to the update workflows specified by the user.
“Stack” patching equals server reboots and downtime, even though you may not need a reboot right after you installed the patch, but because of the architecture of this type of live updates, the security patches pile up on top of each other over time, potentially decreasing performance and stability. The only solution to this problem is to reboot the server to load a fresh kernel into memory.
The vendors providing temporary patching are:
The Persistent Method
In the case of a persistent method, a server stores the latest patches and these patches are called “monolithic” as they contain previous patches. To update the servers, an agent program runs in the background, checking the patch server for patches. If there is a patch for a kernel on the patch server, the agent calls patching module and it applies the patch.
Persistent patching has other important advantages:
- Servers using the persistent method remain up and running even with hardware vulnerabilities that usually require reboots to patch, such as Spectre, Meltdown, and Zombieload;
- It reduces the time and effort required to administer servers by complete patching process automation;
- It enables servers to stay up and running, often for years at a time.
The persistent patching method usually involves vendor fees, with free trial periods available from most vendors:
Set up automatic rebootless Kernel updates in Linux server
Below we will show you how to set up rebootless kernel updates in Linux server using Livepatch, Kpatch, Ksplice and KernelCare services.
Note: Before you begin implementing these instructions, make sure your system is up to date and backed up.
1. Setting Up Canonical Livepatch
Canonical Livepatch Service can be set up either during or after installation. It will install kernel security patches only when you run the command apt-get upgrade (hence, semi-automatic).
Pros: Simple. Semi-automatic. No reboot necessary.
Cons: Expensive for 4 or more hosts (but free up to 3 hosts for all and up to 50 machines if you are a member of Ubuntu Community). No patch rollback.
Fees, per server: Monthly (Not available), Yearly ($225).
To install Livepatch on Ubuntu 20.04 LTS Server (also works on 16.04 LTS, 14.04 LTS and 18.04 LTS versions), open a terminal and run these two commands:
sudo snap install canonical-livepatch
sudo canonical-livepatch enable <your key>
To deregister a server, use this command:
sudo canonical-livepatch disable <your key>
To check the status of the service, use this command:
sudo canonical-livepatch status --verbose
2. Setting Up Oracle Ksplice
Unless you’re running an instance of Ksplice inside the Oracle Cloud, you’ll need an access key to install it. This can be obtained by logging into the Unbreakable Linux Network, and following the instructions to register your system for Ksplice.
To install Ksplice, your system must have access to the internet. If you’re using a proxy, set the proxy in your shell:
# export http_proxy=http://proxy.example.com:port
# export https_proxy=http://proxy.example.com:port
The proxy must support making HTTPS connections, and the proxy string should be in this format:
[protocol://][username:password@]<host>[:port]
- protocol is the protocol to connect to the proxy (http or https)
- username and password are the authentication information needed to use your proxy (if any).
- host and port are the hostname/ip address and port number used to connect to the proxy
Run the following instructions as root, replacing YOUR_ACCESS_KEY with the access key you received in the previous step.
Inside the Oracle Cloud
To install Ksplice inside the Oracle Cloud so kernel updates are installed automatically, run these commands:
# wget -N https://ksplice.oracle.com/uptrack/install-uptrack-oc
# sh install-uptrack-oc --autoinstall
To apply available updates to Uptrack, the application that installs kernel updates automatically, run this command:
# uptrack-upgrade -y
If you have already installed Uptrack, you can turn it on by setting autoinstall = yes in /etc/uptrack/uptrack.conf after Ksplice is installed.
To install Ksplice so updates are manually applied, run these commands:
# wget -N https://ksplice.oracle.com/uptrack/install-uptrack-oc
# sh install-uptrack-oc
Outside the Oracle Cloud
To install Ksplice outside the Oracle Cloud so kernel updates are installed automatically, run these commands:
# wget -N https://ksplice.oracle.com/uptrack/install-uptrack
# sh install-uptrack YOUR_ACCESS_KEY --autoinstall
To install Ksplice so updates are manually applied, run these commands:
# wget -N https://ksplice.oracle.com/uptrack/install-uptrack
# sh install-uptrack YOUR_ACCESS_KEY
Note: If you’re installing Ksplice on a Debian or Ubuntu server, you may first need to install the ca-certificates package with apt-get install ca-certificates. Without this package, you will see a "certificate verification error.”
4. Setting Up Red Hat Kpatch
Kpatch installation is simple and straightforward:
Run the update command to update package repositories and get the latest package information:
sudo apt-get update -y
Run the install command with the -y flag to quickly install the packages and dependencies:
sudo apt-get install -y patch
5. Setting Up CloudLinux KernelCare
To see if the running kernel is supported by KernelCare, run either of these commands:
curl -s -L https://kernelcare.com/checker | python
or
wget -qq -O – https://kernelcare.com/checker | python
To install KernelCare, run either of these commands:
curl -s -L https://kernelcare.com/installer | bash
or:
wget -qq -O - https://kernelcare.com/installer | bash
If you’re using an IP-based license, nothing else is required. If you’re using a key-based license, run this command:
$ /usr/bin/kcarectl --register KEY
KEY is the registration keycode string you received when you bought KernelCare or signed up for a free trial. You can get a key here.
To deregister a server, run:
sudo kcarectl --unregister
To check the status of the service, run:
sudo kcarectl --info
KernelCare automatically checks for new patches every 4 hours. To perform updates manually instead of automatically, run:
/usr/bin/kcarectl –update
Conclusion
These installation instructions for several live patching solutions list all the steps needed to install one in your environment. Once that’s done, you’ll enjoy the benefits of live patching technology: being able to update the kernel without stopping the server, with no subsequent reboots needed for months, or even years.
