SUSE announced kGraft at the beginning of February while Red Hat made its pitch about kpatch towards the end of the same month. The patches have been submitted upstream for incorporation in the kernel – both companies have excellent records of sticking to the principles of open source.

To get an idea of the way the two processes work, iTWire asked both SUSE and Red Hat the following questions:

How does the process of patching differ between kpatch and kGraft?

Are there any features in kGraft/kpatch that you would like to incorporate into kpatch/kGraft?

Red Hat's Ron Pacheco, senior manager of technology product management, responded: "We can't speak to the nuances of kGraft, but the kpatch process works in the following manner:

"Analysis of the code change, which includes determining the patch candidates and an object-level comparison of the kernel pre-patch;

"Patch conversion, the point at which two extra sections are added to the object file to address the base function and the patched function; and

"Insertion of the patched entity.

"Ultimately, the kpatch process is heavily weighted towards step 1, analysing the code, in order to minimise and mitigate as much risk as possible during the patching process.

Regarding the plus points of kpatch vs kGraft, Pacheco said: "From a benefits standpoint, first and foremost, kpatch, like kGraft, is a completely open source solution and works with existing upstream infrastructures and functionalities.

"kpatch emphasises what we feel is the most important part of the patching process: the pre-patch analysis. kpatch provides the functionality to provide a full deep dive into the code change, picking out seemingly minor inconsistencies and issues that could have massive ramifications post-patch, as well as helping determine whether or not a patch function is even suitable for dynamic patching.

Pacheco added: "We want to ensure that IT teams understand the implications, limitations and ramifications of a suggested patch BEFORE they start the process, and kpatch delivers the tools to help answer these questions and eliminate human error."

In response to the first query, the director of SUSE Labs, Vojtech Pavlik, said: "As the goal is the same for both technologies, also the process is similar and at some points even identical. Both use ftrace for patching code, both use patches in a kernel module format, both use the kernel module linker functions to link patched code into the kernel. There are two points, where the process differs, though:

"1) Automated patch generation: kGraft starts with a list of changed functions and uses the DWARF debugging information of the old kernel package to generate a full list of functions that are in any way affected by changes in the patch. kGraft then compiles the post-patch kernel, same as kpatch.

"kpatch recompiles both the pre-patch and post-patch kernel sources with special compiler options (-ffunction-sections) and then compares the resulting object files, arriving at a list of differences.

"2) Patching process: kGraft patches at runtime and uses a RCU-like method of transitioning individual execution threads from pre-patch to post-patch functions that ensures consistency of function call format and semantics. It may take some time to transition all threads, as some processes may be idle and not reaching the transition point (kernel/userspace boundary) soon. Nevertheless, the kernel is protected from the patched bug already during the transition process as any new thread will start using post-patch functions automatically.

"kpatch stops all the processors in the system to make sure no thread is executing. It then checks whether any execution thread was stopped in a function that is to be patched or in any function that is called from a function to be patched by analysing stack return addresses. When that happens, kpatch has to resume all processors again, wait, and try later. If after trying several times it doesn't succeed, perhaps because an idle process is sleeping in a patched function, it fails.

Regarding what SUSE considered to be better in its solution, Matthias Eckermann, senior product manager for SUSE Linux Enterprise, said: "kGraft based patching does not require the kernel to be stopped, while all other methods require this for milliseconds up to seconds.

"kGraft is lightweight and builds on proven kernel infrastructure, namely ftrace'" he added. "For the use cases our customers are asking for, kGraft is just perfect. We are watching the other development paths (kpatch) carefully, yet for the moment our focus is on improving the quality of our mainline submission of the kGraft framework."

On the same question, Pavlik said: "Matthias already mentioned the major plus points - not having to stop the kernel; kGraft patches at runtime, while kpatch has to stop the kernel, potentially several times, for very short time periods, until it succeeds patching and never failing to patch.

"In a subset of scenarios, where kpatch would fail, kGraft may never finish the RCU transition process. This is not an issue, though, as the kernel is protected against the patched bug from the start of the patching process already.

"Less reliance on sane compiler behaviour: Recompiling the pre-patch kernel with additional options should not, but may affect compiler optimisation choices and thus render the object comparison invalid.

"To be fair, I'd like to also mention a plus point for kpatch: easier data format/semantics conversion.

"When the in-memory data format changes as a result of a patch, it is easier to do the surgery on the data while the kernel is standing still. Doing it on a running kernel requires more care to be applied by the programmer writing the conversion code. This code has to be created manually for both kpatch and kGraft."

Pavlik said that rather than incorporate features from kpatch, it was more beneficial to look at the code that was very similar and share it, "so that is what we're looking into now".

He said he had some thoughts about possible future synergies in places where the technologies differed, too.

"kpatch could continue with the kGraft method when it would otherwise fail," Pavlik said. "kGraft could resort to stopping the kernel for certain cases where in-memory data conversion has to be done and doing it at runtime isn't worth the additional developer effort."

He added: "kGraft could use the stack analysis part of the kpatch method to speed up the transition considerably and not be delayed by most idle sleeper processes."

The image used with this article is copyrighted to Larry Ewing, Simon Budig and Anja Gerwinski.