Meta (formerly Facebook) engineers shared how they use the Linux kernel at this week’s Linux Plumbers conference kernel live-patching (KLP) infrastructure.

According to reports, in addition to using the KLP that comes with the kernel, Meta also chose Red Hat’s Kpatch solution. They hot-patch the kernel for “millions of Linux servers” via Kpatch. Hotpatches can also fix kernel functionality while the server is running.

The reason for choosing KLP, Meta said, was mainly to reduce server downtime during kernel updates — in response to a never-ending stream of security updates. After all a full server restart and lengthy POST times can be quite problematic, and with hotpatching, servers can migrate to the new kernel almost seamlessly when everything goes according to plan.

While using the Linux kernel’s hot-patching feature, Meta engineers found some tracking issues to overcome and also encountered performance issues. The specific performance issue is,There will be 1~2 seconds of problems during hot patching, such as higher I/O and fsync latencyand a higher TCP retransmission rate.

The Meta engineers also mentioned that they are actively working on some boundary issues to better handle things like kernel builds compiled with Clang and optimized with PGO, as well as other projects to improve robustness.

Check out the PPT or speech video for details.