OTC Patching March
February has seen retpoline updates for system compilers, kernels and KVM from SUSE, Debian, Ubuntu. New public images have been published by the OTC ImageFactory team.
After the microcode withdrawal from intel, intel has continued to work on stable microcode, this time with more involvement from hardware partners to validate the changed microcode in all environments.
While most Linux kernels and latest KVM releases do rely on retpoline and only require minimal IBPB support from the microcode on Broadwell and newer CPUs for full protection, Xen and Windows still fully rely on the IBC (IBRS,IBPB,STBIP) mitigations provided by the microcode updates.
Microcode Situation
intel was able to reissue the same microcode for some CPU types, while other CPU types have received incremental fixes and have been determined to be stable now by intel and hardware partners.
intel has meanwhile published this new set of microcode updates on 2018-03-12, so Linux distributors can distribute these (in addition for hardware vendors including the updates in their BIOS/Firmware).
More Retpolines Updates
We have been shipping retpoline enabled images since early February.
In early March, also RedHat started shipping retpoline enabled kernels.
Huawei is still testing and comparing their retpoline patches for EulerOS. We expect updated EulerOS images with retpoline enablement in April.
OTC Microcode Patching
We are finishing our validations of the intel provided microcode on the CPU types that were not patched before, i.e. on the v4 (Broadwell) CPUs and E7-88x0v3 (Haswell) CPUs.
After the validation in our reference environments is complete, we will roll out to production. This will happen according to the following schedule.
OTC IBC Exposure
While the OTC team was very fast in implementing IBC based mitigation for the hypervisors in early January, the IBC features were not exposed to the guest VMs. This means that the guest kernels remain vulnerable to information disclosure Spectre-v2 attacks by userspace if they rely on CPU IBC features.
At the time of writing, Windows relies on the microcode IBC features to mitigate Spectre-v2, while all major Linux distributors as well upstream Linux has migrated over to rely on retpolines.
Our partner Huawei is currently testing the hypervisor patches that enable the exposure of the IBC features to the guest VMs.
OTC KVM Update
The initial protection of both Xen and KVM hypervisors against Spectre-v2 relied on the OBC features. For KVM, at lest on older CPUs, it is possible to use retpolines for most situations that need to be mitigated; as for the kernel, performance is better for retpolines, thus this mitigation is preferable eve on CPUs where stable IBC-providing microcode is available.
We have been able to live-patch Xen and KVM to enable IBC based protection. Unfortuately, it is not really feasible to do retpoline enablement using a live patch. Instead we will reboot the hosts with KVM. For normal flavors (s2), we will do large-scale live migrations to avoid trouble for our customers with rebooted VMs. However, the KVM flavors with hardware-passthrough acceleration can not be live-migrated. So we will announce the reboot to our customers in time, so they can prepare. (Of course we will do the reboot AZ by AZ to ensure that cloud-aware applications can run without interruption.)
Note that there is curretly no retpoline based replacement for the Spectre-v2 protection mechanisms for Xen -- so the Xen based flavors will continue to rely on IBC protection for the time being, incuring the larger performance impact.
OTC PaaS images
OTC management zone
Performance impact
We refer to our benchmarking page here.