What could possibly go wrong when migrating a VM?

Here we list some issues that we have fixed – alongside StackHPC – in a busy OpenStack deployment:

1. Live migration unexpectedly fails when migrating large, busy VMs

This issue was particularly tricky. Large VMs, typically with more RAM than your laptop has disk space, would abort a live migration attempt, just as it was about to complete. Fortunately, the migration would fail back to the source hypervisor, leaving the user none-the-wiser.

When attempting to reproduce the issue, the migration always went smoothly until we realised that the failure only occurred when the VM was under heavy load.

Now that we could reproduce it, we tried upgrading to the latest qemu-kvm release. This didn’t help, but resulted in a slightly different failure scenario, with a mention of the TLS connection being terminated and steam emerging from our ears.

Armed with the knowledge that TLS may somehow be involved, we turned it off for testing. Suddenly the migrations were working again and aside from the lack of encryption, everything was running smoothly. But what was it about transferring a large amount of data over a TLS connection that could cause the failure?

Zooming in on the TLS connection, we were using the most recent version of the protocol, TLS 1.3, but not the most recent release of the library. We tried upgrading and that didn’t help, so we tried downgrading. That didn’t help either, so we got the big hammer out and downgraded the protocol to TLS 1.2. Migrations started succeeding and it felt like we were finally on to something.

Scouring for release notes, we stumbled on Nikos Mavrogiannopoulos’ blog. As one of the developers of GnuTLS, he had written a comprehensive post on the new features arriving with TLS 1.3. The section on re-keying stood out:

“Under TLS 1.3 applications can re-key with a very simple mechanism which involves the transmission of a single message. Furthermore, GnuTLS handles re-key transparently and every GnuTLS client and server will automatically re-key after 2^24 messages are exchanged, unless the GNUTLS_NO_AUTO_REKEY flag is specified in gnutls_init(), or the cipher’s security properties requires no re-keying as in the CHACHA20-POLY1305 cipher.”

Assuming the maximum record size of 16KB, a re-key event would be expected every couple of hundred gigabytes. This tied in nicely with the fact that we only saw the migration issues when moving a comparable amount of data.

Checking the QEMU source, we came across a bug report for a nearly identical issue, confirming our suspicions about the handling of AUTO_REKEY. ^[3]

Meanwhile, the immediate pressure of how to move the large VMs was growing. What could be done that didn’t involve disrupting the users? Even if we recompiled QEMU with AUTO_REKEY disabled, we couldn’t just swap it out. The qemu-kvm processes were already live and running.

The solution turned out to be simple.

A neat feature of GnuTLS is that it supports reading a cryptopolicy from a configuration file, despite having been compiled into an application as a library. ^[4] Being Kolla Ansible, qemu-kvm was confined to a container, so we could simply set the system-wide cryptopolicy in the container to use TLS 1.2 without affecting anything else. Even better, we could choose a secure protocol, and we could avoid the REKEY events. But would the policy take effect on a running VM?

Another neat feature of TLS is version negotiation. During the handshake process, an agreement is made on the cryptographic algorithm and version of the protocol to use. ^[5] This allows older applications to communicate with newer ones, and vice versa.

Taking advantage of this, we rolled out containers using the TLS 1.2 crypto policy to some fresh hypervisors. We then live-migrated the large VMs to these hypervisors. A TLS 1.2 session was mutually agreed and as if by magic, the VMs migrated successfully.

3. A two-part catastrophe with an XFS formatted ephemeral volume

A diligent operator was moving a VM when all of a sudden it failed back to the source hypervisor.  Was it reproducible? The operator gave it another go, and lo and behold it worked. At least it appeared to have worked. The VM started fine on the destination, the host operating system was running sweetly, but what happened to the ephemeral volume? The contents had vapourised, with an empty block device in its place.

Further analysis of the logs revealed some bizarre errors about the filesystem label exceeding 12 characters. An XFS filesystem had already been created on the source host, so why the error when cloning the volume to the destination?

This one turned out to be an OpenStack Nova bug, where Nova was inexplicably elongating the label on the destination. For file systems excluding XFS, the issue was subtle. The label change would only show up when the volume was remounted, for example after a reboot. You can imagine the hair pulling scenarios.

Could these issues have been avoided?

At this stage we will take a step back and reflect on the origin of these issues and how they may have been avoided before affecting end users.

The first issue arises from the complex interaction between two independent software components; GnuTLS and QEMU. Both have highly specialised purposes; the former is a library providing secure communications and the latter is an emulator. It makes complete sense that these are distinct. The problem in this case appears to arise from a new feature, auto re-key, which was turned on by default. There is nothing wrong with this, very much the opposite. The motivation is to improve the baseline security level by reducing the chance of a cryptographic cipher becoming compromised during the transfer of a large amount of data.

The second issue also originates from a software update, this time to the Linux kernel. High performance folk running chunky VMs would have rejoiced at the lifting of the eight queue limit. If you have hundreds of cores contending to send or receive data, a 1:1 mapping of CPUs to TX/RX queues can facilitate a huge performance advantage.

The issue was caused by how the uplift in the queue limit was handled. It’s fine to upgrade the virtual hardware in new VMs, or cold migrate existing workloads to the new world, but live soldering at the sub-device level is a no-no.

The third issue is an obscure bug. Not everyone uses ephemeral storage and not everyone would realise if an ephemeral filesystem label changed. The combined probability of both events is low enough that no-one appears to have noticed.

However, a new level of brokenness occurred when the file system used for ephemeral storage was changed from the default.

The moral here is not to stray from the default path unless there is good reason. This also applies at some level to the fourth issue; not everyone uses a dedicated network for live migration traffic but there are good reasons to do so; particularly in the days when it was more difficult to configure TLS everywhere, or you want to use QoS.

So how could we have prevented these issues affecting users?

We know that defects exist in all software, and so long as the fans are whirring, new challenges will continue to arrive.

The primary defence against new bugs landing in production is the pinning of the software stack at every level. A new release of a package or dependency cannot go live without some human intervention. Coupled with some supporting services, Kolla Ansible is at the core of this. ^[8] However, to remain secure, and state-of-the-art, cloud services must remain current.

Requirements are continually evolving, updates are a necessity and the only way to gain trust in a new release is to thoroughly test it.

At G-Research, this has long been recognised, but the challenge is hard and continually evolving. It is important for test developers to better understand user requirements. The task of providing a secure and reliable cloud computing platform is interminable.