VMware edges out Microsoft in virtualization performance test

When the number of CPUs made available to a single virtual machine guest climbed, the cost of virtualization varied more widely. When we allowed a single operating instance SMP access to four vCPUs, the lowest price paid - less than 4 percent - was registered when VMware ESX was supporting a SLES instance. Conversely, the highest operational price paid was a more than 15 percent hit taken when Hyper-V was supporting a SLES instance.

Overall, Hyper-V also loses this round, but by very little when supporting Windows VMs. It falls down more on SLES, likely because of the fact that the LinuxIC kit isn't available to boost performance results.

Testing VMs with business application loads

The second round of performance tests compares iterative VM application performance as VM machines are added to the system. We tracked performance for one, three and six VMs when supporting approved guests. We measured performance when each VM was allocated its own vCPUs and when each was allowed to tap into four vCPUs. This load test would theoretically amplify performance differences.

Our test tool of choice was SPECjbb2005 - a widely used benchmark that mimics distributed transactions in a distribution warehouse-like environment. The SPECjbb2005 test uses Java application components running inside a single host or VM instance. The first component simulates a client generating threads to be processed by the second component, a business logic engine that in turn stores and fetches objects in transactions to/from a set of Java Collection objects (emulating a database engine), logging them through a set of iterative transaction cycles. SPECjbb2005 spawns test parameters it chooses based on the number of CPUs found, as well as the available memory in the host. The measured output is in basic operations per second, or bops per period time with the more bops per test run, the better.

We completed multiple runs with each hypervisor, a set where each VM was allocated its own vCPU and a set where each VM was permitted to tap into four vCPUs.

In both cases, we ran tests with one, three and six VMs. We ran each sequences first with Windows 2008 Server as the hosted operating system and then with SUSE SLES 10.2 as the hosted operating system.

The first round used a ratio of one VM guest operating system per vCPU and limited memory access (2GB) for each operating system instance. This resource allocation is typical of what would happen during a server consolidation process, in which older single-CPU machines are consolidated into a physical-to-virtual re-hosting situation.