Based on numerous VMware vSphere ESXi installations at various clients with differing configurations, I've developed a top 10 list of tips that an ESXi administrator can use to increase the performance of virtual machines (VMs). Hopefully you can use some of these tips to get the best possible performance on your ESXi VMs.
Related: Integrating vSphere and Hyper-V
Tip #1: ESXi Host Memory and Memory Overcommit
The memory overcommit feature in ESXi lets you allocate more memory to VMs running on an ESXi host than the actual amount of physical memory that's installed on that host. Memory overcommit can be useful when you need to get that last VM running in an emergency, but in general, try to avoid using this feature. Many VM performance issues can be traced to not allocating enough memory to a VM or to overcommitting the memory on the ESXi host.
Related: VMware Expands Cloud Management Offering
If using the memory overcommit feature on a standalone ESXi host is a poor idea, it's a potential train wreck when the ESXi host is connected to a cluster. We design ESXi clusters for N-1 hosts. If one host goes down, can the remaining ESXi hosts handle the load? Let's assume that you have a two-node ESXi cluster and each ESX host needs 32GB of memory to avoid memory overcommit, based on the VMs that are running on each host. If one host fails, then all the VMs that were running on that host will automatically start on the remaining host -- if you properly configured your high-availability cluster.
In this example, you actually need 64GB on each host to avoid memory overcommit when one host goes down. If you have only 32GB of memory on each host and one of the hosts fails, then the performance of all the VMs on the remaining host will suffer because the remaining ESXi host will be 50-percent memory overcommitted.
Whenever I install a new ESXi cluster, I use vMotion to migrate all the VMs off one host to another ESXi host (or hosts). Then I listen for the silence. In other words, no one should complain or even notice that one of the ESXi hosts was shut down. This verifies that the design is solid and can meet the computing needs of the client, even when one ESXi host fails. I suggest purchasing an ESXi host that can hold at least 256GB of memory for future expansion.
I get prices for the individual DIMMs and look for the point at which the price/DIMM density becomes nonlinear. For example, Table 1 lists approximate prices of Double Data Rate 3 (DDR3) DIMMs for the HP ProLiant DL380 G7 Server. The prices are relatively linear up through the 8GB DIMM. However, the price difference between 8GB and 16GB is significant. For example, if I need an ESXi host with 96GB of memory, I would purchase twelve 8GB DIMMs ($2,040) rather than six 16GB DIMMs ($2,880). I use this purchasing strategy even though I know that I might need to replace the 8GB DIMMs in the future to reach the maximum capacity of the server. (By the time I need the memory upgrade, the 16GB DIMMs will probably be less expensive than the 8GB DIMMs at today's prices.)
Table 1: Pricing Levels per DIMM
Several clients have reminded me that when you increase the memory in a server over a certain amount -- 96GB with the DL380 G7 -- the memory speeds drop. For example, if you go up to the maximum of 144GB in the DL380 G7 host, speeds drop from 1,333MHz to as low as 800Mhz. Although this is true, I argue that not having the memory will have a greater negative effect than slowing down the memory speeds. Random memory access is approximately 100,000 times faster than disk access. So even with the drop in memory speeds, 800MHz memory is still roughly 60,000 times faster than disk access.
For example, I just helped perform a Microsoft Exchange Server 2010 migration for a client. All the Exchange servers are VMs in an ESXi cluster. To handle the additional load of Exchange 2010, we needed to increase the memory in each host by 96GB, from 64GB to 160GB. With this configuration, Exchange 2010 is fast and stable. No one noticed that the memory speeds dropped to 800MHz, but everyone would have noticed if we hadn't installed the additional memory prior to the Exchange 2010 migration. The bottom line: Make sure that you have enough memory in your ESXi host, and avoid using the memory overcommit feature for the best possible performance.
Tip #2: ESXi Host CPU Cores
Just as you can never be too rich or too thin, you can't have too many CPU cores on your ESXi host. With vSphere 5 Enterprise Plus, you can configure a VM with as many as 32 virtual CPUs (vCPUs). CPU clock speed on the host doesn't matter as much as the number of cores. If I had to choose between a faster clock speed and more CPU cores, I'd select the latter.
vSphere is licensed by socketed CPU, so it makes sense to get a CPU with as many cores as possible. You might be able to configure an ESXi host with fewer physical CPUs, which you'll probably save on vSphere licensing costs. CPU cores are especially important when you have VMs that will run CPU-intensive applications such as SSL encryption or Microsoft SQL Server.
Tip #3: VM Memory
It's important to configure each VM with the proper amount of memory. Exchange 2010 and Exchange 2007 are a few of the more memory-hungry applications. For a 10-user Exchange server running the Client Access Server, Hub Transport Server, and Mailbox Server roles and management tools, the server requires about 16GB of memory! A 300-user Exchange server with the same roles requires about 72GB of memory for optimal performance. Know the memory requirements of your applications, and allocate the appropriate amount of memory to each VM without overcommitting the memory on the ESXi host.
The x64 Windows platform can natively address up to 2TB of memory. If your VM is running an x64 OS, then you can somewhat compensate for poor disk performance on a VM by allocating more memory to the VM and caching the data. Of course, if an ESXi host is already memory-overcommitted and you increase the amount of memory allocated to a VM, you'll probably make the VM slower, not faster.
Tip #4: Number and Configuration of vCPUs on a VM
When calculating the number and configuration of vCPUs on a VM, it's important to know the application running on each VM. If the application is SMP-aware, then increasing the number of vCPUs on the VM should increase its performance -- as long as you've allocated an appropriate amount of memory to the VM and as long as the ESXi host isn't memory-overcommitted.
With vSphere 5, you can now specify both the number of vCPUs and the number of cores on each vCPU. For example, Exchange 2007 isn't an SMP application. If you determine that an Exchange 2007 server is CPU-bound and running on a vSphere 5 ESXi host, you can increase the number of cores on the vCPU while keeping only one vCPU. If the VM is running Exchange 2010, which is an SMP application, you'd increase the number of vCPUs, each with a single core, for the best performance increases. Of course, your mileage might vary, so perform some tests to determine the optimal vCPU/core configuration for your environment.
Tip #5: Paravirtual SCSI Driver
If you're running in an ESXi cluster connected to a Fibre Channel or iSCSI SAN, you can get better disk throughput (10 to 25 percent) at a given level of CPU performance on the host. You'll see this benefit only when the hosts are connected to a SAN, not DAS. The VMware Paravirtual SCSI driver does have some limitations:
- The Paravirtual SCSI driver is supported only on VMs running Windows Server 2008 R2, 2008, 2003 R2, or 2003, or Red Hat Linux 5.
- The VM must run VM hardware version 7 or later.
- Fault-tolerant VMs can't use the Paravirtual SCSI driver.
Tip #6: VM Snapshots
Snapshots are a great tool that can quickly get you out of a bind when an upgrade or patch goes sideways. As a general rule, make a snapshot of a VM just before you perform any upgrades or patches. If the upgrade goes smoothly, you can incorporate the changes from the snapshot or delta file into the base image of the VM. Otherwise, you can revert back to the state of the VM prior to the snapshot.
VMs with active snapshots write any changes to a differencing disk or delta file while the snapshot is active. If you leave a snapshot active on a VM for a long time and multiple changes are made to the VM, the delta file can grow very large; you can even run out of space in the storage group. For this reason, I suggest that you snap a VM, perform the upgrade, verify that everything is working, and then remove the snapshot from the VM. Don't leave the snapshot on the VM active just because you might need it. Of course, an active snapshot also slows down the performance of the VM. This performance hit is most noticeable when the ESXi host is already heavily loaded. Unnecessary active snapshots on a VM hurt VM performance and increase the chances of running out of disk space on the VM.
Tip #7: iSCSI SANs and Jumbo Frames
The Maximum Transmission Unit (MTU) default frame size is 1,500 bytes. Enabling jumbo frames on an iSCSI SAN increases the MTU to 9,000 bytes. This increase allows more data to be transmitted in each packet, boosting performance by 5 to 15 percent on an iSCSI SAN. The benefit tends to be greater on 10 Gigabit Ethernet iSCSI SANs versus 1 Gigabit Ethernet iSCSI SANs.
Of course, you must make sure that you enable jumbo frames on every device that's connected to the SAN fabric. These devices can include the SAN controllers, SAN switches, ESXi NICs, and firewalls that connect to the SAN fabric. If you miss just one device on your iSCSI SAN, you'll probably decrease its performance because of dropped frames. You can use the Iometer utility to determine the performance increase before and after enabling jumbo frames on an iSCSI SAN. If you experience a performance drop, you probably missed one or more devices in your SAN fabric.
Tip #8: SAN Infrastructure
Always use a dedicated network for your SAN fabric. A few years ago, Fibre Channel over Ethernet (FCoE) was all the rage, but it never really gained popularity. In my opinion, the best cost/performance SAN solution is 10 Gigabit Ethernet iSCSI. It has better performance than even 8 Gigabit Ethernet Fibre Channel, and at a lower cost.
Tip #9: Disk and RAID Configuration and SSDs
With vSphere 5, you can now designate a storage group that comprises solid state disk (SSD) drives. If an SSD storage group has been defined, vSphere 5 uses it for memory page swapping. Ideally, you should have enough memory in the ESXi host to avoid page swapping, but if your server already has the maximum memory installed, this option might be interesting. The performance won't be as good as native memory on the host but will still be better than swapping to non-SSD storage on the host. As a general rule, try to avoid using RAID 6 for storage groups, especially in write-intensive environments. RAID 6 has two copies of parity, so writes are very expensive. If you want the additional fault tolerance of RAID 6 without taking a performance hit when writing to the disk, configure a RAID 5 array with one or more hot spares.
Another possible application of SSD drives provides extremely fast performance for x86 VMs that require fast disk access. For example, a client has a VM that's running an x86 version of SQL Server, which can't be upgraded for another year. In an x64 environment, I typically load up the VM with a lot of memory and cache the entire database. But an x86 OS can natively address only 4GB of memory, and the x86 version of SQL Server can address only 2GB. In this situation, the client is going to store this x86 VM on a storage group made from SSD drives, for a significant gain in disk performance.
Tip #10: Thin vs. Thick Disk Provisioning
When you create a VM, you have the option of thin or thick provisioning of the VM's hard disks. Thin provisioning initially creates a small .vmdk file. This file starts out using only the space that's stored on the disk and grows to the provisioned size. Thick provisioning creates a file on the ESXi host; this file is the same size as the disk size, regardless of what's initially stored on the disk. Thin-provisioned disks can save space on the storage group. However, you have a greater probability of quickly running out of disk space on the storage group, and the performance of the disk can suffer because of potential disk fragmentation. When the storage group is on a SAN, your results can vary because some SAN vendors actually use thin provisioning, even when the disks are thick-provisioned on a VM. As a general rule, however, I suggest thick provisioning disks. You'll probably get better performance than with a thin-provisioned disk and will reduce the probability of running out of disk space on a storage group.
A Happy Tune
These performance tips are highly dependent upon factors in your environment. Getting the memory configuration right on both the ESXi host and the VM is the most crucial factor for VM performance. Happy performance tuning!
