vSphere, Hardware Version 7 and Hot Plug

VMware's vSphere added hot plug features in hardware version 7 (first introduced in VMware Workstation 6.5) that were not available in the earlier version 4 virtual hardware. Virtual hardware version 7 adds the following new features to VMware virtual machines:

• LSI SAS virtual device - provides support for Windows Server 2008 fail-over cluster configurations


• Paravirtual SCSI devices - recently updated to allow booting, can allow higher-performance (greater throughput and lower CPU utilization) than the standard virtual SCSI adapter - especially in SAN environments where I/O-intensive applications are used. Currently supported in Windows Server 2003/2008 and Red Hat Linux 5 - although any version of Linux could be modified to support PVSCSI.


• IDE virtual device - useful for older OSes that don't support SCSI drivers


• VMXNET 3 - next generation Vmxnet device with enhanced performance and enhanced networking features.


• Hot plug virtual devices, memory and CPU - supports hot add/remove of virtual devices, memory and CPU for supported OSes.

While the "upgrade" process from version 4 to version 7 is well-known, some of the side effects are not well publicised. The most obvious change after the migration from version 4 to version 7 is the affect hot plug has on the PCI bus adapters - some are now hot plug by default, including the network adapters!

[caption id="attachment_1342" align="aligncenter" width="417" caption="Safe to remove network adapters. Really?"][/caption]

Note that the above example demonstrates also that the updated hardware re-enumerates the network adapters (see #3 and #4) because they have moved to a new PCI bus - one that supports hot plug. Removing the "missing" devices requires a trip to device manager (set devmgr_show_nonpresent_devices=1 in your shell environment first.) This hot plug PCI bus also allows for an administrator to mistakenly remove the device from service - potentially disconnecting tier 1 services from operations (totally by accident, of course.

[caption id="attachment_1340" align="aligncenter" width="450" caption="Devices that can be added while the VM runs with hardware version 4"][/caption]

In virtual hardware version 4, only SCSI devices and hard disks were allowed to be added to a running virtual machine. Now with hardware version 7,

[caption id="attachment_1341" align="aligncenter" width="450" caption="Devices that can be added while the VM runs with hardware version 7"][/caption]

additional devices (USB and Ethernet) are available for hot add. You could change memory and CPU on the fly too, if the OS supports that feature and they are enabled in the virtual machine properties prior to running the VM:

[caption id="attachment_1339" align="aligncenter" width="450" caption="CPU and Memory Hot Plug Properties"][/caption]

However, the hot plug NIC issue isn't discussed in the documentation, but Carlo Costanzo at VMwareInfo.com passes on Chris Hahn's great tip to disable hot plug behaviour in his blog post complete with visual aids. The key is to add a new "Advanced Configuration Parameter" to the virtual machine configuration: this new parameter is called "devices.hotplug" and its value should be set to "false." However, adding this parameter requires the virtual machine to be turned-off, so it is currently an off-line fix.