VMware vSphere Update 2, View 4.x Issues

Right on the heels of vSphere 4 Update 2, a flurry of twitter and blog postings suggestion caution if upgrading a VMware View 4.x environment. Now, VMware has updated its KB to include a workaround for those needing to move to Update 2 for the feature and hardware support benefits but can't afford to break their PCoIP-based View environment: in short, it's all VMware Tools fault! Seems the SVGA drive in VMware Tools for Update 2 has a problem with PCoIP - preventing desktops from connecting.

Now that makes some sense given the fact that VMware is quite adamant about the ORDER in which VMware Tools, .NET framework and the View Agent must be installed. The current options for PCoIPers out there needing Update 2 are three-fold:

1. Don't upgrade VMware Tools;


2. Upgrade VMware Tools with "Advanced Options" including
   
   
   
   1. For x86 guests:
      /s /v "/qn REINSTALLMODE=voums REINSTALL=WYSE,VMXNet,
      VMCI,Mouse,MemCtl,Hgfs,VSS,VMXNet3,vmdesched,ThinPrint,Sync,
      PVSCSI,Debug,BootCamp,Audio,Buslogic,VICFSDK,VAssertSDK,
      Toolbox,Upgrader,GuestSDK,PerfMon,Common,microsoft_dlls_x86,
      ArchSpecific"
      (line breaks included for formatting only)
   
   
   2. For x64 guests:
      /s /v "/qn REINSTALLMODE=voums REINSTALL=WYSE,VMXNet,
      VMCI,Mouse,MemCtl,Hgfs,VSS,VMXNet3,vmdesched,ThinPrint,Sync,
      PVSCSI,Debug,BootCamp,Audio,Buslogic,VICFSDK,VAssertSDK,Toolbox,
      Upgrader,GuestSDK,PerfMon,Common,microsoft_dlls_x64,ArchSpecific"
      (line breaks included for formatting only)
   
   
   
   


3. Use RDP (not really an option for PCoIP environments)

If you're already STUNG by the VMware Tools, the KB has a procedure for back-tracking the SVGA driver after VMware Tools for Update 2 has been installed. Here it is:

For customers that are experiencing the issue and want to continue using PCoIP, use this workaround:

Note: VMware recommends that you have VMware View 4.0.1 installed prior to performing these steps.

1. Log in to the affected virtual machine(s) with Administrator rights. This is any virtual machine that has the VMware Tools shipped with ESX 4.0 Update 2 installed.


2. Rename the C:\Program File\Common Files\VMware\Drivers\Video folder to C:\Program File\Common Files\VMware\Drivers\Video-OLD.


3. From a working virtual machine, copy the C:\Program File\Common Files\Vmware\Drivers\Video folder to C:\Program File\Common Files\Vmware\Drivers\ on an affected virtual machine.


4. Click Start > Settings > Control Panel > System.


5. Click the Hardware tab and click Device Manager.


6. Expand Display Adapters.


7. Choose VMWARE SVGA II. Right-click on it and choose Properties.


8. Click the Driver tab. You are presented with the SVGA driver properties.
   
   The version is 11.6.0.34 (Driver Date 01/03/2010).


9. Click Update Driver. The Hardware Update wizard displays.


10. When prompted with the question, "Can windows connect to Windows Update to search for software?", click No, not at this time and click Next.


11. When prompted with the question, "What do you want the wizard to do", click Install from a list or specific location (Advanced) and click Next.


12. Click Don't search. I will choose the driver to install and click Next.


13. On the next screen, you are presented a list of two or more VMWARE SVGA II Versions to choose from. For example:
    
    VMWARE SVGA II Version: 11.6.0.31 (21/09/2009)
    VMWARE SVGA II Version: 11.6.0.32 (24/09/2009)
    VMWARE SVGA II Version: 11.6.0.34 (01/03/2010)
    
    Choose VMWARE SVGA II Version: 11.6.0.32 (24/09/2009).


14. Click Next. The driver begins installing.


15. Click Continue Anyway when Windows notifies you that VMware SVGA II has not passed Windows logo testing.
    
    The driver install completes.


16. Click Finish.


17. Click YES when prompted to restart your computer.


18. Verify that PCoIP is now working.

Note: VMware is investigating a permanent fix for the issue so our customers can upgrade to ESX 4.0 Update 2 without experiencing this issue.

- VMware KB1022830, 6/17/2010

In-the-Lab: Install VMware Tools on NexentaStor VSA

Physical lab resources can be a challenge to "free-up" just to test a potential storage appliance. With NexentaStor, you can download a pre-configured VMware (or Xen) appliance from NexentaStor.Org, but what if you want to build your own? Here's a little help on the subject:

1. Download the ISO from NexentaStor.Org (see link above);


2. Create a VMware virtual machine:
   
   
   
   1. 2 vCPU
   
   
   2. 4GB RAM (leaves about 3GB for ARC);
   
   
   3. CD-ROM (mapped to the ISO image);
   
   
   4. One (optionally two if you want to simulate the OS mirror) 4GB, thin provisioned SCSI disks (LSI Logic Parallel);
   
   
   5. Guest Operating System type: Sun Solaris 10 (64-bit)
   
   
   6. One E1000 for Management/NAS
   
   
   7. (optional) One E1000 for iSCSI
   
   
   
   


3. Streamline the guest by disabling unnecessary components:
   
   
   
   1. floppy disk
   
   
   2. floppy controller (remove from BIOS)
   
   
   3. primary IDE controller (remove from BIOS)
   
   
   4. COM ports (remove from BIOS)
   
   
   5. Parallel ports (remove from BIOS)
   
   
   
   


4. Boot to ISO and install NexentaStor CE
   
   
   
   1. (optionally) choose second disk as OS mirror during install
   
   
   
   


5. Register your installation with Nexenta
   
   
   
   1. http://www.nexenta.com/register-eval
   
   
   2. (optional) Select "Solori" as the partner
   
   
   
   


6. Complete initial WebGUI configuration wizard
   
   
   
   1. If you will join it to a domain, use the domain FQDN (i.e. microsoft.msft)
   
   
   2. If you choose "Optimize I/O performance..." remember to re-enable ZFS intent logging under Settings>Preferences>System
      
      
      
      1. Sys_zil_disable = No
      
      
      
      
   
   
   
   


7. Shutdown the VSA
   
   
   
   1. Settings>Appliance>PowerOff
   
   
   
   


8. Re-direcect the CD-ROM
   
   
   
   1. Connect to Client Device
   
   
   
   


9. Power-on the VSA and install VMware Tools
   
   
   
   1. login as admin
      
      
      
      1. assume root shell with "su" and root password
      
      
      
      
   
   
   2. From vSphere Client, initiate the VMware Tools install
   
   
   3. cd /tmp
      
      
      
      1. untar VMware Tools with "tar zxvf  /media/VMware\ Tools/vmware-solaris-tools.tar.gz"
      
      
      
      
   
   
   4. cd to /tmp/vmware-tools-distrib
      
      
      
      1. install VMware Tools with "./vmware-install.pl"
      
      
      2. Answer with defaults during install
      
      
      
      
   
   
   5. Check that VMware Tools shows and OK status
      
      
      
      1. IP address(es) of interfaces should now be registered
         
         [caption id="attachment_1580" align="aligncenter" width="300" caption="VMware Tools are registered."][/caption]
      
      
      
      
   
   
   
   


10. Perform a test "Shutdown" of your VSA
    
    
    
    1. From the vSphere Client, issue VM>Power>Shutdown Guest
       
       [caption id="attachment_1581" align="aligncenter" width="300" caption="System shutting down from VMware Tools request. "][/caption]
    
    
    2. Restart the VSA...
       
       [caption id="attachment_1582" align="aligncenter" width="300" caption="VSA restarting in vSphere "][/caption]
    
    
    
    

Now VMware Tools has been installed and you're ready to add more virtual disks and build ZFS storage pools. If you get a warning about HGFS not loading properly at boot time:

[caption id="attachment_1586" align="aligncenter" width="300" caption="HGFS module mismatch warning."][/caption]

it is not usually a big deal, but the VMware Host-Guest File System (HGFS) has been known to cause issues in some installations. SInce the NexentaStor appliance is not a general purpose operating system, you should customize the install to not use HGFS at all. To disable it, perform the following:

1. Edit "/kernel/drv/vmhgfs.conf"
   
   
   
   1. Change:     name="vmhgfs" parent="pseudo" instance=0;
   
   
   2. To:     #name="vmhgfs" parent="pseudo" instance=0;
   
   
   
   


2. Re-boot the VSA

Upon reboot, there will be no complaint about the offending HGFS module. Remember that, after updating VMware Tools at a future date, the HGFS configuration file will need to be adjusted again. By the way, this process works just as well on the NexentaStor Commercial edition, however you might want to check with technical support prior to making such changes to a licensed/supported deployment.

NexentaStor CIFS Shares with Active Directory Authentication

Sharing folders in NexentaStor is pretty easy in Workgroup mode, but Active Directory integration takes a few extra steps.  Unfortunately, it's not (yet) as easy as point-and-click, but it doesn't have to be too difficult either. (The following assumes/requires that the NexentaStor appliance has been correctly configured-in and joined-to Active Directory.)

[caption id="attachment_1562" align="alignright" width="170" caption="Typical user and group permissions for a local hard disk in Windows."][/caption]

Let's examine the case where a domain admin group will have "Full Control" of the share, and "Everyone" will have read/execute permissions. This is a typical use case where a single share contains multiple user directories under administrative control. It's the same configuration as local disks in a Windows environment. For our example, we're going to mimic this setup using a CIFS share from a NexentaStor CE appliance and create the basic ACL to allow for Windows AD control.

For this process to work, we need to join the NexentaStor appliance to the Active Directory Domain. The best practice is to create the machine account in AD first, assign control user/group rights (if possible) and then attempt to join. It is IMPORTANT that the host name and DNS configuration of the NexentaStor appliance match domain norms, or things will come crashing to a halt pretty quickly.

That said, assuming that your DC is 1.1.1.1 and your BDC is 1.1.1.2 with a "short" domain of "SOLORI" and a FQDN of "SOLORI.MSFT" your NexentaStor's name server configuration (Settings->Network->Name Servers) would look something like this:

This is important because the AD queries will pull service records from the configured domain name servers. If these point to an "Internet" DNS server, the AD entries may not be reflected in that server's database and AD authentication (as well as join) will fail.

The other way the NexentaStor appliance knows what AD Domain to look into is by its own host name. For AD authentication to work properly, the NexentaStor host name must reflect the AD domain. For example, if the FQDN of your AD domain is "SOLORI.MSFT" then your domain name on the appliance would be configured like this (Appliance->Basic Settings->Domainname):

The next step is to create the machine account in AD using "Active Directory Users and Computers" administrator's configuration tool. Find your domain folder and right-click "Computers" - select New->Computer from the menu and enter the computer name (no domain). The default user group assigned to administrative control should be Domain Admins. Since this works for our example, no changes are necessary so click "OK" to complete.

Now it's time to join the AD domain from NexentaStor. Any user with permissions to join a machine to the domain will do. Armed with that information, drill down to Data Management->Shares->CIFS Server->Join AD/DNS Server and enter the AD/DNS server. AD server, AD user and user password into the configuration box:

If your permissions and credentials are good, your NexentaStor appliance is not a member of your domain. As such, it can now identify AD users and groups by unique gid and uid data created from AD. This gid and uid information will be used to create our ACLs for the CIFS share.

To uncover the gid for the "Domain Admins" and "Domain Users" groups, we issue the following from the NexentaStor NMC (CLI):

nmc@san01:/$ idmap dump -n | grep "Domain Admins"

wingroup:Domain Admins@solori.msft     ==      gid:3036392745

nmc@san01:/$ idmap dump -n | grep "Domain Users"
wingroup:Domain Users@solori.msft     ==      gid:1238392562

Now we can construct a CIFS share (with anonymous read/write disabled) and apply the Domain Admin gid to an ACL - just click on the share, and then click "(+) Add Permissions for Group":

[caption id="attachment_1568" align="aligncenter" width="300" caption="Applying administrative permissions with the AD group ID for Domain Admins."][/caption]

We do similarly with the Domain User gid:

[caption id="attachment_1569" align="aligncenter" width="300" caption="Applying the Domain User gid to CIFS share ACL."][/caption]

Note that the "Domain Users" group gets only "execute" and "read" permissions while the "Domain Admins" group gets full control - just like the local disk! Now, with CIFS sharing enabled and the ACL suited to our AD authentication, we can access the share from any domain machine provided our user is in the Domain Users or Admins group.

Administrators can now create "personal" folders and assign detailed user rights just as they would do with any shared storage device. The only trick is in creating the initial ACL for the CIFS share - as about - and you've successfully integrated your NexentaStor appliance into you AD domain.

NOTE: If you're running Windows Server 2008 (or SBS 2008) as your AD controller, you will need to update the share mode prior to joining the domain using the following command (from root CLI):

# sharectl set -p lmauth_level=2 smb

NOTE: I’ve also noticed that, upon reboot of the appliance (i.e. after a major update of the kernel/modules) your ephemeral id mapping takes some time to populate during which time authentication failures to CIFS shares can fail. This appears to have something to do with the state of ephemeral-to-SID mapping after re-boot.

To enable the mapping of unresolvable SIDs, do the following:

$ svccfg -s idmap setprop config/unresolvable_sid_mapping = boolean: true
$ svcadm refresh idmap

vSphere 4 Update 2 Released

VMware vSphere 4, Update 2 has been released with the following changes to ESXi:

The following information provides highlights of some of the enhancements available in this release of VMware ESXi:

• Enablement of Fault Tolerance Functionality for Intel Xeon 56xx Series processors— vSphere 4.0 Update 1 supports the Intel Xeon 56xx Series processors without Fault Tolerance. vSphere 4.0 Update 2 enables Fault Tolerance functionality for the Intel Xeon 56xx Series processors.


• Enablement of Fault Tolerance Functionality for Intel i3/i5 Clarkdale Series and Intel Xeon 34xx Clarkdale Series processors— vSphere 4.0 Update 1 supports the Intel i3/i5 Clarkdale Series and Intel Xeon 34xx Clarkdale Series processors without Fault Tolerance. vSphere 4.0 Update 2 enables Fault Tolerance functionality for the Intel i3/i5 Clarkdale Series and Intel Xeon 34xx Clarkdale Series processors.


• Enablement of IOMMU Functionality for AMD Opteron 61xx and 41xx Series processors— vSphere 4.0 Update 1 supports the AMD Opteron 61xx and 41xx Series processors without input/output memory management unit (IOMMU). vSphere 4.0 Update 2 enables IOMMU functionality for the AMD Opteron 61xx and 41xx Series processors.


• Enhancement of the resxtop utility— vSphere 4.0 U2 includes an enhancement of the performance monitoring utility, resxtop. The resxtop utility now provides visibility into the performance of NFS datastores in that it displays the following statistics for NFS datastores: Reads/s, writes/s, MBreads/s, MBwrtn/s, cmds/s, GAVG/s (guest latency).


• Additional Guest Operating System Support— ESX/ESXi 4.0 Update 2 adds support for Ubuntu 10.04. For a complete list of supported guest operating systems with this release, see the VMware Compatibility Guide.

Resolved Issues – In addition, this release delivers a number of bug fixes that have been documented in theResolved Issues section.

ESXi 4 Update 2 Release Notes

Noted in the release is the official support for AMD's IOMMU in Opteron 6100 and 4100 processors - available in 1P, 2P and 4P configurations. This finally closes the (functional) gap between AMD Opteron and Intel's Nehalem line-up. Likewise, FT support for many new Intel processors has been added. Also, the addition of NFS performance counters in ESXTop will make storage troubleshooting a bit easier. Grab you applicable update at VMware's download site now (SnS required.)

Microsoft Update Kills vSphere Client

Got a problem running vSphere Client today? Seeing the following pop-up when trying to access your VMware stack?

Login doesn't really continue, but in fact, ends with the following error:

Your environment has not been hacked! It's a problem with your most recent Windows Update, introducing a .NET exception that your "old" version of VMware vSphere Client can't handle. While you can uninstall the offending patch(es) to resolve the problem, the best remedy is to login to VMware's site and download the latest vSphere Client (VMware KB 1022611).

By the way, if you're vSphere Client is old enough to be affected (prior to Update 1), you might need to scan your vSphere environment for updates too. If you have SnS, run over to VMware's download page for vSphere and get the updated packages, starting with the vSphere Client: you can find the installable Client package with the vCenter Server Update 2 downloads.

ESXi Patches fix VSS, NTP and VMkernel

Two patches were made available on May 27, 2010 for ESXi 4.0 to fix certain bugs and security vulnerabilities in the platform. These patches are identified as ESXi400-201005401-SG and ESXi400-201005402-BG.

The first patch is security related and requires a reboot of the ESXi host:

This patch fixes a security issue. The updated NTP daemon fixes a flaw in the way it handled certain malformed NTP packets. The NTP daemon logged information about all such packets and replied with a NTP packet that was treated as malformed when received by another ntpd. A remote attacker could use this flaw to create an NTP packet reply loop between two ntpd servers through a malformed packet with a spoofed source IP address and port, causing ntpd on those servers to use excessive amounts of CPU time and fill disk space with log messages. The Common Vulnerabilities and Exposures Project (cve.mitre.org) has assigned the name CVE-2009-3563 to this issue.

ESXi 4.0 hosts might stop responding when interrupts are shared between VMkernel and service console. You might also observe the following additional symptoms:

• Network pings to the ESXi hosts might fail.


• Baseboard management controllers (BMC) such as HP Integrated Lights-Out (iLO) console might appear to be in a non-responsive state.

- VMware KB Article, 1021041

An interesting note is the reference to the service console in ESXi, however the sharing of interrupts between ESX drivers and the service console has long been a problem in ESX (not ESXi since there is no service console)... The second patch does not require a reboot, although it includes an update to VMware Tools which could impact uptime on affected virtual machines (Windows Server 2008 R2 and Windows 7). The KB article says this about the patch:

The VMware Snapshot Provider service is not listed in the Services panel. The quiesced snapshots do not use VMware Tools VSS components in Windows Server 2008 R2 or Windows 7 operating systems. This issue is seen when the user or backup software performs a quiesced snapshot on virtual machines on ESXi 4.0 hosts. This patch fixes the issue.

- VMware KB Article, 1021042

Since VSS quiescence is at issue here, DR snapshots and backups relying on VMware Data Recovery may be unreliable without the new VMware Tools installed. If your systems rely on VMware Data Recovery APIs for backup, this patch should be considered mandatory.