In-the-Lab: Windows Server 2008 R2 Template for VMware

As it turns out, the reasonably simple act of cloning a Windows Server 2008 R2 (insert addition here) has been complicated by the number of editions, changes from 2008 release through 2008 R2 as well as user profile management changes since its release. If you're like me, you like to tweak your templates to limit customization steps in post-deployment. While most of these customizations can now be setup in group policies from AD, the deployment of non-AD members has become a lot more difficult - especially where custom defaults are needed or required.

Here's my quick recipe to build a custom image of Windows Server 2008 R2 that has been tested with Standard, Enterprise and Foundation editions.

Create VM, use VMXNET3 as NIC(s), 40GB "thin" disk, using 2008 R2 Wizard

This is a somewhat "mix to taste" step. We use ISO images and encourage their use. The size of the OS volume will end-up being somewhere around 8GB of actual space-on-disk after this step, making 40GB sound like overkill. However, the OS volume will bloat-up to 18-20GB pretty quick after updates, roles and feature additions. Adding application(s) will quickly chew-up the rest.

• Edit Settings... ->
  
  
  
  • Options -> Advanced -> General -> Uncheck "Enable logging"
  
  
  • Hardware -> CD/DVD Drive 1 ->
    
    
    
    • Click "Datastore ISO File"
      
      
      
      • Browse to Windows 2008 R2 ISO image
      
      
      
      
    
    
    • Check "Connect at power on"
    
    
    
    
  
  
  • Options -> Advanced -> Boot Options -> Force BIOS Setup
    
    
    
    • Check "The next time the virtual machine boots, force entry into the BIOS setup screen"
    
    
    
    
  
  
  
  


• Power on VM


• Install Windows Server 2008 R2

Use Custom VMware Tools installation to disable "Shared Folders" feature:

It is important that VMware Tools be installed next, if for no other reason than to make the rest of the process quicker and easier. The additional step of disabling "Shared Folders" is for ESX/vSphere environments where shared folders are not supported. Since this option is installed by default, it can/should be removed in vSphere installations.

• VM -> Guest -> Install VMware Tools ->
  
  
  
  • Custom -> VMware Device Drivers -> Disable "Shared Folder" feature
  
  
  
  


• Retstart

Complete Initial Configuration Tasks:

Once the initial installation is complete, we need to complete the 2008 R2 basic configuration. If you are working in an AD environment, this is not the time to join the template to the domain as GPO conflicts may hinder manual template defaults. We've chosen a minimal package installation based on our typical deployment profile. Some features/roles may differ in your organization's template (mix to taste).

• Set time zone -> Date and Time ->
  
  
  
  • Internet Time -> Change Settings... -> Set to local time source
  
  
  • Date and Time -> Change time zone... -> Set to local time zone
  
  
  
  


• Provide computer name and domain -> Computer name ->
  
  
  
  • Enterprise Edition: W2K8R2ENT-TMPL
  
  
  • Standard Edition: W2K8R2STD-TMPL
  
  
  • Foundation Edition: W2K8R2FND-TMPL
  
  
  • Note: Don't join to a domain just yet...
  
  
  
  


• Restart Later


• Configure Networking
  
  
  
  • Disable QoS Packet Scheduler
  
  
  
  


• Enable automatic updating and feedback
  
  
  
  • Manually configure settings
    
    
    
    • Windows automatic updating -> Change Setting... ->
      
      
      
      • Important updates -> "check for updates but let me choose whether to download and install them"
      
      
      • Recommended updates -> Check "Give me recommended updates the same way I receive important updates"
      
      
      • Who can install updates -> Uncheck "Allow all users to install updates on this computer"
      
      
      
      
    
    
    • Windows Error Reporting -> Change Setting... ->
      
      
      
      • Select "I don't want to participate, and don't ask me again"
      
      
      
      
    
    
    • Customer Experience Improvement Program -> Change Setting... ->
      
      
      
      • Select "No, I don't want to participate"
      
      
      
      
    
    
    
    
  
  
  
  


• Download and install updates
  
  
  
  • Bring to current (may require several reboots)
  
  
  
  


• Add features (to taste)
  
  
  
  • .NET Framwork 3.5.1 Feautures
    
    
    
    • Check WCF Activation, Non-HTTP Activation
      
      
      
      • Pop-up: Click "Add Required Features"
      
      
      
      
    
    
    
    
  
  
  • SNMP Services
  
  
  • Telnet Client
  
  
  • TFTP Client
  
  
  • Windows PowerShell Integrated Scripting Environment (ISE)
  
  
  
  


• Check for updates after new features
  
  
  
  • Install available updates
  
  
  
  


• Enable Remote Desktop
  
  
  
  • System Properties -> Remote
    
    
    
    • Windows 2003 AD
      
      
      
      • Select "Allow connection sfrom computers running any version of Remote Desktop"
      
      
      
      
    
    
    • Windows 2008 AD (optional)
      
      
      
      • Select "Allow connections only from computers runnign Remote Desktop with Network Level Authentication"
      
      
      
      
    
    
    
    
  
  
  
  


• Windows Firewall
  
  
  
  • Turn Windows Firewall on of off
    
    
    
    • Home or work location settings
      
      
      
      • Turn off Windows Firewall
      
      
      
      
    
    
    • Public network location settings
      
      
      
      • Turn off Windows Firewall
      
      
      
      
    
    
    
    
  
  
  
  


• Complete Initial Configuration Tasks
  
  
  
  • Check "Do not show this window at logon" and close
  
  
  
  

Modify and Silence Server Manager

(Optional) Parts of this step may violate your local security policies, however, it's more than likely that a GPO will ultimately override this configuration. We find it useful to have this disabled for "general purpose" templates - especially in a testing/lab environment where the security measures will be defeated as a matter of practice.

• Security Information -> Configure IE ESC
  
  
  
  • Select Administrators Off
  
  
  • Select Users Off
  
  
  
  


• Select "Do not show me this console at logon" and close

Modify Taskbar Properties

Making the taskbar usable for your organization is another matter of taste. We like smaller icons and maximizing desktop utility. We also hate being nagged by the notification area...

• Right-click Taskbar -> Taskbar and Start Menu Properties ->
  
  
  
  • Taskbar -> Check "Use small icons"
  
  
  • Taskbar -> Customize... ->
    
    
    
    • Set all icons to "Only show notifications"
    
    
    • Click "Turn system icons on or off"
      
      
      
      • Turn off "Volume"
      
      
      
      
    
    
    
    
  
  
  • Start Menu -> Customize...
    
    
    
    • Uncheck "Use large icons"
    
    
    
    
  
  
  
  

Modify default settings in Control Panel

Some Control Panel changes will help "optimize" the performance of the VM by disabling unnecessary features like screen saver and power management. We like to see our corporate logo on server desktops (regardless of performance implications) so now's the time to make that change as well.

• Control Panel -> Power Options -> High Performance
  
  
  
  • Change plan settings -> Turn off the display -> Never
  
  
  
  


• Control Panel -> Sound ->
  
  
  
  • Pop-up: "Would you like to enable the Windows Audio Service?" - No
  
  
  • Sound -> Sounds -> Sound Scheme: No Sounds
  
  
  • Uncheck "Play Windows Startup sound"
  
  
  
  


• Control Panel -> VMware Tools -> Uncheck "Show VMware Tools in the taskbar"


• Control Panel -> Display -> Change screen saver -> Screen Saver -> Blank, Wait 10 minutes


• Change default desktop image (optional)
  
  
  
  • Copy your desktop logo background to a public folder (i.e. "c:\Users\Public\Public Pictures")
  
  
  • Control Panel -> Display -> Change desktop background -> Browse...
  
  
  • Find picture in browser, Picture position stretch
  
  
  
  

Disable Swap File

Disabling swap will allow the defragment step to be more efficient and will disable VMware's advanced memory management functions. This is only temporary and we'll be enabling swap right before committing the VM to template.

• Computer Properties -> Visual Effects -> Adjust for best performance


• Computer Properties -> Advanced System Settings ->
  
  
  
  • System Properties -> Advanced -> Performance -> Settings... ->
  
  
  • Performance Options -> Advanced -> Change...
    
    
    
    • Uncheck "Automatically manage paging file size for all drives"
    
    
    • Select "No paging file"
    
    
    • Click "Set" to disable swap file
    
    
    
    
  
  
  
  

Remove hibernation file and set boot timeout

It has been pointed out that the hibernation and timeout settings will get re-enabled by the sysprep operation. Removing the hibernation files will help in defragment now. We'll reinforce these steps in the customization wizard later.

• cmd: powercfg -h off


• cmd: bcdedit /timeout 5

Disable indexing on C:

Indexing the OS disk can suck performance and increase disk I/O unnecessarily. Chances are, this template (when cloned) will be heavily cached on your disk array so indexing in the OS will not likely benefit the template. We prefer to disable this feature as a matter of practice.

• C: -> Properties -> General ->
  
  
  
  • Uncheck "Allow files on this drive to have contents indexed in addition to file properties"
  
  
  • Apply -> Apply changes to C:\ only (or files and folders, to taste)
  
  
  
  

Housekeeping

Time to clean-up and prepare for a streamlined template. The first step is intended to aid the copying of "administrator defaults" to "user defaults." If this does not apply, just defragment.

Remove "Default" user settings:

• C:\Users -> Folder Options -> View -> Show hidden files...


• C:\Users\Default -> Delete "NTUser.*" Delete "Music, Pictures, Saved Games, Videos"

Defragment

• C: -> Properties -> Tools -> Defragment Now...
  
  
  
  • Select "(C:)"
  
  
  • Click "Defragment disk"
  
  
  
  

Copy Administrator settings to "Default" user

The "formal" way of handling this step requires a third-party utility. We're giving credit to Jason Samuel for consolidating other bloggers methods because he was the first to point out the importance of the "unattend.xml" file and it really saved us some time. His blog post also includes a link to an example "unattend.xml" file that can be modified for your specific use, as we have.

• Jason Samuel points out a way to "easily" copy Administrator settings to defaults, by activating the CopyProfile node in an "unattend.xml" file used by sysprep.


• Copy your "unattend.xml" file to C:\windows\system32\sysprep


• Edit unattend.xml for environment and R2 version
  
  
  
  • Update offline image pointer to correspond to your virtual CD
    
    
    
    • E.g. wim:d:... -> wim:f:...
    
    
    
    
  
  
  • Update OS offline image source pointer, valid sources are:
    
    
    
    • Windows Server 2008 R2 SERVERDATACENTER
    
    
    • Windows Server 2008 R2 SERVERDATACENTERCORE
    
    
    • Windows Server 2008 R2 SERVERENTERPRISE
    
    
    • Windows Server 2008 R2 SERVERENTERPRISECORE
    
    
    • Windows Server 2008 R2 SERVERSTANDARD
    
    
    • Windows Server 2008 R2 SERVERSTANDARDCORE
    
    
    • Windows Server 2008 R2 SERVERWEB
    
    
    • Windows Server 2008 R2 SERVERWEBCORE
    
    
    • Windows Server 2008 R2 SERVERWINFOUNDATION
    
    
    
    
  
  
  • Any additional changes necessary
  
  
  
  


• NOTE: now would be a good time to snapshot/backup the VM


• cmd: cd \windows\system32\sysprep


• cmd: sysprep /generalize /oobe /reboot /unattend:unattend.xml
  
  
  
  • Check "Generalize"
  
  
  • Shutdown Options -> Reboot
  
  
  
  


• Login


• Skip Activation


• Administrator defaults are now system defaults

• Reset Template Name
  
  
  
  • Computer Properties -> Advanced System Settings -> Computer name -> Change...
    
    
    
    • Enterprise Edition: W2K8R2ENT-TMPL
    
    
    • Standard Edition: W2K8R2STD-TMPL
    
    
    • Foundation Edition: W2K8R2FND-TMPL
    
    
    
    
  
  
  • If this will be an AD member clone, join template to the domain now
  
  
  
  
  
  
  • Restart
  
  
  
  

• Enable Swap files
  
  
  
  • Computer Properties -> Advanced System Settings ->
    
    
    
    • System Properties -> Advanced -> Performance -> Settings... ->
    
    
    • Performance Options -> Advanced -> Change...
      
      
      
      • Check "Automatically manage paging file size for all drives"
      
      
      
      
    
    
    
    
  
  
  
  


• Release IP
  
  
  
  • cmd: ipconfig /release
  
  
  
  


• Shutdown


• Convert VM to template

Convert VM Template to Clone

Use the VMware Customization Wizard to create a re-usable script for cloning the template. Now's a good time to test that your template will create a usable clone. If it fails, go check the "red letter" items and make sure your setup is correct. The following hints will help improve your results.

• Remove hibernation related files and reset boot delay to 5 seconds in Customization Wizard
  
  
  
  • Run Once:
    
    
    
    • powercfg -h off
    
    
    • bcdedit /timeout 5
    
    
    
    
  
  
  
  


• Remember that the ISO is still mounted by default. Once VM's are deployed from the template, it should be removed after the customization process is complete and additional roles/features are added.

That's the process we have working at SOLORI. It's not rocket science, but if you miss an important step you're likely to be visited by an error in "pass [specialize]" that will have you starting over. Note: this also happens when your AD credentials are bad, your license key is incorrect (version/edition mismatch, typo, etc.) or other nondescript issues - too bad the error code is unhelpful...

Short-Take: SQL Performance Notes

Here are some Microsoft SQL performance notes from discussions that inevitably crop-up when discussing SQL storage:

1. Where do I find technical resources for the current version of MS SQL?


2. I'm new to SQL I/O performance, how can I learn the basics?


3. The basics talk about SQL 2000, but what about performance considerations due to changes in SQL 2005?


4. How does using SQL Server 6.x versus SQL Server 7.0 and change storage I/O performance assumptions?


5. How does TEMPDB affect storage (and memory) requirements and architecture?


6. How does controller and disk caching affect SQL performance and data integrity?


7. How can I use NAS for storage of SQL database in a test/lab environment?


8. What additional considerations are necessary to implement database mirroring in SQL Server?


9. When do SQL dirty cache pages get flushed to disk?


10. Where can I find Microsoft's general reference sheet on SQL I/O requirements for more information?

From performance tuning to performance testing and diagnostics:

1. I've heard that SQLIOStress has been replaced by SQLIOSim: where can I find out about SQLIOSim to evaluate my storage I/O system before application testing?


2. How do I diagnose and detect "unreported" SQL I/O problems?


3. How do I diagnose stuck/stalled I/O problems in SQL Server?


4. What are Bufwait and Writelog Timeout messages in SQL Server indicating?


5. Can I control SQL Server checkpoint behavior to avoid additional I/O during certain operations?


6. Where can I get the SQLIO benchmark tool to assess the potential of my current configuration?

That should provide a good half-day's reading for any storage/db admin...

NexentaStor CIFS Shares with Active Directory Authentication

Sharing folders in NexentaStor is pretty easy in Workgroup mode, but Active Directory integration takes a few extra steps.  Unfortunately, it's not (yet) as easy as point-and-click, but it doesn't have to be too difficult either. (The following assumes/requires that the NexentaStor appliance has been correctly configured-in and joined-to Active Directory.)

[caption id="attachment_1562" align="alignright" width="170" caption="Typical user and group permissions for a local hard disk in Windows."][/caption]

Let's examine the case where a domain admin group will have "Full Control" of the share, and "Everyone" will have read/execute permissions. This is a typical use case where a single share contains multiple user directories under administrative control. It's the same configuration as local disks in a Windows environment. For our example, we're going to mimic this setup using a CIFS share from a NexentaStor CE appliance and create the basic ACL to allow for Windows AD control.

For this process to work, we need to join the NexentaStor appliance to the Active Directory Domain. The best practice is to create the machine account in AD first, assign control user/group rights (if possible) and then attempt to join. It is IMPORTANT that the host name and DNS configuration of the NexentaStor appliance match domain norms, or things will come crashing to a halt pretty quickly.

That said, assuming that your DC is 1.1.1.1 and your BDC is 1.1.1.2 with a "short" domain of "SOLORI" and a FQDN of "SOLORI.MSFT" your NexentaStor's name server configuration (Settings->Network->Name Servers) would look something like this:

This is important because the AD queries will pull service records from the configured domain name servers. If these point to an "Internet" DNS server, the AD entries may not be reflected in that server's database and AD authentication (as well as join) will fail.

The other way the NexentaStor appliance knows what AD Domain to look into is by its own host name. For AD authentication to work properly, the NexentaStor host name must reflect the AD domain. For example, if the FQDN of your AD domain is "SOLORI.MSFT" then your domain name on the appliance would be configured like this (Appliance->Basic Settings->Domainname):

The next step is to create the machine account in AD using "Active Directory Users and Computers" administrator's configuration tool. Find your domain folder and right-click "Computers" - select New->Computer from the menu and enter the computer name (no domain). The default user group assigned to administrative control should be Domain Admins. Since this works for our example, no changes are necessary so click "OK" to complete.

Now it's time to join the AD domain from NexentaStor. Any user with permissions to join a machine to the domain will do. Armed with that information, drill down to Data Management->Shares->CIFS Server->Join AD/DNS Server and enter the AD/DNS server. AD server, AD user and user password into the configuration box:

If your permissions and credentials are good, your NexentaStor appliance is not a member of your domain. As such, it can now identify AD users and groups by unique gid and uid data created from AD. This gid and uid information will be used to create our ACLs for the CIFS share.

To uncover the gid for the "Domain Admins" and "Domain Users" groups, we issue the following from the NexentaStor NMC (CLI):

nmc@san01:/$ idmap dump -n | grep "Domain Admins"

wingroup:Domain Admins@solori.msft     ==      gid:3036392745

nmc@san01:/$ idmap dump -n | grep "Domain Users"
wingroup:Domain Users@solori.msft     ==      gid:1238392562

Now we can construct a CIFS share (with anonymous read/write disabled) and apply the Domain Admin gid to an ACL - just click on the share, and then click "(+) Add Permissions for Group":

[caption id="attachment_1568" align="aligncenter" width="300" caption="Applying administrative permissions with the AD group ID for Domain Admins."][/caption]

We do similarly with the Domain User gid:

[caption id="attachment_1569" align="aligncenter" width="300" caption="Applying the Domain User gid to CIFS share ACL."][/caption]

Note that the "Domain Users" group gets only "execute" and "read" permissions while the "Domain Admins" group gets full control - just like the local disk! Now, with CIFS sharing enabled and the ACL suited to our AD authentication, we can access the share from any domain machine provided our user is in the Domain Users or Admins group.

Administrators can now create "personal" folders and assign detailed user rights just as they would do with any shared storage device. The only trick is in creating the initial ACL for the CIFS share - as about - and you've successfully integrated your NexentaStor appliance into you AD domain.

NOTE: If you're running Windows Server 2008 (or SBS 2008) as your AD controller, you will need to update the share mode prior to joining the domain using the following command (from root CLI):

# sharectl set -p lmauth_level=2 smb

NOTE: I’ve also noticed that, upon reboot of the appliance (i.e. after a major update of the kernel/modules) your ephemeral id mapping takes some time to populate during which time authentication failures to CIFS shares can fail. This appears to have something to do with the state of ephemeral-to-SID mapping after re-boot.

To enable the mapping of unresolvable SIDs, do the following:

$ svccfg -s idmap setprop config/unresolvable_sid_mapping = boolean: true
$ svcadm refresh idmap

Microsoft Update Kills vSphere Client

Got a problem running vSphere Client today? Seeing the following pop-up when trying to access your VMware stack?

Login doesn't really continue, but in fact, ends with the following error:

Your environment has not been hacked! It's a problem with your most recent Windows Update, introducing a .NET exception that your "old" version of VMware vSphere Client can't handle. While you can uninstall the offending patch(es) to resolve the problem, the best remedy is to login to VMware's site and download the latest vSphere Client (VMware KB 1022611).

By the way, if you're vSphere Client is old enough to be affected (prior to Update 1), you might need to scan your vSphere environment for updates too. If you have SnS, run over to VMware's download page for vSphere and get the updated packages, starting with the vSphere Client: you can find the installable Client package with the vCenter Server Update 2 downloads.

ESXi Patches fix VSS, NTP and VMkernel

Two patches were made available on May 27, 2010 for ESXi 4.0 to fix certain bugs and security vulnerabilities in the platform. These patches are identified as ESXi400-201005401-SG and ESXi400-201005402-BG.

The first patch is security related and requires a reboot of the ESXi host:

This patch fixes a security issue. The updated NTP daemon fixes a flaw in the way it handled certain malformed NTP packets. The NTP daemon logged information about all such packets and replied with a NTP packet that was treated as malformed when received by another ntpd. A remote attacker could use this flaw to create an NTP packet reply loop between two ntpd servers through a malformed packet with a spoofed source IP address and port, causing ntpd on those servers to use excessive amounts of CPU time and fill disk space with log messages. The Common Vulnerabilities and Exposures Project (cve.mitre.org) has assigned the name CVE-2009-3563 to this issue.

ESXi 4.0 hosts might stop responding when interrupts are shared between VMkernel and service console. You might also observe the following additional symptoms:

• Network pings to the ESXi hosts might fail.


• Baseboard management controllers (BMC) such as HP Integrated Lights-Out (iLO) console might appear to be in a non-responsive state.

- VMware KB Article, 1021041

An interesting note is the reference to the service console in ESXi, however the sharing of interrupts between ESX drivers and the service console has long been a problem in ESX (not ESXi since there is no service console)... The second patch does not require a reboot, although it includes an update to VMware Tools which could impact uptime on affected virtual machines (Windows Server 2008 R2 and Windows 7). The KB article says this about the patch:

The VMware Snapshot Provider service is not listed in the Services panel. The quiesced snapshots do not use VMware Tools VSS components in Windows Server 2008 R2 or Windows 7 operating systems. This issue is seen when the user or backup software performs a quiesced snapshot on virtual machines on ESXi 4.0 hosts. This patch fixes the issue.

- VMware KB Article, 1021042

Since VSS quiescence is at issue here, DR snapshots and backups relying on VMware Data Recovery may be unreliable without the new VMware Tools installed. If your systems rely on VMware Data Recovery APIs for backup, this patch should be considered mandatory.

vSphere, Hardware Version 7 and Hot Plug

VMware's vSphere added hot plug features in hardware version 7 (first introduced in VMware Workstation 6.5) that were not available in the earlier version 4 virtual hardware. Virtual hardware version 7 adds the following new features to VMware virtual machines:

• LSI SAS virtual device - provides support for Windows Server 2008 fail-over cluster configurations


• Paravirtual SCSI devices - recently updated to allow booting, can allow higher-performance (greater throughput and lower CPU utilization) than the standard virtual SCSI adapter - especially in SAN environments where I/O-intensive applications are used. Currently supported in Windows Server 2003/2008 and Red Hat Linux 5 - although any version of Linux could be modified to support PVSCSI.


• IDE virtual device - useful for older OSes that don't support SCSI drivers


• VMXNET 3 - next generation Vmxnet device with enhanced performance and enhanced networking features.


• Hot plug virtual devices, memory and CPU - supports hot add/remove of virtual devices, memory and CPU for supported OSes.

While the "upgrade" process from version 4 to version 7 is well-known, some of the side effects are not well publicised. The most obvious change after the migration from version 4 to version 7 is the affect hot plug has on the PCI bus adapters - some are now hot plug by default, including the network adapters!

[caption id="attachment_1342" align="aligncenter" width="417" caption="Safe to remove network adapters. Really?"][/caption]

Note that the above example demonstrates also that the updated hardware re-enumerates the network adapters (see #3 and #4) because they have moved to a new PCI bus - one that supports hot plug. Removing the "missing" devices requires a trip to device manager (set devmgr_show_nonpresent_devices=1 in your shell environment first.) This hot plug PCI bus also allows for an administrator to mistakenly remove the device from service - potentially disconnecting tier 1 services from operations (totally by accident, of course.

[caption id="attachment_1340" align="aligncenter" width="450" caption="Devices that can be added while the VM runs with hardware version 4"][/caption]

In virtual hardware version 4, only SCSI devices and hard disks were allowed to be added to a running virtual machine. Now with hardware version 7,

[caption id="attachment_1341" align="aligncenter" width="450" caption="Devices that can be added while the VM runs with hardware version 7"][/caption]

additional devices (USB and Ethernet) are available for hot add. You could change memory and CPU on the fly too, if the OS supports that feature and they are enabled in the virtual machine properties prior to running the VM:

[caption id="attachment_1339" align="aligncenter" width="450" caption="CPU and Memory Hot Plug Properties"][/caption]

However, the hot plug NIC issue isn't discussed in the documentation, but Carlo Costanzo at VMwareInfo.com passes on Chris Hahn's great tip to disable hot plug behaviour in his blog post complete with visual aids. The key is to add a new "Advanced Configuration Parameter" to the virtual machine configuration: this new parameter is called "devices.hotplug" and its value should be set to "false." However, adding this parameter requires the virtual machine to be turned-off, so it is currently an off-line fix.

vSphere 4, Update 1 and ESXi

On November 19, 2009 VMware released Update 1 for vSphere 4.0 which, among other bug fixes and errata, adds the following new features:

• ESX/ESXi
  
  
  
  • VMware View 4.0 support (previously unsupported)
  
  
  • Windows 7 and Windows 2008 R2 support (previously "experimental") - guest customizations now supported
  
  
  • Enhanced Clustering support for Microsoft Windows - adds support for VMware HA and DRS by allowing HA/DRS to be disabled per MSCS VM instead of per ESX host
  
  
  • Enhanced VMware Paravirtualized SCSI support (pvSCSI boot disks now supported in Windows 2003/2008)
  
  
  • Improved vNetwork Distributed Switch performance
  
  
  • Increased vCPU per Core limit (raised from 20 to 25)
  
  
  • Intel Xeon 3400 (uni-processor variant of the Nehalem)
  
  
  
  


• vCenter Server
  
  
  
  • Support for IBM DB2 (Enterprise, Workgroup and Express 9, Express C)
  
  
  • Windows 7 and Windows 2008 R2 support (previously "experimental") - guest customizations now supported
  
  
  
  


• vCenter Update Manager
  
  
  
  • Does not support IBM DB2
  
  
  • Still no scan or remediate for Windows Server 2003 SP2/R2 64-bit, Windows Server 2008 or Windows 7
  
  
  
  


• vSphere Client
  
  
  
  • Windows 7 officially supported (no more workaround hacks)
  
  
  
  


• vSphere Command-Line Interface
  
  
  
  • allows the use of comma-separated bulletins with --bundle option in "vihostupdate"
  
  
  
  

Authorized VMware users can download the necessary updates for vSphere Update 1 directly from VMware. For ESX and ESXi, updates can be managed and installed from the vCenter Update Manager within the vSphere Client. In addition to the normal backup procedure and those steps recommend by VMware, the following observations may be helpful to you:

• DRS/HA cluster members CAN be updated auto-magically, however we observed very low end-to-end success rates in our testing lab. We recommend the following:
  
  
  
  • Manually enter maintenance mode for the ESXi server
  
  
  • Manually stage/remediate the patches to avoid conflicts
  
  
  • Manually re-boot ESXi servers if they do not reboot on their own
  
  
  • Re-scan patches when re-boot is complete, to check/confirm upgrade success
  
  
  • Manually recover from maintenance mode and confirm HA host configuration
  
  
  
  


• For the vSphere Client on Windows 7, completely remove the "hacked" version and clean-install the latest version (download from the updated ESX/ESXi server(s))

SOLORI's Notes: When upgrading ESXi "auto-magically" we experienced the following failures and unwanted behavior:

• Update manager failure to stage pending updates and upgrades correctly, resulting in a "time-out" failure. However, updates are/were applied properly after manual reboot.


• DRS/DPM conflicts with upgrade process:
  
  
  
  • inadequate time given to servers for servers to recover from sleep mode
  
  
  • Hosts sent to sleep while updates being evaluated, causing DRS to hold maintenance mode while sleeping hosts awakened, resulting in failed/time-out update process
  
  
  
  


• Off-line VMs and templates not automatically migrated during update (maintenance mode) process causing extended unavailability of these assets during update

Additional Notes: Directed to the question of which updates/patches may or may not be "rolled-up" into this update, the release notes are very clear. However, for the sake of convenience, we repeat them here:

Patch Release ESX400-Update01 contains the following individual bulletins:

• ESX400-200911201-UG: Updates Core


• ESX400-200911205-UG: Updates lpfc820 driver to version 8.2.0.30.52vmw-2vmw


• ESX400-200911206-UG: Updates hpsa driver to version 3.6.14.27vmw-2vmw


• ESX400-200911207-UG: Updates mpt2sas driver to version 02.00.00.00.1vmw-2vmw


• ESX400-200911208-UG: Updates Service Console Kernel


• ESX400-200911209-UG: Updates forcedeth driver to version 0.61-2vmw


• ESX400-200911210-UG: Updates ixgbe driver to version 1.3.36-2vmw


• ESX400-200911211-UG: Updates megaraid2 driver to version 2.00.4-2vmw


• ESX400-200911212-UG: Updates megaraid-sas driver to version


• ESX400-200911213-UG: Updates mptsas driver to version


• ESX400-200911214-UG: Updates ahci driver to version 2.0-2vmw


• ESX400-200911215-UG: Updates ata-piix driver to version 2.00ac6-2vmw


• ESX400-200911216-UG: Updates mptspi driver to version


• ESX400-200911217-UG: Updates qla2xxx driver to version 821.k1.42vmw-2vmw


• ESX400-200911218-UG: Updates s2io driver to version 2.1.4.13427-2vmw


• ESX400-200911219-UG: Updates nx-nic driver to version 4.0.301-2vmw


• ESX400-200911220-UG: Updates cciss driver to version 3.6.14.8vmw-2vmw


• ESX400-200911221-UG: Updates Service Console glibc


• ESX400-200911222-UG: Updates enic driver to version 1.1.0.68-1vmw


• ESX400-200911223-UG: Updates Web Access


• ESX400-200911224-UG: Updates fnic driver to version 1.1.0.49.2vmw-2vmw


• ESX400-200911225-UG: Updates bnx2x driver to version 1.45.20-2vmw


• ESX400-200911226-UG: Updates the VMware ESX 4.0 e1000


• ESX400-200911227-UG: Updates the ESX 4.0 Intel igb driver


• ESX400-200911228-UG: Updates the ESX 4.0 Serverworks PATA for ATA driver


• ESX400-200911229-UG: Updates the ESX 4.0 ATA


• ESX400-200911230-UG: Updates the ESX 4.0 SCSI/iSCSI driver


• ESX400-200911231-UG: Updates the ESX 4.0 qlogic SCSI driver


• ESX400-200911232-UG: Updates the ESX 4.0 curl component


• ESX400-200911233-UG: Updates the ESX 4.0 gnutls component


• ESX400-200911234-UG: Updates the ESX 4.0 libxml2 component


• ESX400-200911235-UG: Updates the ESX 4.0 python component


• ESX400-200911236-UG: Updates ESX 4.0 SNMP component


• ESX400-200911237-UG: Updates the ESX 4.0 bind component


• ESX400-200911238-UG: Updates the ESX 4.0 ntp component


• ESX400-200911239-UG: Updates the ESX 4.0 audit component:

ESXi 4.0 Update 1 also contains all fixes in the following previously released bundles:

Patch Release ESX400-200906001
Patch Release ESX400-200907001
Patch Release ESX400-200909001

Quick Take: Red Hat and Microsoft Virtual Inter-Op

This week Red Hat and Microsoft announced support of certain of their OSes as guests in their respective hypervisor implementations: Kernel Virtual Machine (KVM) and Hyper-V, respectively. This comes on the heels of Red Hat's Enterprise Server 5.4 announcement last month.

KVM is Red Hat's new hypervisor that leverages the Linux kernel to accelerate support for hardware and capabilities. It was Red Hat and AMD that first demonstrated live migration between AMD and Intel-based hypervisors using KVM late last year - then somewhat of a "Holy Grail" of hypervisor feats. With nearly a year of improvements and integration into their Red Hat Enterprise Server and Fedora "free and open source" offerings, Red Hat is almost ready to strike-out in a commercially viable way.

Microsoft now officially supports the following Red Hat guest operating systems in Hyper-V:

Red Hat Enterprise Linux 5.2, 5.3 and 5.4

Red Hat likewise officially supports the following Microsoft quest operating systems in KVM:

Windows Server 2003, 2008 and 2008 R2

The goal of the announcement and associated agreements between Red Hat and Microsoft was to enable a fully supported virtualization infrastructure for enterprises with Red Hat and Microsoft assets. As such, Microsoft and Red Hat are committed to supporting their respective products whether the hypervisor environment is all Red Hat, all Hyper-V or totally heterogeneous - mixing Red Hat KVM and Microsoft Hyper-V as necessary.

"With this announcement, Red Hat and Microsoft are ensuring their customers can resolve any issues related to Microsoft Windows on Red Hat Enterprise Virtualization, and Red Hat Enterprise Linux operating on Microsoft Hyper-V, regardless of whether the problem is related to the operating system or the virtualization implementation."

- Red Hat press release, October 7, 2009

Many in the industry cite Red Hat's adoption of KVM as a step backwards [from Xen] requiring the re-development of significant amount of support code. However, Red Hat's use of libvirt as a common management API has allowed the change to happen much more rapidly that critics assumptions had allowed. At Red Hat Summit 2009, key Red Hat officials were keen to point out just how tasty their "dog food" is:

Tim Burke, Red Hat's vice president of engineering, said that Red Hat already runs much of its own infrastructure, including mail servers and file servers, on KVM, and is working hard to promote KVM with key original equipment manufacturer partners and vendors.

And Red Hat CTO Brian Stevens pointed out in his Summit keynote that with KVM inside the Linux kernel, Red Hat customers will no longer have to choose which applications to virtualize; virtualization will be everywhere and the tools to manage applications will be the same as those used to manage virtualized guests.

- Xen vs. KVM, by Pam Derringer, SearchDataCenter.com

For system integrators and virtual infrastructure practices, Red Hat's play is creating opportunities for differentiation. With a focus on light-weight, high-performance, I/O-driven virtualization applications and no need to support years-old established processes that are dragging on Xen and VMware, KVM stands to leap-frog the competition in the short term.

SOLORI's Take: This news is good for all Red Hat and Microsoft customers alike. Indeed, it shows that Microsoft realizes that its licenses are being sold into the enterprise whether or not they run on physical hardware. With 20+:1 consolidation ratios now common, that represents a 5:1 license to hardware sale for Microsoft, regardless of the hypervisor. With KVM's demonstrated CPU agnostic migration capabilities, this opens the door to an even more diverse virtualization infrastructure than ever before.

On the Red Hat side, it demonstrates how rapidly Red Hat has matured its offering following the shift to KVM earlier this year. While KVM is new to Red Hat, it is not new to Linux or aggressive early adopters since being added to the Linux kernel as of 2.6.20 back in September of 2007. With support already in active projects like ConVirt (VM life cycle management), OpenNebula (cloud administration tools), Ganeti, and Enomaly's Elastic Computing Platform, the game of catch-up for Red Hat and KVM is very likely to be a short one.

Quick Take: Licensing Benefits of Virtualization

For some, the real licensing benefit of virtualization is a hidden entity. This is aided by some rather nebulous language in end user licenses from Microsoft and others.

Steve Kaplan, over at DABCC, has a brief and informative article on how licensing affects the deployment costs of virualized Microsoft products - sometimes offsetting the cost of the hypervisor costs in a VMware environment, for instance.

SOLORI's 1st take: the virtual data center has new ways to increase costs with equal or better offsets to speed ROI - especially where new initiatives are concerned. When in doubt, talk to your software vendor and press for clear information about implementation licensing costs.

SOLORI's 2nd take: Steve's report relies on Gartner's evaluation which is based on Microsoft policies that are outdated. For instance, Server 2003 R2 is NOT "the only edition that will allow the customer to run one instance in a physical operating system (OS) environment and up to four instances in virtual OS environments for one license fee." This also applies to Server 2008... (see Microsoft links).


Check out Steve's evaluation here. Also, see Microsoft's updated policy here and their current Server 2003 policies here.

SBS 2008 Panics, Needs IPv6

Remember how you were told to disable all unused applications and protocols when securing a compute environment? If you've been in networking for years - like I have - it's almost a reflex action. This is also more recently codified in PCI/DSS Section 2.2.2, right? It also seems like a really basic, logical approach. Apparently Microsoft doesn't think so. Apparently, there is a "somewhat artificial albeit deeply ingrained" dependency on IPv6 in Windows Server 2008.

2.2.2 Disable all unnecessary and insecure services and protocols (services and protocols not directly needed to perform the device’s specified function).

- PCI Security Standards Council

Considering the lackluster adoption rate of IPv6 in the Internet domain, it is hard to argue that IPv6 on the local network is a new requirement. Given that most system administrators have enough difficulty understanding IPv4 networks, a dependency on IPv6 seems both premature and an unnecessary complexity.

Corollary: Disabling IPv6 Kills SBS 2008

Simply disabling IPv6 at the network card level carries no dire warning. Services continue to function properly with no warnings or klaxon calls. However, a reboot tells a different story: the absence of IPv6 KILLS a myriad of service on reboot.

Virtualizing Microsoft Small Business Server 2008

Microsoft Small Business Server 2008 appears to be a good option for "Microsoft shops" looking to consolidated Active Directory Domain Controller, Windows Update Services, Security and Virus Protection, SharePoint and Exchange functions on a single server. In practice, however, it takes a really "big" server to support this application – even for a very small office environment.

According to Microsoft's system requirements page a target system will need a 2GHz core, 4GB of RAM and 60GB of disk space to be viable (minimum). As with past experiences with Microsoft's products, this “minimum system specification” is offered as a “use this only if you want to under-perform horribly as to be unusable” configuration.

In reality, a single-user, clean installation with two 2.3GHz cores and 4GB of RAM committed to the installation, the server struggles with the minimum amount of memory. The “real” minimum should have been specified as 6GB instead of the optimistic 4GB. Where does the memory go?

It appears that Microsoft Exchange-related memory utilization weighs in at over 1.2GB (including 600MB for “store.exe”). Another 1.2GB is consumed by IIS, SharePoint and SQL. Forefront Client Security-related memory usage is around 1GB all by itself. That's 3.4GB of “application space” drag in Windows Small Business Server 2008. Granted, the “killer app” for SBS is Exchange 2007 and some might argue SharePoint integration as well, so the use of the term “drag” may not be appropriate. However, let's just agree that 4GB is a ridiculously low (optimistic) minimum.