Showing posts with label esxi. Show all posts
Showing posts with label esxi. Show all posts

Thursday, June 3, 2010

ESXi Patches fix VSS, NTP and VMkernel

Two patches were made available on May 27, 2010 for ESXi 4.0 to fix certain bugs and security vulnerabilities in the platform. These patches are identified as ESXi400-201005401-SG and ESXi400-201005402-BG.

The first patch is security related and requires a reboot of the ESXi host:
This patch fixes a security issue. The updated NTP daemon fixes a flaw in the way it handled certain malformed NTP packets. The NTP daemon logged information about all such packets and replied with a NTP packet that was treated as malformed when received by another ntpd. A remote attacker could use this flaw to create an NTP packet reply loop between two ntpd servers through a malformed packet with a spoofed source IP address and port, causing ntpd on those servers to use excessive amounts of CPU time and fill disk space with log messages. The Common Vulnerabilities and Exposures Project (cve.mitre.org) has assigned the name CVE-2009-3563 to this issue.

ESXi 4.0 hosts might stop responding when interrupts are shared between VMkernel and service console. You might also observe the following additional symptoms:

  • Network pings to the ESXi hosts might fail.

  • Baseboard management controllers (BMC) such as HP Integrated Lights-Out (iLO) console might appear to be in a non-responsive state.


- VMware KB Article, 1021041



An interesting note is the reference to the service console in ESXi, however the sharing of interrupts between ESX drivers and the service console has long been a problem in ESX (not ESXi since there is no service console)... The second patch does not require a reboot, although it includes an update to VMware Tools which could impact uptime on affected virtual machines (Windows Server 2008 R2 and Windows 7). The KB article says this about the patch:




The VMware Snapshot Provider service is not listed in the Services panel. The quiesced snapshots do not use VMware Tools VSS components in Windows Server 2008 R2 or Windows 7 operating systems. This issue is seen when the user or backup software performs a quiesced snapshot on virtual machines on ESXi 4.0 hosts. This patch fixes the issue.


- VMware KB Article, 1021042



Since VSS quiescence is at issue here, DR snapshots and backups relying on VMware Data Recovery may be unreliable without the new VMware Tools installed. If your systems rely on VMware Data Recovery APIs for backup, this patch should be considered mandatory.

Friday, January 15, 2010

Quick-Take: It's Time to Upgrade from ESX to ESXi

That's right, I said upgrade from ESX to ESXi - not ESX 3.x to vSphere, but ESX (any version) to vSphere's ESXi! Ever since VMware PartnerExchange 2009 (April), SOLORI has been advising clients and prospects to focus on ESXi and move away from ESX-based hosts - you know, the one with the "Linux" service console. Personally, I'm glad VMware has strengthened their message about the virtues of ESXi and the direction of their flagship product.
ESXi has a superior architecture and we encourage customers to deploy ESXi as part of any new vSphere deployment. Our future posts will compare ESX 4 and ESXi 4 in detail on topics like hardware compatibility list, performance, and management to demonstrate that ESXi is either on par with or superior than ESX. But for now, here are some key points you should know about ESXi vs. ESX:

  1. The functionality and performance of VMware ESX and ESXi are the same; the difference between the two hypervisors resides in their packaging architecture and operational management. VMware ESXi is the latest hypervisor architecture from VMware. It has an ultra thin footprint with no reliance on a general-purpose OS, setting a new bar for security and reliability (learn more).

  2. In the future, ESXi’s superior architecture will be the exclusive focus of VMware's development efforts.

  3. New and existing customers are highly encouraged to deploy ESXi. Many Fortune 100 companies have already standardized on the ESXi platform.


- VMware Blog, June, 2009



Not unfamiliar with the VI3 version of ESXi, its ease of installation, configuration and management and smaller footprint, I was one of about 10 participants in an "ESXi BoF Breakout Session" with Charu Chaubal of VMware. While discussing vSphere's ESXi with Charu, I never heard the words "ESXi is a superior architecture," but I did get a clear message that ESXi was the way of the future. From that point on, it seemed as though any efforts concentrated on (net new) ESX deployments was going to be "time wasted."

However, it was clear by the whispered tone about ESXi's virtues that the timing was not right for the real message to be spoken aloud. Remember, this was the launch of vSphere and "ESXi" was strongly associated with "ESXi Free" - not a clear marketing message when license sales and adoption curves are on the line. Perhaps that's why the "talking points" at PEX2009 always suggested that ESX was the "flagship" hypervisor and ESXi was "targeted for embedded and OEM" deployments.

In practical terms, migration from ESX 3.x to vSphere/ESXi didn't make a lot of sense for many large or institutional customers at the time due to the lack of third-party driver parity between ESX and ESXi in vSphere. However, for net new installations where thrid-party drivers and service console agents were not a concern, the message about ESXi's superiority was getting lost. Thomas Paine once said "what we obtain too cheap, we esteem too lightly" and I'd attribute the slow uptake on ESXi to the perception that it was somehow inferior to ESX - a misconception owed to it being offered in a "free" version.
Price is what you pay; value is what you get.

- Warren Buffet



By Warren Buffet's standards, ESXi is a tremendous value. For the novice entering bare-metal virtualization for the first time, the relative ease with which ESXi installs to a USB flash drive allows for a trial experience that is both quick and reassuring. I'd argue that ESXi's tiny memory footprint and generous hardware support make it the easiest platform for testing bare-metal virtualization. As I've blogged about, ESXi takes about 14 minutes to install to USB flash - from initial CD-ROM boot to configuration console.

Anyone experienced with ESX will have encountered service and performance affecting IRQ sharing issues with the service console. The hours discovering, tracking and mitigating these kinds of issues after a BIOS update or hardware refresh can never be reclaimed. The fact is these problems simply do not exist for ESXi - something many ESXers will appreciate.

I've attended a great deal of WebEx sessions on VMware products over the last year and I'm still hearing hushed tones and uncertainty about the role of ESXi versus ESX. To be clear, ESXi is being talked about as "enterprise ready," but much of the focus is still on ESX. These overtones were still present in an "vSphere: Install, Configure and Manage" course I recently attended to qualify for VCP410. While our instructors were very knowledgeable and experienced, there seemed to be much less confidence when discussing ESXi versus ESX. In fact, the lab guide clearly states:
If you are new to VMware vSphere and you do not have any special needs for more advanced features, use ESXi.

- Page 599, Module 13, Installing VMware ESX and ESXi



The manual - and VMware's choice of message here - seems to indicate that ESX has "more advanced features" than ESXi. While the "advanced features" VMware is talking about are service console related, it leaves many regarding ESXi as the inferior product in sharp contrast to today's message. If the statement "ESXi’s superior architecture will be the exclusive focus of VMware's development efforts" isn't writing on the wall for the rest of you, here's a list of VMware's new talking points on ESXi:

  • Improved Reliability and Security

  • Fewer Patches

  • Less Disk Space

  • Streamlined Deployment and Configuration

  • Reduced Management Overhead

  • Next Generation Hypervisor

  • Superior Architecture

  • Drastically Reduced Hypervisor Footprint

  • Smaller Code Base, Smaller Attach Surface

  • Certified on over 1,000 Server Systems - including USB keys

  • New, Operating System Independent


In contrast, the ESX platform is being re-imaged. Here are some new talking points about ESX:

  • The Older Architecture

  • Relies on Linux OS for Resource Management

  • 20x Larger on-disk Footprint

  • More Complex to Configure

  • Console OS Administration for Configuration and Diagnostics

  • Prone to Arbitrary Code Execution (Console OS)


For many of us familiar with both ESXi and ESX, nothing here is really new. The only real change is the message: build your eco-system around ESXi...

SOLORI's Take: It's clear to me that VMware took inventory of its customers and chose to lead with ESX when vSphere was released. I suspect this was a practical decision due to the overwhelming numbers of ESX hosts already installed. However, the change in marketing and positioning we're witnessing signals that we're moving toward a time when ESX will be openly considered a dead-end variant.

When will ESX be phased-out? That's up to market forces and VMware, but the cloud loves efficiency and ESXi is certainly more resource efficient and compartmentalized than its brother ESX. Furthermore, VMware has to maintain two development and support chains with ESX and ESXi and Darwin likes ESXi. If I had to bet, I wouldn't put my money on seeing an ESX version of the next major release. In any case, when ESX is gone VMware can stop having to make excuses for the "linux console" and the implications that VMware is somehow "based on Linux."

Monday, November 30, 2009

vSphere 4, Update 1 and ESXi

On November 19, 2009 VMware released Update 1 for vSphere 4.0 which, among other bug fixes and errata, adds the following new features:

  • ESX/ESXi

    • VMware View 4.0 support (previously unsupported)

    • Windows 7 and Windows 2008 R2 support (previously "experimental") - guest customizations now supported

    • Enhanced Clustering support for Microsoft Windows - adds support for VMware HA and DRS by allowing HA/DRS to be disabled per MSCS VM instead of per ESX host

    • Enhanced VMware Paravirtualized SCSI support (pvSCSI boot disks now supported in Windows 2003/2008)

    • Improved vNetwork Distributed Switch performance

    • Increased vCPU per Core limit (raised from 20 to 25)

    • Intel Xeon 3400 (uni-processor variant of the Nehalem)



  • vCenter Server

    • Support for IBM DB2 (Enterprise, Workgroup and Express 9, Express C)

    • Windows 7 and Windows 2008 R2 support (previously "experimental") - guest customizations now supported



  • vCenter Update Manager

    • Does not support IBM DB2

    • Still no scan or remediate for Windows Server 2003 SP2/R2 64-bit, Windows Server 2008 or Windows 7



  • vSphere Client


  • vSphere Command-Line Interface

    • allows the use of comma-separated bulletins with --bundle option in "vihostupdate"




Authorized VMware users can download the necessary updates for vSphere Update 1 directly from VMware. For ESX and ESXi, updates can be managed and installed from the vCenter Update Manager within the vSphere Client. In addition to the normal backup procedure and those steps recommend by VMware, the following observations may be helpful to you:

  • DRS/HA cluster members CAN be updated auto-magically, however we observed very low end-to-end success rates in our testing lab. We recommend the following:

    • Manually enter maintenance mode for the ESXi server

    • Manually stage/remediate the patches to avoid conflicts

    • Manually re-boot ESXi servers if they do not reboot on their own

    • Re-scan patches when re-boot is complete, to check/confirm upgrade success

    • Manually recover from maintenance mode and confirm HA host configuration



  • For the vSphere Client on Windows 7, completely remove the "hacked" version and clean-install the latest version (download from the updated ESX/ESXi server(s))


SOLORI's Notes: When upgrading ESXi "auto-magically" we experienced the following failures and unwanted behavior:

  • Update manager failure to stage pending updates and upgrades correctly, resulting in a "time-out" failure. However, updates are/were applied properly after manual reboot.

  • DRS/DPM conflicts with upgrade process:

    • inadequate time given to servers for servers to recover from sleep mode

    • Hosts sent to sleep while updates being evaluated, causing DRS to hold maintenance mode while sleeping hosts awakened, resulting in failed/time-out update process



  • Off-line VMs and templates not automatically migrated during update (maintenance mode) process causing extended unavailability of these assets during update


Additional Notes: Directed to the question of which updates/patches may or may not be "rolled-up" into this update, the release notes are very clear. However, for the sake of convenience, we repeat them here:

Patch Release ESX400-Update01 contains the following individual bulletins:




ESXi 4.0 Update 1 also contains all fixes in the following previously released bundles:


Patch Release ESX400-200906001
Patch Release ESX400-200907001
Patch Release ESX400-200909001

Tuesday, August 18, 2009

In-the-Lab: Full ESX/vMotion Test Lab in a Box, Part 2

In Part 1 of this series we introduced the basic Lab-in-a-Box platform and outlined how it would be used to provide the three major components of a vMotion lab: (1) shared storage, (2) high speed network and (3) multiple ESX hosts. If you have followed along in your lab, you should now have an operating VMware ESXi 4 system with at least two drives and a properly configured network stack.

In Part 2 of this series we're going to deploy a Virtual Storage Appliance (VSA) based on an open storage platform which uses Sun's Zetabyte File System (ZFS) as its underpinnings. We've been working with Nexenta's NexentaStor SAN operating system for some time now and will use it - with its web-based volume management - instead of deploying OpenSolaris and creating storage manually.

Part 2, Choosing a Virtual Storage Architecture


To get started on the VSA, we want to identify some key features and concepts that caused us to choose NexentaStor over a myriad of other options. These are:

  • NexentaStor is based on open storage concepts and licensing;

  • NexentaStor comes in a "free" developer's version with 4TB 2TB of managed storage;

  • NexentaStor developer's version includes snapshots, replication, CIFS, NFS and performance monitoring facilities;

  • NexentaStor is available in a fully supported, commercially licensed variant with very affordable $/TB licensing costs;

  • NexentaStor has proven extremely reliable and forgiving in the lab and in the field;

  • Nexenta is a VMware Technology Alliance Partner with VMware-specific plug-ins (commercial product) that facilitate the production use of NexentaStor with little administrative input;

  • Sun's ZFS (and hence NexentaStor) was designed for commodity hardware and makes good use of additional RAM for cache as well as SSD's for read and write caching;

  • Sun's ZFS is designed to maximize end-to-end data integrity - a key point when ALL system components live in the storage domain (i.e. virtualized);

  • Sun's ZFS employs several "simple but advanced" architectural concepts that maximize performance capabilities on commodity hardware: increasing IOPs and reducing latency;


While the performance features of NexentaStor/ZFS are well outside the capabilities of an inexpensive "all-in-one-box" lab, the concepts behind them are important enough to touch on briefly. Once understood, the concepts behind ZFS make it a compelling architecture to use with virtualized workloads. Eric Sproul has a short slide deck on ZFS that's worth reviewing.

ZFS and Cache - DRAM, Disks and SSD's


Legacy SAN architectures are typically split into two elements: cache and disks. While not always monolithic, the cache in legacy storage typically are single-purpose pools set aside to hold frequently accessed blocks of storage - allowing this information to be read/written from/to RAM instead of disk. Such caches are generally very expensive to expand (when possible) and may only accomodate one specific cache function (i.e. read or write, not both). Storage vendors employ many strategies to "predict" what information should stay in cache and how to manage it to effectively improve overall storage throughput.

[caption id="attachment_968" align="aligncenter" width="405" caption="New cache model used by ZFS allows main memory and fast SSDs to be used as read cache and write cache, reducing the need for large DRAM cache facilities."]New cache model used by ZFS allows main memory and fast SSDs to be used as read cache and write cache, reducing the need for large DRAM cache facilities.[/caption]

Friday, May 22, 2009

Preview: Install ESXi 4.0 to Flash

VMware's vSphere's ESXi 4.0 now installs directly to USB flash from the install CD without the "funky" methods we've explained in earlier posts. By comparison, the installation process is straight-forward, simple and painless.

[caption id="attachment_591" align="aligncenter" width="450" caption="vSphere ESXi Install to USB Flash (click for animation)"]vSphere ESXi Install to USB Flash[/caption]

After the quick installation to USB flash, the system reboots into ESXi for the first time:

Thursday, April 30, 2009

VMware ESXi Update: Build 158874

VMware has released a series of critical patches for ESXi and VMware tools discussed in knowledge base articles 1010135 and 1010136. These are the highlights:

Patch for ESXi:

  • Fixes an issue in the VMkernel TCP/IP stack where adding a system to an HA (High Availability) cluster results in a timeout error.

  • Fixes an issue where a virtual machine might fail if a reserved register is accessed within the guest operating system.

  • Fixes an issue where a virtual machine might stop responding or the progress bar in the graphical user interface might appear to be stuck at 95% when you consolidate a snapshot of a powered on virtual machine.

Sunday, April 12, 2009

Update: ESXi on GT28

Just an updated post to confirm that the GT28 (with KVMoIP SMDC module) has proven to be an excellent platform both in manageability and performance. At a $150/node premium over Supermicro's AS-1021 dual-node, the quad-gigabit interface appointed Tyan GT28 is great for clustered consolidation and the "roomy" memory allowance gives it scalability. The only thing keeping the GT28 from being "perfect" is its lack of a redundant power option.

Our test systems have been running continuously in the lab for over 3 months without a single failure. We'd like to see an updated BIOS for the SMDC module that gives us a bit more control, but the current platform beats Supermicro's BIOS stability with their current AGESA 3.3.0+ and KVMoIP offering (AS-1021, AS-2021, etc.).

However, Supermicro has taken it's 1U/2-node platform to the redundant power stage by offering a 2U/4-node platform with redundant supply built-in. Perfect? Nope - still only twin-gigabit, requiring the single slot be dedicated to additional Ethernet capacity. But then there's the issue with NO knock-out for its KVMoIP-SMDC module, requiring either ugly hack-it-in wiring or the loss of the expansion slot. A serious mistake? Time will tell...

Monday, February 23, 2009

Citrix Waving the White Flag? XenServer now free - as in ESXi...

According to a recent announcement from Citrix, XenServer (without advanced features) is now Free (as in April 2009). The question now? How does XenServer/Free match-up to ESXi/Free and what does it mean for the enterprise customer?

See the community announcement here... and the official press release here...

According to their approach - which reads more like ESXi plus motion - the "free" server still needs a significant investment in "management" products to be "enterprise worthy." This still means an "enterprise-class" virtualization product WITH live motion technology can be had for the U-build price, but the revenue shifts from "product license" to "management and service license." Who does that sound like? VMware (ESXi) and RedHat (CentOS).

Since Citrix will retain the intellectual property rights to its closed-source version of Xen, there is no reason to believe a huge number of open source offerings will immediately crop up. It is more likely this is the first salvo in an ever increasing spiral towards Microsoft's acquisition of Citrix and wholesale incorporation of XenSource into its product line.

Still, the "free" tag line is compelling. Citrix is claiming that for "free" you will get from Citrix and XenSource what you would have to pay $5,000 in licensing to VMware. However, the free version of XenSource will NOT have HA, detailed monitoring or cluster management. You will need their $5,000 "Essentials" license for that...