Short-Take: New Oracle/Sun ZFS Goodies

I wanted to pass-on some information posted by Joerg Moellenkamp at c0t0d0s0.org - some good news for Sun/ZFS users out there about Solaris Express 2010.11 availability, links to details on ZFS encryption features in Solaris 11 Express and clarification on "production use" guidelines. Here's the pull quotes from his posting:

"Darren (Moffat) wrote three really interesting articles about ZFS encryption: The first one is Introducing ZFS Crypto in Oracle Solaris 11 Express. This blog entry gives you a first overview how to use encryption for ZFS datasets. The second one..."

-  Darren Moffat about ZFS encryption, c0t0d0s0.org, 11-16-2010

"There is a long section in the FAQ about licensing and production use: The OTN license just covers development, demo and testing use (Question 14) . However you can use Solaris 11 Express on your production system as well..."

- Solaris 11 Express for production use, c0t0d0s0.org, 11-16-2010

"A lot of changes found their way into the newest release of Solaris, the first release of Oracle Solaris Express 2010.11. The changes are summarized in a lengthy document, however..."

- What's new for the Administrator in Oracle Solaris  Express 2010.11, c0t0d0s0.org, 11-15-2010

Follow the links to Joerg's blog for more details and links back to the source articles. Cheers!

Short-Take: ZFS version ZPOOL Versions

As features are added to ZFS - the ZFS (filesystem) code may change and/or the underlying ZFS POOL code may change. When features are added, older versions of ZFS/ZPOOL will not be able to take advantage of these new features without the ZFS filesystem and/or pool being updated first.

Since ZFS filesystems exist inside of ZFS pools, the ZFS pool may need to be upgraded before a ZFS filesystem upgrade may take place. For instance, in ZFS pool version 24, support for system attributes was added to ZFS. To allow ZFS filesystems to take advantage of these new attributes, ZFS filesystem version 4 (or higher) is required. The proper order to upgrade would be to bring the ZFS pool up to at least version 24, and then upgrade the ZFS filesystem(s) as needed.

Systems running a newer version of ZFS (pool or filesystem) may "understand" an earlier version. However, older versions of ZFS will not be able to access ZFS streams from newer versions of ZFS.

For NexentaStor users, here are the current versions of the ZFS filesystem (see "zfs upgrade -v"):

VER  DESCRIPTION
---  --------------------------------------------------------
 1   Initial ZFS filesystem version
 2   Enhanced directory entries
 3   Case insensitive and File system unique identifier (FUID)
 4   userquota, groupquota properties
 5   System attributes

For NexentaStor users, here are the current versions of the ZFS pool (see "zpool upgrade -v"):

VER  DESCRIPTION
---  --------------------------------------------------------
 1   Initial ZFS version
 2   Ditto blocks (replicated metadata)
 3   Hot spares and double parity RAID-Z
 4   zpool history
 5   Compression using the gzip algorithm
 6   bootfs pool property
 7   Separate intent log devices
 8   Delegated administration
 9   refquota and refreservation properties
 10  Cache devices
 11  Improved scrub performance
 12  Snapshot properties
 13  snapused property
 14  passthrough-x aclinherit
 15  user/group space accounting
 16  stmf property support
 17  Triple-parity RAID-Z
 18  Snapshot user holds
 19  Log device removal
 20  Compression using zle (zero-length encoding)
 21  Deduplication
 22  Received properties
 23  Slim ZIL
 24  System attributes
 25  Improved scrub stats
 26  Improved snapshot deletion performance

As versions change, upgrading the ZFS pool and filesystem is possible using the respective upgrade command. To upgrade all imported ZFS pools, issue the following command as root:

zpool upgrade -a

Likewise, to upgrade the ZFS filesystem(s) inside the pool and all child filesystems, issue the following command as root:

zfs upgrade -r -a

The new ZFS features available to these pool and filesystem version(s) will now be available to the upgraded pools/filesystems.

Short-Take: vSphere Multi-core Virtual Machines

Virtual machines were once relegated to a second class status of single-core vCPU configurations. To get multiple process threads, you had to add to add one "virtual CPU" for each thread. This approach, while functional, had potential serious software licensing ramifications. This topic drew some attention on Jason Boche's blog back in July, 2010 with respect to vSphere 4.1.

With vSphere 4U2 and vSphere 4.1 you have the option of using an advanced configuration setting to change the "virtual cores per socket" to allow thread count needs to have a lesser impact on OS and application licensing. The advanced configuration parameter name is "cpuid.coresPerSocket" (default 1) and acts as a divisor for the virtual hardware setting "CPUs" which must be an integral multiple of the "cpuid.coresPerSocket" value. More on the specifics and limitations of this setting can be found in "Chapter 7, Configuring Virtual Machines" (page 79) of the vSphere Virtual Machine Administrator Guide for vSphere 4.1. [Note: See also VMware KB1010184.]

The value of "cpuid.coresPerSocket" is effectively ignored when "CPUs" is set to 1. In case "cpuid.coresPerSocket" is an imperfect divisor, the power-on operation will fail with the following message in the VI Client's task history:

[caption id="attachment_1726" align="aligncenter" width="328" caption="Virtual core count is imperfect divisor of CPUs"][/caption]

If virtual machine logging is enabled, the following messages (only relevant items listed) will appear in the VM's log (Note: CPUs = 3, cpuid.coresPerSocket = 2):

Nov 08 14:17:43.676: vmx| DICT         virtualHW.version = 7
Nov 08 14:17:43.677: vmx| DICT                  numvcpus = 3
Nov 08 14:17:43.677: vmx| DICT      cpuid.coresPerSocket = 2
Nov 08 14:17:43.727: vmx| VMMon_ConfigMemSched: vmmon.numVCPUs=3
Nov 08 14:17:43.799: vmx| NumVCPUs 3
Nov 08 14:17:44.008: vmx| Msg_Post: Error
Nov 08 14:17:44.008: vmx| [msg.cpuid.asymmetricalCores] The number of VCPUs is not a multiple of the number of cores per socket of your VM, so it cannot be powered on.----------------------------------------
Nov 08 14:17:44.033: vmx| Module CPUID power on failed.

While the configuration guide clearly states (as Jason Boche rightly pointed out in his blog):

The number of virtual CPUs must be divisible by the number of cores per socket. The coresPerSocketsetting must be a power of two.

- Virtual Machine Configuration Guide, vSphere 4.1

We've found that "cpuid.coresPerCPU" simply needs to be a perfect divisor of the "CPUs" value. This tracks much better with prior versions of vSphere where "odd numbered" socket/CPU counts were allowed, so therefore odd numbers of cores-per-CPU allowed provided the division of CPUs by coresPerCPU is integral. Suffice to say, if the manual says "power of two" (1, 2, 4, 8, etc.)  then those are likely the only "supported" configuration available. Any other configuration that "works" (i.e. 3, 5, 6, 7, etc.) will likely be unsupported by VMware in the event of a problem.

That said, odd values of "cpuid.coresPerCPU" do work just fine. Since SOLORI has a large number of AMD-only eco-systems, it is useful to test configurations that match the physical core count of the underlying processors (i.e. 2, 3, 4, 6, 8, 12). For instance, we were able to create a single, multi-core virtual CPU with 3-cores (CPUs = 3, cpuid.coresPerSocket = 3) and run Windows Server 2003 without incident:

[caption id="attachment_1727" align="aligncenter" width="403" caption="Windows Server 2003 with virtual "tri-core" CPU"][/caption]

It follows, then, that we were likewise able to run a 2P virtual machine with a total of 6-cores (3-per CPU) running the same installation of Windows Server 2003 (CPUs = 6, cpuid.coresPerSocket = 3):

[caption id="attachment_1728" align="aligncenter" width="403" caption="Virtual Dual-processor (2P), Tri-core (six cores total)"][/caption]

Here are the relevant vmware log messages associated with this 2P, six total core virtual machine boot-up:

Nov 08 14:54:21.892: vmx| DICT         virtualHW.version = 7
Nov 08 14:54:21.893: vmx| DICT                  numvcpus = 6
Nov 08 14:54:21.893: vmx| DICT      cpuid.coresPerSocket = 3
Nov 08 14:54:21.944: vmx| VMMon_ConfigMemSched: vmmon.numVCPUs=6
Nov 08 14:54:22.009: vmx| NumVCPUs 6
Nov 08 14:54:22.278: vmx| VMX_PowerOn: ModuleTable_PowerOn = 1
Nov 08 14:54:22.279: vcpu-0| VMMon_Start: vcpu-0: worldID=530748
Nov 08 14:54:22.456: vcpu-1| VMMon_Start: vcpu-1: worldID=530749
Nov 08 14:54:22.487: vcpu-2| VMMon_Start: vcpu-2: worldID=530750
Nov 08 14:54:22.489: vcpu-3| VMMon_Start: vcpu-3: worldID=530751
Nov 08 14:54:22.489: vcpu-4| VMMon_Start: vcpu-4: worldID=530752
Nov 08 14:54:22.491: vcpu-5| VMMon_Start: vcpu-5: worldID=530753

It's clear from the log that each virtual core spawns a new virtual machine monitor thread within the VMware kernel. Confirming the distribution of cores from the OS perspective is somewhat nebulous due to the mismatch of the CPU's ID (follows the physical CPU on the ESX host) and the "arbitrary" configuration set through the VI Client. CPU-z shows how this can be confusing:

[caption id="attachment_1729" align="aligncenter" width="407" caption="CPU#1 as described by CPU-z"][/caption]

[caption id="attachment_1730" align="aligncenter" width="406" caption="CPU#2 as described by CPU-z"][/caption]

Note that CPU-z identifies the first 4-cores with what it calls "Processor #1" and the remaining 2-cores with "Processor #2" - this appears arbitrary due to CPU-z's "knowledge" of the physical CPU layout. In (virtual) reality, this assessment by CPU-z is incorrect in terms of cores per CPU, however it does properly demonstrate the existence of two (virtual) CPUs. Here's the same VM with a "cpuid.coresPerSocket" of 6 (again, not 1, 2, 4 or 8 as supported ):

[caption id="attachment_1731" align="aligncenter" width="405" caption="CPU-z demonstrating a 1P, six-core virtual CPU"][/caption]

Note again that CPU-z correctly identifies the underlying physical CPU as an Opteron 2376 (2.3GHz quad-core) but shows 6-cores, 6-threads as configured through VMware. Note also that the "grayed-out" selection for "Processor #1" demonstrates a single processor is enumerated in virtual hardware. [Note: VMware's KB1030067 demonstrates accepted ways of verifying cores per socket in a VM.)

How does this help with per-CPU licensing in a virtual world? It effectively evens the playing field between physical and virtual configurations. In the past (VI3 and early vSphere 4) multiple virtual threads were only possible through the use of additional virtual sockets. This paradigm did not track with OS licensing and CPU-socket-aware application licensing since the OS/applications would recognize the additional threads as CPU sockets in excess of the license count.

With virtual cores, the underlying CPU configuration (2P, 12 total cores, etc.) can be emulated to the virtual machine layer and deliver thread-count parity to the virtual machine. Since most per-CPU licenses speak to the physical hardware layer, this allows for parity between the ESX host CPU count and the virtual machine CPU count, regardless of the number of physical cores.

Also, in NUMA systems where core/socket/memory affinity is a potential performance issue, addressing physical/virtual parity is potentially important. This could have performance implications for AMD 2400/6100 and Intel 5600 systems where 6 and 12 cores/threads are delivered per physical CPU socket.

Short-Take: Jeff Bonwick Leaves Oracle after Two Decades

Jeff Bonwick's last day at Oracle may be September 30, 2010 after two decades with Sun, but his contributions to ZFS and Solaris will live on through Oracle and open source storage for decades to come. In 2007, Bill Moore, Jeff Bonwick (co-founders of ZFS) and Pawel Jakub Dawidek (ported ZFS to FreeBSD) were interviewed by David Brown for the Association for Computing Machinery and discussed the future of file systems. The discussion gave good insights into the visionary thinking behind ZFS and how the designers set out to solve problems that would plague future storage systems.

One thing that has changed, as Bill already mentioned, is that the error rates have remained constant, yet the amount of data and the I/O bandwidths have gone up tremendously. Back when we added large file support to Solaris 2.6, creating a one-terabyte file was a big deal. It took a week and an awful lot of disks to create this file.

Now for comparison, take a look at, say, Greenplum’s database software, which is based on Solaris and ZFS. Greenplum has created a data-warehousing appliance consisting of a rack of 10 Thumpers (SunFire x4500s). They can scan data at a rate of one terabyte per minute. That’s a whole different deal. Now if you’re getting an uncorrectable error occurring once every 10 to 20 terabytes, that’s once every 10 to 20 minutes—which is pretty bad, actually.

- Jeff Bonwick, ACM Queue, November, 2007

But it's quotes like this from Jeff's blog in 2007 that really resonate with my experience:

Custom interconnects can't keep up with Ethernet.  In the time that Fibre Channel went from 1Gb to 4Gb -- a factor of 4 -- Ethernet went from 10Mb to 10Gb -- a factor of 1000.  That SAN is just slowing you down.

Today's world of array products running custom firmware on custom RAID controllers on a Fibre Channel SAN is in for massive disruption. It will be replaced by intelligent storage servers, built from commodity hardware, running an open operating system, speaking over the real network.

- Jeff Bonwick, Sun Blog, April 2007

My old business partner, Craig White, philosopher and network architect at BT let me in on that secret back in the late 90's. At the time I was spreading Ethernet across a small city while Craig was off to Level3 - spreading gigabit Ethernet across entire continents. He made it clear to me that Ethernet - in its simplicity and utility - was like the loyal mutt that never let you down and always rose to meet a fight. Betting against Ethernet's domination as an interconnect was like betting against the house: ultimately a losing proposition. While there will always be room for exotic interconnects, the remaining 95% of the market will look to Ethernet. Lookup "ubiquity" in the dictionary - it's right there next to Ethernet, and it's come a long way since it first appeared on Bob Metcalf's napkin in '73.

Looking back at Jeff's Sun blog, it's pretty clear that Sun's "near-death experience" had the same profound change on the his thinking; and perhaps that change made him ultimately incompatible with the Oracle culture. I doubt a culture that embraces the voracious acquisition and marketing posture of former HP CEO Mark Hurd would likewise embrace the unknown risk and intangible reward framework of openness.

In each case, asking the question with a truly open mind changed the answer.  We killed our more-of-the-same SPARC roadmap and went multi-core, multi-thread, and low-power instead.  We started building AMD and Intel systems.  We launched a wave of innovation in Solaris (DTrace, ZFS, zones, FMA, SMF, FireEngine, CrossBow) and open-sourced all of it.  We started supporting Linux and Windows.  And most recently, we open-sourced Java.  In short, we changed just about everything.  Including, over time, the culture.

Still, there was no guarantee that open-sourcing Solaris would change anything.  It's that same nagging fear you have the first time you throw a party: what if nobody comes?  But in fact, it changed everything: the level of interest, the rate of adoption, the pace of communication.  Most significantly, it changed the way we do development.  It's not just the code that's open, but the entire development process.  And that, in turn, is attracting developers and ISVs whom we couldn't even have spoken to a few years ago.  The openness permits us to have the conversation; the technology makes the conversation interesting.

- Jeff Bonwick, Sun blog, April 2007

This lesson, I fear, cannot be unlearned, and perhaps that's a good thing. There's an side to an engineer's creation that goes way beyond profit and loss, schedules and deadlines, or success and failure. This side probably fits better in the subjective realm of the arts than the objective realm of engineering and capitalism. It's where inspiration and disruptive ideas abide. Reading Bonwick's "fairwell" posting, it's clear to see that the inspirational road ahead has more allure than recidivism at Oracle. I'll leave it in his words:

For me, it's time to try the Next Big Thing. Something I haven't fully fleshed out yet. Something I don't fully understand yet. Something way outside my comfort zone. Something I might fail at. Everything worth doing begins that way. I'll let you know how it goes.

- Jeff Bonwick, Sun blog, September 2010

Short-Take: OpenSolaris mantle assumed by Illumos, OpenIndiana

While Oracle is effectively "closed the source" to key Solaris code by making updates available only when "full releases" are distributed, others in the "formerly OpenSolaris" community are stepping-up to carry the mantle for the community. In an internal memo - leaked to the OpenSolaris news group last month - Oracle makes the new policy clear:

We will distribute updates to approved CDDL or other open source-licensed code following full releases of our enterprise Solaris operating system. In this manner, new technology innovations will show up in our releases before anywhere else. We will no longer distribute source code for the entirety of the Solaris operating system in real-time while it is developed, on a nightly basis.

- Oracle Memo to Solaris Engineering, Aug, 2010

Frankly, Oracle clearly sees the issue of continuous availability to code updates as a threat to its control over its "best-of-breed" acquisition in Solaris. It will be interesting to see how long Oracle takes to reverse the decision (and whether or not it will be too late...)

However, at least two initiatives are stepping-up to carry the mantle of "freely accessible and open" Solaris code to the community: Illumos and OpenIndiana. Illumos' goal can be summed-up as follows:

Well the first thing is that the project is designed here to solve a key problem, and that is that not all of OpenSolaris is really open source. And there's a lot of other potential concerns in the community, but this one is really kind of a core one, and from solving this, I think a lot of other issues can be solved.

- Excerpt, Illumos Announcement Transcript

That said, it's pretty clear that Illumos will be a distinct fork away from "questionable" code (from a licensing perspective.) We already see a lot of chatter/concerns about this in the news/mail groups.

The second announcement comes from thje OpenIndiana group (part of the Illumos Foundation) and appears to be to Solaris as CentOS is to RedHat Enterprise Server. OpenIndiana's press release says it like this:

OpenIndiana, an exciting new distribution of OpenSolaris, built by the community, for the community - available for immediate download! OpenIndiana is a continuation of the OpenSolaris legacy and aims to be binary and package compatible with Oracle Solaris 11 and Solaris 11 Express.

- OpenIndiana Press Release, September 2010

Does any of this mean that OpenSolaris is going away or being discontinued? Strictly speaking: no - it lives on as Solaris 11 Express, et al. It does means control of code changes will be more tightly controlled by Oracle, and - from the reaction of the developer community - this exertion of control may slow or eliminate open source contribution to the Solaris/OpenSolaris corpus. Further, Solaris 11 won't be "free for production use"as earlier versions of Solaris were. It also means that distributions and appliance derivatives (like NexentaStor and Nexenta Core) will be able to thrive despite Oracle's tightening.

Illumous has yet to release a distribution. OpenIndiana has distributions available for download today.

Short-Take: SQL Performance Notes

Here are some Microsoft SQL performance notes from discussions that inevitably crop-up when discussing SQL storage:

1. Where do I find technical resources for the current version of MS SQL?


2. I'm new to SQL I/O performance, how can I learn the basics?


3. The basics talk about SQL 2000, but what about performance considerations due to changes in SQL 2005?


4. How does using SQL Server 6.x versus SQL Server 7.0 and change storage I/O performance assumptions?


5. How does TEMPDB affect storage (and memory) requirements and architecture?


6. How does controller and disk caching affect SQL performance and data integrity?


7. How can I use NAS for storage of SQL database in a test/lab environment?


8. What additional considerations are necessary to implement database mirroring in SQL Server?


9. When do SQL dirty cache pages get flushed to disk?


10. Where can I find Microsoft's general reference sheet on SQL I/O requirements for more information?

From performance tuning to performance testing and diagnostics:

1. I've heard that SQLIOStress has been replaced by SQLIOSim: where can I find out about SQLIOSim to evaluate my storage I/O system before application testing?


2. How do I diagnose and detect "unreported" SQL I/O problems?


3. How do I diagnose stuck/stalled I/O problems in SQL Server?


4. What are Bufwait and Writelog Timeout messages in SQL Server indicating?


5. Can I control SQL Server checkpoint behavior to avoid additional I/O during certain operations?


6. Where can I get the SQLIO benchmark tool to assess the potential of my current configuration?

That should provide a good half-day's reading for any storage/db admin...

Short-Take: iSCSI+Nexenta, Performance Notes

Here are a few performance tips for running iSCSI with NexentaStor in a Windows environment:

1. When using the Windows iSCSI Software Initiator with some workloads, disabling the Nagle Algorithm on the Windows Server is sometimes recommended;


2. Tuning TCP window and iSCSI parameters on each side of the connection can deliver better performance;


3. VMware part of the equation? Adjusting the way VMware handles congestion events could be useful;


4. On NexentaStor, disable the Nagle Algorithm with a value of "1" (default, 4095, enabled)

For storage applications where latency is a paramount issue, these hints just might help...